Multimedia content search method and apparatus

Abstract

The application provides a multimedia content search method and device. After receiving a search sentence input by a user, the application identifies image attribute features contained in the search sentence based on a pre-constructed pattern tree and slot tree. For example, for multimedia content containing a person object, the application can identify a person title and a person quantity. Further, the application searches for a matched picture from an index library based on the identified image attribute features and a text vector corresponding to the search sentence. The method accurately identifies image attribute features (such as a person title and a person quantity) in the search sentence by using a pattern tree and a slot tree corresponding to attribute information of the search sentence, thereby improving the accuracy of a semantic understanding result of the search sentence and finally improving the accuracy of a search result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a multimedia content search method and device. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, multi-modal models enable computers to have the ability to understand multi-modal media resources. Among them, the multi-modal model represented by CLIP (Contrastive Language-Image Pretraining) encodes images and texts into a unified vector space to understand the relationship between different modal resources in language and vision. However, due to the limitations of training data sets and other aspects, the current multi-modal model often has difficulty in mapping the user's personalized search sentence describing image information to the correct image, resulting in poor search results. SUMMARY

[0003] Therefore, the present application provides a multimedia content search method and device to solve the above problems, and the disclosed technical solutions are as follows:

[0004] In a first aspect, the present application provides a multimedia content search method applied to an electronic device, which includes: receiving a search sentence for multimedia content; identifying image attribute features contained in the search sentence based on a pre-constructed pattern tree and a slot tree, the pattern tree representing a plurality of description patterns including the search sentence, each description pattern including at least two attribute types, each attribute type corresponding to a slot tree, and each slot tree including a type of attribute information of the multimedia content, the image attribute features including non-visual information of the multimedia content; converting the search sentence into a text vector; searching for multimedia content search results matching the search sentence from a pre-constructed index library based on the image attribute features and the text vector; and displaying the multimedia content search results. As can be seen, this method accurately identifies the image attribute features in the search sentence using the pattern tree of the search sentence and the slot tree corresponding to the attribute information, wherein the image attribute features represent non-visual information of the multimedia content, such as the title of a person and the number of people. Using this method can accurately identify the non-visual attribute features in the search sentence, thereby improving the accuracy of the semantic understanding result of the search sentence and ultimately improving the accuracy of the search results.

[0005] In a possible implementation, the index library includes a plurality of index information items corresponding to the multimedia content, and each index information item includes attribute information corresponding to a multimedia content and a visual semantic vector corresponding to the multimedia content. In this way, the search can be directly performed in the index library based on the attribute features and the visual semantic vector, thereby improving the accuracy of the search results.

[0006] In a possible implementation, the multimedia content search result matching the search statement is searched from the pre-constructed index library based on the image attribute feature and the text vector, including: comparing the image attribute feature with corresponding information fields of index information items in the index library one by one, and screening out a first index information item set matching the image attribute feature; and comparing the similarity between the text vector and the visual semantic vector, and screening out a second index information item set with a similarity greater than or equal to a preset threshold; and obtaining the multimedia content search result according to the first index information item and the second index information item set. It can be seen that the method searches the image matching the search statement from the index library based on the information in two dimensions of the image attribute feature and the visual semantic feature, and improves the accuracy of the search result.

[0007] In a possible implementation, the image attribute feature contained in the search statement is identified based on the pre-constructed pattern tree and slot tree, including: reading all first child nodes of a root node in the pattern tree; reading first slot nodes from the slot tree with the same name according to the slot name of each first child node; sequentially querying whether there is a matching first slot node according to the character order in the search statement, and if there is a matching first slot node, determining that the current character in the search statement matches the current node in the pattern tree successfully, and the current node is a node corresponding to the matching slot node in the pattern tree; reading all second child nodes of the current node in the pattern tree, and reading corresponding second slot nodes according to the slot name of each second child node; and sequentially querying the remaining characters in the search statement until a leaf node in the pattern tree is matched successfully to determine that the pattern tree has a target search pattern matching the search statement; and determining the attribute feature corresponding to the multimedia content based on the target search pattern and the search statement. It can be seen that the method first reads the node in the pattern tree, and then determines whether the character in the search statement matches the node in the pattern tree according to the slot tree matching the slot name of the node, and if matched, the next character in the search statement is continuously matched until all characters in the search statement are matched successfully with the corresponding nodes in the pattern tree, and the image attribute feature contained in the search statement is determined according to the slot name of the node in the pattern tree. The scheme stores the description pattern (i.e., the pattern tree) of the search statement and the attribute information (i.e., the slot tree) of the multimedia content in the form of a dictionary tree, and the data structure has high query efficiency and small storage space.

[0008] In a possible implementation, the step of sequentially querying whether there is a first type of slot node matching the characters in the search statement according to the character order in the search statement comprises the following steps: reading a first character in the search statement, and querying whether there is a slot node containing the first character in each first sub-node corresponding slot tree; if there is a first slot tree containing the first character, determining whether the first matching slot node corresponding to the first character is a leaf node in the first slot tree; if the slot node corresponding to the first character is not a leaf node in the first slot tree, reading a next character of the first character, and continuing to query whether the next character is contained in all child nodes of the first matching slot node in the first slot tree, until the matching slot node is a leaf node, and it is determined that there is a first type of slot node matching the characters in the search statement.

[0009] In a possible implementation, the step of determining the attribute features of the multimedia content based on the target search pattern and the search statement comprises the following steps: determining that the content matching the person title node in the search statement is a person title based on the person title node contained in the target search pattern; and determining that the content matching the person quantity node in the search statement is a person quantity based on the person quantity node contained in the target search pattern. It can be seen that the scheme can accurately identify the attribute features such as the person title and the person quantity in the search statement, and thus improves the accuracy of the search result.

[0010] In a possible implementation, the process of constructing the slot tree comprises the following steps: constructing a slot tree corresponding to attribute information based on existing information, the existing information comprising at least one of the attribute information of the existing multimedia content in the gallery and the knowledge in the encyclopedia, and the attribute information of the multimedia content comprising at least one of the following: the annotation information input by the user for the multimedia content, the shooting information of the multimedia content, and the entity object label recognized by the electronic device for the multimedia content; and for the multimedia content containing a person object, the annotation information comprises a person title, and the person title comprises at least one of the name of the person, the nickname, and the relationship of the person. It can be seen that the scheme constructs the slot tree based on the attribute information of the existing multimedia content in the gallery and the knowledge in the encyclopedia, and the construction process is simple.

[0011] In a possible implementation, the slot tree comprises a person title slot tree, a numeral slot tree, a measure word slot tree, a stop word slot tree, and a suffix word slot tree, and the person title slot tree comprises at least one of a person name, a nickname, and a relationship of a person; or the slot tree comprises a name slot tree, a nickname slot tree, a relationship slot tree, a numeral slot tree, a measure word slot tree, a stop word slot tree, and a suffix word slot tree.

[0012] In a possible implementation, the attribute information corresponding slot tree is constructed based on existing information, including: constructing a person title slot tree based on the person title annotation information and the person relationship of the existing multimedia content in the electronic device; constructing a number slot tree based on the number knowledge and the numbers contained in the pre-obtained search sentence corpus; constructing a quantifier slot tree based on the quantifier knowledge and the quantifiers contained in the search sentence corpus; constructing a stop word slot tree based on the knowledge and the words without actual meaning contained in the search sentence corpus; and constructing a suffix word slot tree based on the suffix words contained in the search sentence corpus.

[0013] In a possible implementation, the process of constructing the pattern tree includes: obtaining a description pattern corresponding to the search sentence based on the word arrangement order of the attribute corresponding to each slot tree contained in the pre-obtained search sentence corpus based on the knowledge; and constructing a pattern tree based on various description patterns.

[0014] In a second aspect, the present application further provides an electronic device, including: one or more processors, a memory and a touch screen; the memory is configured to store program code; and the processor is configured to run the program code, so that the electronic device implements the multimedia content search method according to any one of the first aspect.

[0015] In a third aspect, the present application further provides a computer readable storage medium having instructions stored thereon, when the instructions are run on an electronic device, the electronic device executes the multimedia content search method according to any one of the first aspect.

[0016] In a fourth aspect, the present application further provides a computer program product having instructions stored thereon, when the computer program product is run on an electronic device, the electronic device executes the network service optimization method according to any one of the first aspect.

[0017] It should be understood that the description of technical features, technical solutions, advantages or similar language in the present application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it can be understood that the description of a feature or advantage means that the specific technical feature, technical solution or advantage is included in at least one embodiment. Therefore, the description of technical features, technical solutions or advantages in the specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and advantages described in the embodiments can be combined in any appropriate manner. Those skilled in the art will understand that the embodiments can be implemented without one or more specific technical features, technical solutions or advantages of a specific embodiment. In other embodiments, additional technical features and advantages can be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0019] Figure 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0020] Figure 2 is a flowchart of a multimedia search method provided by an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a title slot tree provided by an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a name slot tree provided by an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of a nickname slot tree provided by an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of a person relationship slot tree provided by an embodiment of the present application;

[0025] Figure 7 is a schematic diagram of a number slot tree provided by an embodiment of the present application;

[0026] Figure 8 is a schematic diagram of a quantifier slot tree provided by an embodiment of the present application;

[0027] Figure 9 is a schematic diagram of a suffix word slot tree provided by an embodiment of the present application;

[0028] Figure 10 is a schematic diagram of a stop word slot tree provided by an embodiment of the present application;

[0029] Figure 11 is a schematic diagram of a pattern tree provided by an embodiment of the present application;

[0030] Figure 12 is a schematic diagram of a gallery application interface provided by an embodiment of the present application;

[0031] Figure 13 is a schematic diagram of another gallery application interface provided by an embodiment of the present application. DETAILED DESCRIPTION

[0032] The terms “first”, “second”, and “third” and the like in the specification and claims of this application and the accompanying drawings are used for distinguishing between similar objects, not for describing a particular sequential or numerical order. The terms “first”, “second”, and “third” and the like can be understood as a name basis, and used according to costum or only to distinguish similar objects.

[0033] In the embodiments of the present application, the words “exemplary” or “for example” are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as “exemplary” or “for example” in the embodiments of the present application should not be interpreted as being superior or more advantageous than other embodiments or design. In fact, a word “exemplary” or “for example” is used to present concepts in a concrete manner.

[0034] In the related art, in the scene of searching for a portrait picture or a video by using a multi-modal search represented by CLIP, there are the following difficulties in accurately identifying attribute information such as a person's title (such as a person's name, a nickname, a person's relationship, etc.) and the number of persons from search words describing the portrait picture or the video:

[0035] 1. Polysemy of the person's title: The person's title defined by the user is not standardized, such as Zhang San, younger brother, Hua Kai Fu Gui, dear, etc. It can be seen that the person's title in actual application has polysemy and is difficult to accurately identify.

[0036] 2. The limited object of the quantifier has diversity: The quantifier can be used to limit a variety of subjects, and the quantifier limiting the number of persons needs to be accurately extracted, for example, “three” in “three pots of flowers” is not a quantifier limiting the number of persons, but “double” in “double portrait” is a quantifier limiting the number of persons.

[0037] 3. Privacy and power consumption problems: For a solution that needs to upload user data (such as the person's title annotated by the user in the gallery) to the cloud side to train a language model, uploading user data to the cloud side may cause user privacy leakage.

[0038] 4. Identification accuracy problem: One is to directly match and query the search statement and the person's title, which is easy to cause misidentification, for example, the user searches for the lyrics information containing the song “Zhang San's song”, not “Zhang San”. Another is to train a language model by using natural language understanding technology, which is easy to cause missed identification due to the limitation of training corpus, for example, “Xiao Mei's photo” is to find “Xiao Mei”, and the language model word segmentation gets “small”, “beautiful”, and “photo”.

[0039] To solve the above technical problems, the application provides a multimedia content search method, which constructs a slot tree corresponding to image attribute information in advance based on existing information, and constructs a mode tree of search statement mode based on encyclopedic knowledge and search statement corpus. Specific image attribute features in the search statement, such as the title of a person and the number of people, are identified based on the mode tree and the slot tree, that is, the accuracy of the semantic understanding result of the search statement is improved. Further, the matching picture or video is searched based on the two-dimensional features of the image attribute features and the visual semantic vector of the search statement, and the accuracy of the search result is improved.

[0040] The multimedia content search method provided by the application is suitable for end-side electronic devices, such as mobile phones, tablet computers, desktop computers, laptop computers, notebook computers, ultra-mobile personal computers (UMPC), handheld computers, netbooks, personal digital assistants (PDA), wearable electronic devices, smart watches and the like.

[0041] Please refer to Figure 1 , which shows a structural schematic diagram of an electronic device provided by an embodiment of the application.

[0042] As shown in Figure 1 , the electronic device can include a processor 110, a memory 120, a camera 130, and a display screen 140, etc.

[0043] It can be understood that the structure shown in the embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0044] The processor 110 can include one or more processing units, and different processing units can be independent devices or integrated in one or more processors.

[0045] The memory 120 is used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the memory. For example, in the embodiment, the processor 110 can perform multimedia content search by executing the instructions stored in the memory.

[0046] The memory 120 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as an image or video shooting function, an image or video playing function, etc.), and the like. The data storage area can store data (such as image data, video data, etc.) created during use of the electronic device, and the like.

[0047] The camera 130 is configured to capture a still image or a video. The multimedia content of the present application is an image or a video.

[0048] The display screen 140 is configured to display an image, a video, and the like.

[0049] In addition, an operating system runs on the above-mentioned components. For example, The operating system, The operating system, The operating system, and the like. An application program can run in the operating system. For example, the application program related to the present application includes a gallery.

[0050] To solve the problem of poor search effect of the current multimedia content search scheme using a multi-modal model, a multimedia content search method is provided in the embodiments of the present application. The method can be applied to Figure 1 The electronic device shown in the figure, and specifically can run in the gallery application. Moreover, the method can be applied to image search and video search.

[0051] First, the processing modules related to the multimedia content search method in the gallery application are introduced. In the embodiments of the present application, the gallery application can include a gallery service, a search module, and a multi-modal understanding module.

[0052] The gallery service is configured to implement the basic functions of the gallery application, such as storing images and videos, further displaying the stored images or videos, and according to the attribute information corresponding to the received and stored multimedia content of the user, such as adding attribute information or modifying attribute information.

[0053] In the embodiments of the present application, the search module is configured to construct a slot tree, a pattern tree, and a picture index, so as to realize the search of the images or videos stored in the gallery.

[0054] In the embodiments of the present application, the multi-modal understanding module is configured to convert the visual semantic understanding of the image content or the video content into a corresponding vector, and convert the search sentence input by the user into a vector.

[0055] The multimedia content search method provided in the present application will be described in detail below. Figure 2 The method can include the following steps:

[0056] S101, the search module constructs a slot tree corresponding to the image attribute information based on existing information.

[0057] In the embodiments of the present application, the image attribute information includes various types of information related to the image, for example, can include the shooting information of the image (such as shooting time, geographical position, etc.), the user's annotation information of the objects contained in the image (such as person name, nickname, character relationship, etc.), and the image content label. The content label can be the object contained in the image which is automatically identified based on the image content, such as sky, sunrise, sunset, seaside, building, child, animal, etc.

[0058] The existing information here can include at least one of the attribute information of the existing pictures in the gallery and the encyclopedic knowledge.

[0059] In the embodiments of the present application, a dictionary tree, also known as a slot tree, is constructed for part of the type attribute information. The dictionary tree is a tree structure, and the common prefix of different strings does not need to be stored repeatedly, so that the storage space can be saved. At the same time, the common prefix of the string can reduce the query time, maximize the reduction of unnecessary string comparison, and improve the query efficiency.

[0060] In an exemplary embodiment, the information annotated by the user in the image of the portrait, such as the person name, nickname, character relationship, etc. can be collectively referred to as the title, and the slot tree constructed for this type of attribute information is the title slot tree. The title information for constructing the title slot tree can include the title information annotated by the user in the pictures in the gallery. In other embodiments, the title slot tree can also be expanded according to the contact names saved in the address book of the electronic device.

[0061] In an example, Figure 3 A title slot tree is shown, which includes person name, nickname, character relationship, etc., for example, Figure 3 In the example shown, the person name includes Wang Ming, the nickname includes Xiaomei and Xiaoshuai, and the character relationship includes younger brother, son, daughter, mother, and friend. Figure 3 This is only an example, and in actual applications, the title slot tree can include more nodes and more complex structures, which will not be shown one by one.

[0062] In another exemplary embodiment, slot trees can be constructed for person name, nickname, character relationship, etc. respectively, such as person name slot tree, nickname slot tree, character relationship slot tree, etc.

[0063] The person name slot tree only includes the name corresponding to the portrait, which can be created according to the name annotated by the user in the portrait image or video in the gallery, and further can be expanded according to the names of the contacts in the address book. For example, in an example, Figure 4is a schematic diagram of a person name slot tree. The names in the person name slot tree in this example include Zhang Sanfeng, Zhang Sanfeng, Zhang Ming, Wang Ming, and the like.

[0064] The nickname slot tree only includes nicknames corresponding to the person image, and can be created based on the nicknames labeled by the user on the person image or video in the gallery. Further, the nickname slot tree can be extended according to the nicknames of the contacts in the instant messaging application. For example, in an example, Figure 5 is a schematic diagram of a nickname slot tree. The nicknames include Hua Kai Fu Gui, Hua Kai Wang You, Xia Tian, Mian Hua Tang, Xiao Ke Ai, and the like.

[0065] The person relationship slot tree is a dictionary tree created based on the words representing the relationship between people summarized based on encyclopedic knowledge. For example, in an example Figure 6 is a schematic diagram of a person relationship slot tree. The person relationship can include younger brother, younger sister, older brother, older sister, father, mother, son, daughter, friend, classmate, and the like.

[0066] In addition, in the embodiments of the present application, a corresponding slot tree, also referred to as a label slot tree, can be constructed for image content labels. The label slot tree is constructed according to the content labels marked in the gallery.

[0067] In the embodiments of the present application, in order to accurately identify the semantic content of the search statement input by the user, a corresponding slot tree can also be constructed for the numerals, quantifiers, suffix words and stop words (i.e. words without actual semantics) that can appear in the search statement.

[0068] The quantity slot tree is a dictionary tree created based on the numbers that can appear in the search statement. The commonly used number words can be summarized based on encyclopedic knowledge and the corpus of search statements. For example, in an example, Figure 7 is a schematic diagram of a quantity slot tree. The quantity can include double, two, two, three, four, and the like.

[0069] The quantifier slot tree is a dictionary tree created based on the quantifiers that can appear in the search statement. The commonly used quantifiers can be summarized based on encyclopedic knowledge and the corpus of search statements. For example, in an example, Figure 8 is a schematic diagram of a quantifier slot tree. The quantifier can include person, bit, name, and the like.

[0070] The suffix word slot tree is a dictionary tree created based on the suffix words that can appear in the search statement. The dictionary tree can be created based on the corpus of search statements and the commonly used suffix words. For example, Figure 9 is a schematic diagram of a suffix word slot tree. In this example, the suffix words can include group photo, group photo, photo, group photo, group photo, and the like.

[0071] The stop word slot tree is based on the words without actual semantics in the search sentence, such as conjunctions, auxiliary words, etc. The commonly used stop words can be obtained based on the encyclopedia knowledge and the summary of the search sentence corpus. For example, in an example, Figure 10 FIG. 1 is a schematic diagram of a stop word slot tree. The stop words in the example can include and, of, with, follow, and the like.

[0072] S102, the search module constructs a pattern tree corresponding to the search sentence.

[0073] In the embodiments of the present application, the dictionary tree constructed for the arrangement order of the word types in the search sentence is also called a pattern tree.

[0074] In an exemplary embodiment, the rules of the search sentence corpus (i.e. the search sentences of images or videos of multiple users) are summarized and analyzed in combination with the encyclopedia knowledge, and different types of word arrangement modes frequently appearing in the search sentence are obtained.

[0075] The present embodiment takes the search pattern describing portrait information as an example for illustration. For example, as shown in FIG. 2, in an example, the pattern tree can include the following three search patterns describing portrait information: Figure 11

[0076] Search pattern 1: [appellation] [stop word] [suffix]. In the search sentence "brother and small beauty's double photo", "brother" is a title, "and" is a stop word, "small beauty" is a title, "is" is a stop word, "double" is a numeral, "person" is a quantifier, and "photo" is a suffix.

[0077] Search pattern 2: [appellation] [stop word] [suffix], for example, "small handsome photo" in which "small handsome" is a title, "is" is a stop word, and "photo" is a suffix.

[0078] Search pattern 3: [number] [quantifier] [suffix], for example, "five people in the photo" in which "five" is a numeral, "person" is a quantifier, and "photo" is a suffix.

[0079] The above S101-S102 shows the dictionary tree construction process, which only needs to be performed before the first search, and does not need to be performed before each search, but can be updated according to the need.

[0080] S103, the image and its attribute information added or modified by the user in the gallery application. ​

[0081] In an example embodiment, a user can add attribute information of an image in the gallery application, for example, label the attribute information of a picture of a younger brother as "younger brother".

[0082] In another example embodiment, a user can also modify the labeled attribute information, for example, after labeling a picture as "Zhang San", the user modifies the label to "younger brother".

[0083] In yet another example embodiment, a user can add new labels to an image with labeled attribute information, for example, after labeling a picture as "son", the user adds a new label "little love".

[0084] In addition, a user can also modify the image content, for example, doodling, pruning, filter, adjusting image parameters (such as brightness, contrast, saturation, sharpness, etc.

[0085] S104, the gallery service stores the image and its attribute information.

[0086] After the gallery service receives the image and its attribute information, it is stored in the gallery database.

[0087] S105, the gallery service transmits the image to the multi-modal understanding module for visual semantic understanding.

[0088] The gallery service transmits the received image to the multi-modal understanding module, and the multi-modal understanding module performs visual semantic understanding on the image to obtain the corresponding vector of the picture, also known as the visual semantic vector.

[0089] Visual semantic understanding refers to automatically identifying high-level concepts such as objects, scenes, actions, and their relationships from an image. It can help computers understand images more deeply and analyze and understand image content more finely.

[0090] S106, the multi-modal understanding module returns the visual semantic vector to the gallery service.

[0091] The multi-modal understanding module transmits the visual semantic vector obtained through visual semantic understanding to the gallery service.

[0092] S107, the gallery service stores the visual semantic vector.

[0093] For example, the gallery service can store the visual semantic vector returned by the multi-modal understanding module in the gallery database.

[0094] S108, the gallery service transmits the visual semantic vector and the attribute information required for building an index to the search module.

[0095] In practical applications, users usually only pay attention to part of the attribute information of the image (e.g., shooting time, geographical location, who is the shooting object, and the relationship with the object, etc.), and do not pay attention to the information such as the sensitivity of the image, the exposure value, the white balance, and the shutter speed, and only need to construct an index for the attribute that the user pays attention to. Therefore, the gallery service only needs to deliver the visual semantic vector and the attribute information required for constructing the index to the search module.

[0096] For example, the attribute information required for constructing the index can be agreed with the search module and written in the code of the gallery service.

[0097] S109, the search module constructs an index information item corresponding to the picture based on the visual semantic vector and the attribute information.

[0098] In an exemplary embodiment, the visual semantic vector sent by the gallery service includes a vector and a unique identifier of the picture, also known as a picture ID. After receiving the visual semantic vector and the attribute information, the search module writes the visual semantic vector, each item of attribute information, and the picture ID into the corresponding fields of the index library, respectively.

[0099] For example, the attribute information of the image includes the shooting time, the shooting location (i.e., the geographical location), the name of the person, etc., and the index information item of the index library includes the fields of time, location, title, picture semantic vector, picture ID, etc. In this example, the shooting time is written into the time field, the shooting location is written into the location field, the name of the person is written into the title, the visual semantic vector is written into the visual semantic vector field, and the picture ID is written into the picture ID field. In this way, all information is written into the corresponding field, and the index information item corresponding to the picture is obtained.

[0100] The above S103-S109 shows the index construction process, which does not need to be executed before each search, but is executed when the user adds or modifies the picture and its attribute information.

[0101] In addition, in order to improve the endurance of the electronic device, the above index construction process can be performed when the electronic device is charging or in a screen-off state, thereby reducing the power consumption of the index construction process.

[0102] S110, the gallery service receives a search statement.

[0103] For example, the user can input the search statement in the search bar provided by the gallery application, such as Figure 12 As shown, the bottom of the page 100 of the gallery application includes a tab navigation bar 101, such as four tab options of photos 102, albums 103, moments 104, and discovery 105. The page 100 is a photo tab interface. The top of the page 100 includes the title "Photos" 106 of the current tab interface. The search bar 107 is located below the tab title 106, and the user can input the search statement in the search bar 107.

[0104] S111, the gallery service delivers the search sentence to the search module.

[0105] S112, the search module identifies the image attribute features in the search sentence based on the pattern tree and the slot tree.

[0106] In an exemplary embodiment, the process shown in S112 can include the following steps:

[0107] (1) read the root node of the pattern tree as the current node.

[0108] (2) read all child nodes of the current node.

[0109] (3) read the slot nodes in the slot tree with the corresponding name in sequence according to the name of the child node.

[0110] (4) match the slot nodes one by one in the order of characters in the search sentence, if there is a matching slot information, it means that the current node in the pattern tree is hit. Otherwise, stop the query of the subsequent nodes.

[0111] (5) repeat steps (2)-(4) until all characters in the search sentence hit the child nodes in the pattern tree, i.e. the search sentence hits the search pattern in the pattern tree; otherwise, it means that the search sentence does not hit the search pattern in the pattern tree.

[0112] (6) obtain the relevant information contained in the search sentence from the hit search pattern, such as the title of the person, the number of people, the sky, the building, the animal, the sunrise and sunset, etc.

[0113] S113, the search module delivers the search sentence to the multi-modal semantic understanding module for vector conversion.

[0114] S114, the multi-modal semantic understanding module returns the text vector corresponding to the search sentence to the search module.

[0115] The search module delivers the received search sentence to the multi-modal semantic understanding module, the multi-modal semantic understanding module converts the search sentence to obtain the text vector corresponding to the search sentence, and returns the text vector to the search module.

[0116] In an exemplary embodiment, all characters of the search sentence can be directly converted into corresponding vectors.

[0117] In another example embodiment, the multi-modal semantic module can identify non-visual information in the search sentence (also referred to as a first search sentence), and delete the words of the non-visual information in the search sentence to obtain a second search sentence related to visual information, and further perform vector conversion on the second search sentence to obtain a text vector. For example, the first search sentence is "the sunset shot in summer 2023", and the non-visual information "2023" is deleted, and the second search sentence after the non-visual information is deleted is "the sunset shot in summer".

[0118] In S115, the search module searches the index library for pictures matching the image attribute features and the text vector corresponding to the search sentence.

[0119] In an example embodiment, the search module compares the image attribute features with the corresponding fields in the index library one by one. At the same time, the similarity between the text vector corresponding to the search sentence and the visual semantic vector in the index library is compared, and if the similarity is greater than or equal to a threshold value, it is determined that the visual semantic vector matches the search sentence.

[0120] The matching results of the image attribute features and the matching results of the text vector are combined to obtain a search result matching the search sentence.

[0121] In an example embodiment, the search result can include pictures matching both the text vector and all image attribute features, and can also include pictures matching the text vector and partially matching the image attribute features. For example, the search sentence is "the sunset shot in summer 2023 in Beijing", and the search result can include pictures shot in summer 2023, shot in Beijing, and containing the sunset as the shooting object, i.e. pictures completely matching the search sentence, in addition to pictures of all sunsets shot in summer 2023 in other locations, or sunsets shot at other times.

[0122] In S116, the search module sorts the searched pictures.

[0123] In an example embodiment, the pictures in the search result are sorted from high to low according to the matching degree with the search sentence. For example, a search sentence corresponds to multiple image attribute features, and the more attribute features match, the higher the matching degree, and vice versa, the fewer attribute features match, the lower the matching degree.

[0124] For example, in the example in S115, the search sentence is "the sunset shot in summer 2023 in Beijing", and the pictures shot in summer 2023, shot in Beijing, and containing the sunset as the shooting object have the highest matching degree, the pictures of sunsets shot in summer 2023 in other locations have the next highest matching degree, and the pictures of sunsets shot in other times and other locations have the lowest matching degree.

[0125] S117, the search module returns image search results to the image library service.

[0126] In this embodiment of the application, the image search results obtained by the search module are a set of image IDs that match the search query.

[0127] S118, Image Gallery service displays image search results.

[0128] After receiving the search results from the search module, the image library service reads the corresponding image from the image database based on the image ID in the search results and displays it on the search results display interface.

[0129] like Figure 13 The image shown is the image search results display interface 200 of the gallery application. Interface 200 includes a search bar 201, which contains the search content, such as a photo of Xiaomei and her younger brother. A folder display area 202 is located below the search bar 201; this area displays the folder to which the searched images belong. A search results display area 203 is located below the folder display area 202 and displays a preset number of images.

[0130] The multimedia content search method provided in this embodiment pre-constructs a slot tree corresponding to image attribute information based on existing information, building one slot tree for each type of attribute information. It also constructs a pattern tree for search statement patterns based on encyclopedic knowledge and search statement corpus. Upon receiving a user-input search statement, the method identifies the image attribute features contained in the search statement based on the pattern tree and slot tree, and further searches for matching images based on the identified image attribute features and the corresponding text vector of the search statement. This method accurately identifies image attribute features in the search statement by utilizing the pattern tree and the slot tree corresponding to the attribute information, improving the accuracy of the semantic understanding results of the search statement and ultimately increasing the accuracy of the search results.

[0131] Furthermore, this method executes within the electronic device, eliminating the need to report user data (such as information annotated by users on images or videos, like names of people) to the cloud server, thus improving user data security. Moreover, this method identifies image attribute features by traversing a pattern tree and a slot tree; both are trie trees, resulting in high lookup efficiency, low computational load, and low power consumption.

[0132] The following example illustrates the process of identifying image attribute features in the search statement shown in S112:

[0133] Example 1: If the search query is "a photo of Xiaomei and her younger brother", the process of identifying image attribute features is as follows:

[0134] (1) Read the root node of the pattern tree as the current node;

[0135] (2) As shown in FIG. 2, the number node and the appellation node are read from the pattern tree according to the current node; Figure 11

[0136] (3) The number slot node is read from the number slot tree, and the appellation slot node is read from the appellation slot tree;

[0137] (4) The first character "xiao" of the search statement is read, and no node containing the "xiao" character is found in the number slot tree as shown in FIG. 3, so it is determined that the first character of the search statement does not hit the number node. Figure 7

[0138] (5) The node containing the "xiao" character is found in the child nodes of the root node of the appellation slot tree as shown in FIG. 4, and the next character "mei" of the "xiao" character is read from the search statement, and the character is compared with the child nodes of the "xiao" node in the appellation slot tree. As shown in FIG. 5, the child nodes of the "xiao" node in the appellation slot tree include two child nodes "mei" and "shuai", and the "mei" node is hit. Figure 3 Figure 3

[0139] (6) The child node of the appellation node in the pattern tree, i.e. the stop word node, is read, the next character "he" of the "mei" character is read from the search statement, and the node containing the "he" character in the stop word slot tree is found, and it is determined that "he" hits the stop word node.

[0140] (7) The child node of the stop word node in the pattern tree, i.e. the appellation and suffix node, is read, and the next character "di" of the "he" character is further read from the search statement. The node containing the "di" character in the appellation slot tree is found, and the next character "di" of the "di" in the search statement is read, which is the same as the child node of the "di" node in the appellation slot tree, and the "di" in the slot tree is a leaf node, so it is determined that "dididi" in the search statement hits the appellation node in the pattern tree.

[0141] (8) The child node of the appellation node in the pattern tree, i.e. the stop word node, is read, the next character "de" of the second "di" character is read from the search statement, the "de" node is found from the stop word slot tree, and the "de" node is a leaf node, so it is determined that the "de" character in the search statement hits the stop word node in the pattern tree.

[0142] ​​​​(9) Read the next node of the stop word node in the pattern tree, i.e. the number node, read the next character "double" to be compared in the search sentence, find the node containing the "double" character in the number slot tree, and the "double" node is a leaf node, then determine that "double" hits the number node in the pattern tree.

[0143] (10) Read the next node to be compared in the pattern tree, i.e. the quantifiter node, read the next character "person" in the search sentence, find the node of the "person" character from the quantifier slot tree, and the "person" node is a leaf node, then determine that "person" hits the quantifiter node in the pattern tree.

[0144] (11) Read the next node of the pattern tree, i.e. the suffix node, read the next character "photo" in the search sentence, find the node of the "photo" character from the suffix slot tree, and the "photo" node is not a leaf node, continue to read the next character "shadow" in the search sentence, find the node of the "shadow" character from the suffix slot tree, and the "shadow" node is a leaf node, then determine that "photo and shadow" hit the suffix node in the pattern tree. The suffix node is a leaf node of the pattern tree, then determine that the current search sentence hits the search pattern [appellation][stop word][appellation][stop word][number][quantifier][suffix].

[0145] (12) According to the search pattern hit by the search sentence, determine that "Xiao Mei" and "little brother" are person appellations, and "double" is the number of persons.

[0146] Example 2: The search sentence is "Xiao Zhang and little brother's double photo", and the process of recognizing image attribute features is as follows:

[0147] (1) Read the root node of the pattern tree as the current node;

[0148] (2) Read the child nodes of the root node in the pattern tree, i.e. the number node and the appellation node;

[0149] (3) Read the number slot node from the number slot tree, and read the appellation slot node from the appellation slot tree;

[0150] (4) Read the first character "Xiao" of the search sentence, and find no node containing the "Xiao" character in the number slot tree as shown in Figure 7 , then determine that the first character of the search sentence does not hit the number node.

[0151] (5) Read the next character "Zhang" in the search sentence, find the node containing the "Zhang" character in the appellation slot tree, and the "Zhang" node is a leaf node, then determine that "Zhang" hits the appellation node in the pattern tree. Figure 3The node where the character "small" is found in the child nodes of the root node of the title slot tree is shown. The next character "beautiful" of the character "small" is further read from the search statement, and the character is compared with the child nodes of the "small" character node in the title slot tree. As shown in the figure, Figure 3 As shown, the child nodes of the "small" node in the title slot tree include two child nodes "beautiful" and "handsome", and do not include the "Zhang" node, that is, "small Zhang" does not hit any title node.

[0152] (6) The query is stopped, and the current search statement does not hit any search pattern, and the person title and the number of persons are not recognized.

[0153] In this example, when further searching based on the search statement, the text vector corresponding to the search statement is compared with the visual semantic vectors in the index library for similarity to obtain an image matching the search statement.

[0154] Example 3: The search statement is "three flowerpots", and the process of recognizing image attribute features is as follows:

[0155] (1)-(3) are the same as the first three steps of Example 1 and Example 2, which will not be repeated here.

[0156] (4) The first character "three" of the search statement is read, and the "three" node is found in the number slot tree, and the node is a leaf node, so it is determined that the search statement starts with a number. The second character "pot" in the search statement does not hit the slot node corresponding to the quantifier node in the pattern tree, that is, "pot" does not hit any node of the quantifier slot tree, so it is determined that the current pattern is not hit.

[0157] (5) The first character "three" does not query any node in the title slot tree, and it is determined that the search statement does not start with a title.

[0158] (6) The query is stopped, and it is determined that the current search statement does not hit any search pattern.

[0159] This example is the same as Example 2 described above, and when further searching based on the search statement, the text vector corresponding to the search statement is compared with the visual semantic vectors in the index library for similarity to obtain an image matching the search statement.

[0160] In addition, in other embodiments, the search statement is "this year's double photo of small Zhang and his younger brother", as known from Example 2, small Zhang is not matched successfully, that is, the person title and the number of persons contained in the search statement cannot be recognized. In this scenario, the time attribute information "this year" in the search statement is retained, compared with the time field in the index library, and at the same time, the text vector based on the search statement is compared with the visual semantic vectors in the index library for similarity, and the time attribute information and the text vector are combined to obtain an image matching the search statement.

[0161] Those skilled in the art can clearly understand the system, device and unit described above through the description of the embodiments, for the convenience and brevity of description, only the division of the above functional modules is exemplified, in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0162] In several embodiments provided in the embodiment, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0163] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0164] In addition, each functional unit in each embodiment of the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0165] The integrated unit, if realized in the form of software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiment can be embodied in the form of software product in essence or the part of the technical solutions that make contributions to the prior art or all or part of the technical solutions. The computer software product is stored in a storage medium, including a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor execute all or part of the steps of the method described in each embodiment. The foregoing storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk, and various program code storage media.

[0166] The above description is only specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any change or replacement within the technical scope disclosed in the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of searching for multimedia content, characterized by, The method is applied to an electronic device and comprises the following steps: receiving a search sentence for multimedia content; identifying image attribute features contained in the search sentence based on a pre-constructed pattern tree and a slot tree, the pattern tree representing a plurality of description patterns including the search sentence, each description pattern including at least two pattern nodes, each pattern node corresponding to an attribute type, and each pattern node corresponding to a slot tree, each slot tree including a type of attribute information of the multimedia content, the image attribute features including non-visual information of the multimedia content; converting the search sentence into a text vector; searching for multimedia content search results matching the search sentence from a pre-constructed index library based on the image attribute features and the text vector, the index library being constructed based on image attribute features and visual semantic vectors of pictures in a gallery of the electronic device, and each index information item in the index library including a visual semantic vector, a shooting time, a shooting location, and a title of a person in an image; displaying the multimedia content search results; wherein the identification of the image attribute features contained in the search sentence based on the pre-constructed pattern tree and the slot tree comprises the following steps: reading slot trees corresponding to current pattern nodes in the order from a root node to a leaf node of the pattern tree, reading characters in the order contained in the search sentence, and querying slot nodes in the slot trees of the current pattern nodes one by one for a match with the currently read characters until the characters in the search sentence match the slot nodes of the slot trees corresponding to the leaf nodes in the pattern tree, thereby obtaining the image attribute features contained in the search sentence.

2. The method of claim 1, wherein, The index library includes a plurality of index information items corresponding to the multimedia content, and each index information item includes attribute information corresponding to a piece of multimedia content and a visual semantic vector corresponding to the piece of multimedia content.

3. The method of claim 2, wherein, The searching for multimedia content search results matching the search sentence from the pre-constructed index library based on the image attribute features and the text vector comprises the following steps: comparing the image attribute features with corresponding information fields of index information items in the index library one by one to filter out a first set of index information items matching the image attribute features; comparing the similarity between the text vector and the visual semantic vector to filter out a second set of index information items with a similarity greater than or equal to a preset threshold value; and obtaining the multimedia content search results based on the first set of index information items and the second set of index information items.

4. The method according to any one of claims 1 to 3, characterized in that, The identification of the image attribute features contained in the search sentence based on the pre-constructed pattern tree and the slot tree comprises the following steps: reading all first child nodes of a root node in the pattern tree; reading first slot nodes from slot trees with the same name in the order of slot names of the first child nodes; querying whether there are matching first slot nodes in the order of characters in the search sentence one by one, and determining that a current character in the search sentence matches a current node in the pattern tree if there is a matching first slot node, the current node being a node corresponding to the matching slot node in the pattern tree. reading all second child nodes of the current node in the pattern tree, and reading corresponding second type slot nodes according to slot names of each second child node, sequentially querying remaining characters in the search statement until a leaf node in the pattern tree is matched to determine that the pattern tree has a target search pattern matching the search statement; determining attribute features corresponding to the multimedia content based on the target search pattern and the search statement.

5. The method of claim 4, wherein, sequentially querying whether there is a first type slot node matching the characters in the search statement, including: reading a first character in the search statement, and querying whether there is a slot node containing the first character in each slot tree corresponding to each first child node; if there is a first slot tree containing the first character, determining whether a first matching slot node corresponding to the first character is a leaf node in the first slot tree; if the slot node corresponding to the first character is not a leaf node in the first slot tree, reading a next character of the first character, and continuing to query whether there is the next character in all child nodes of the first matching slot node in the first slot tree until a matching slot node that is a leaf node is found, and it is determined that there is a first type slot node matching the characters in the search statement.

6. The method of claim 4, wherein, determining attribute features corresponding to the multimedia content based on the target search pattern and the search statement, including: determining that content in the search statement matching a person title node contained in the target search pattern is a person title based on the person title node; determining that content in the search statement matching a person number node contained in the target search pattern is a person number based on the person number node.

7. The method according to any one of claims 1 to 3, characterized in that, a process of constructing a slot tree, including: constructing a slot tree corresponding to attribute information based on existing information, the existing information including at least one of attribute information corresponding to multimedia content in a gallery and encyclopedic knowledge, the attribute information corresponding to the multimedia content including at least one of the following: annotation information input by a user for the multimedia content, shooting information of the multimedia content, and an entity object label recognized by the electronic device for the multimedia content; for multimedia content containing a person object, the annotation information includes a person title, and the person title includes at least one of a person's name, nickname, and person relationship.

8. The method according to any one of claims 1 to 3, characterized in that, the slot tree includes a person title slot tree, a number slot tree, a measure word slot tree, a stop word slot tree, and a suffix word slot tree, and the person title slot tree includes at least one of a person's name, nickname, and person relationship. alternatively, the slot tree includes a name slot tree, a nickname slot tree, a person relationship slot tree, a number slot tree, a measure word slot tree, a stop word slot tree, and a suffix word slot tree.

9. The method of claim 7, wherein, the constructing of the slot tree corresponding to the attribute information based on the existing information, including: constructing a person title slot tree based on person title annotation information corresponding to existing multimedia content in the electronic device and encyclopedic knowledge of person relationship; constructing a number slot tree based on encyclopedic knowledge of numbers and numbers contained in a search statement corpus obtained in advance; constructing a quantifier slot tree based on quantifiers in the encyclopedia knowledge and the search sentence corpus; constructing a stop word slot tree based on words without actual meaning in the encyclopedia knowledge and the search sentence corpus; constructing a suffix word slot tree based on suffix words in the search sentence corpus.

10. The method according to any one of claims 1 to 3, characterized in that, The process of constructing the pattern tree comprises: obtaining a description pattern corresponding to the search sentence based on an arrangement order of words of a corresponding attribute of each slot tree in the search sentence corpus obtained in advance according to the encyclopedia knowledge; constructing the pattern tree based on various description patterns.

11. An electronic device, comprising: The electronic device comprises one or more processors, a memory and a touch screen; the memory is used to store program code; the processor is used to run the program code, so that the electronic device implements the multimedia content search method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, An instruction is stored thereon, and when the instruction is run on an electronic device, the electronic device executes the multimedia content search method according to any one of claims 1 to 10.

Images