Image searching method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to an image search method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: receiving an image search request of a user based on input information; in response to the image search request, displaying an image search result corresponding to the input information; wherein the image search result is determined based on a first matching result or / and a second matching result, and the first matching result is determined based on the similarity between a feature vector of a text corresponding to the input information and a feature vector of an image in a database; the second matching result is determined based on the similarity between the element information of the text corresponding to the input information and the element information of the image.

Description

Technical Field

[0001] The present disclosure relates to image search technology, and more specifically, to an image search method, device, electronic device, storage medium and program product. Background Art

[0002] In the Internet age, most users have hundreds or thousands of pictures in their photo albums. When users need to find pictures that meet their needs from these hundreds or thousands of pictures, they usually search through each list in the album or search according to a provided time period. The above picture search methods are inefficient and cannot meet users' needs in a timely manner. Summary of the invention

[0003] One object of the present disclosure is to provide a new technical solution for an image search method.

[0004] According to a first aspect of the present disclosure, there is provided an image search method, comprising:

[0005] Receiving an image search request from a user based on input information;

[0006] In response to the image search request, image search results corresponding to the input information are displayed; wherein the image search results are determined based on a first matching result or / and a second matching result, the first matching result is determined based on the similarity between a feature vector of the text corresponding to the input information and a feature vector of an image in a database, and the second matching result is determined based on the similarity between element information of the text corresponding to the input information and element information of the image.

[0007] Optionally, the feature vector of the text corresponding to the input information is obtained by inputting the text corresponding to the input information into a set multimodal model, and the feature vector of the image is obtained by inputting the corresponding image into the multimodal model.

[0008] Optionally, the feature vector of the text corresponding to the input information is obtained by identifying extended text, and the extended text is obtained by expanding the text corresponding to the input information to a content that meets the user's intention.

[0009] Optionally, the element information of the text corresponding to the input information is pre-classified information of the input information, including at least one of person, time, place, scene and event, and the element information of the image is pre-classified information of the searched image, including at least one of person, time, place, scene and event.

[0010] Optionally, the image search result is the intersection of the first matching result and the second matching result.

[0011] Optionally, the characters, scenes and events in the element information of the image are obtained by inputting the corresponding image into a set image processing model.

[0012] Optionally, the characters, scenes and events in the element information of the image are determined based on first recognition information and second recognition information, wherein the first recognition information is information obtained by inputting the corresponding image into a set image processing model, and the second recognition information is text information extracted from the corresponding image.

[0013] Optionally, when the identified image type is inconsistent with any of the preset image types, the characters, scenes and events in the element information of the image are obtained through an image processing model set by the corresponding image input, wherein the preset image types include cartoon images, anime images, emoticon images, social media interfaces, and file display interfaces.

[0014] Optionally, when the identified image type is a file display interface, the element information of the image also includes a document type, wherein the document type is a certificate, a contract, a notice, an announcement, or a PPT presentation.

[0015] Optionally, the time in the element information of the image is the shooting time, and the location in the element information of the image is the shooting location.

[0016] Optionally, the images in the database are images that are authorized by the user and uploaded to a network disk.

[0017] According to a second aspect of the present disclosure, there is provided an image search method, comprising:

[0018] Receiving an image search request from a user based on input information;

[0019] Determine a first matching result based on the similarity between a feature vector of the text corresponding to the input information and a feature vector of an image in a database, and determine a second matching result based on the similarity between element information of the text corresponding to the input information and element information of the image;

[0020] Determine, according to the first matching result and / or the second matching result, an image search result corresponding to the input information;

[0021] The image search results are returned to the client for display.

[0022] Optionally, the images in the database are images that are authorized by the user and uploaded to a network disk.

[0023] According to a third aspect of the present disclosure, there is provided an image search device, comprising:

[0024] A receiving module, used for receiving an image search request from a user based on text corresponding to the input information;

[0025] A display module is used to display image search results corresponding to the text corresponding to the input information in response to the image search request; wherein the image search results are determined based on a first matching result or / and a second matching result, the first matching result is determined based on the similarity between the feature vector of the text corresponding to the input information and the feature vector of the image in the database, and the second matching result is determined based on the similarity between the element information of the text and the element information of the image.

[0026] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is used to control the processor to operate so as to execute the method according to any one of the first aspects of the present disclosure.

[0027] According to a fifth aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described in any one of the first aspects is implemented.

[0028] According to a sixth aspect of the present disclosure, a computer program product is provided. When instructions in the computer program product are executed by a processor, the method described in any one of the first aspects is implemented.

[0029] The image search method provided by the present disclosure receives an image search request based on input information from a user, and in response to the image search request, displays image search results corresponding to the input information. The image search results are determined based on a first matching result and a second matching result. The first matching result is determined based on the similarity between a feature vector of a text and a feature vector of an image in a database, and the second matching result is determined based on the similarity between element information of the text and element information of the image, thereby realizing intelligent search of images. At the same time, the image search results are determined by combining two types of matching results, one type of matching result is determined based on the similarity between the feature vector of the text and the feature vector of the image in the database, and the other type of matching result is determined based on the similarity between the element information of the text and the element information of the image, thereby realizing accurate search of images and improving user experience.

[0030] Features and advantages of the embodiments of the present specification will become apparent from the following detailed description of exemplary embodiments of the present specification with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the specification and, together with the description, serve to explain the principles of the embodiments of the specification.

[0032] Figure 1 A schematic diagram showing a flow chart of an image search method according to some embodiments of the present disclosure is shown;

[0033] Figure 2 A schematic diagram showing a flow chart of an image search method according to some embodiments of the present disclosure is shown;

[0034] Figure 3 A schematic diagram showing a flow chart of an image search method according to some embodiments of the present disclosure is shown;

[0035] Figure 4 A schematic diagram of the process of image processing according to some embodiments of the present disclosure is shown;

[0036] Figure 5 A schematic diagram showing a flow chart of an image search method according to some embodiments of the present disclosure is shown;

[0037] Figure 6 A structural block diagram of an image search device according to some embodiments of the present disclosure is shown;

[0038] Figure 7 A structural block diagram of an image search device according to some embodiments of the present disclosure is shown;

[0039] Figure 8 A structural block diagram of an electronic device according to some embodiments of the present disclosure is shown;

[0040] Fig. 9 A structural block diagram of an electronic device according to some other embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0041] Various exemplary embodiments of the present specification will now be described in detail with reference to the accompanying drawings.

[0042] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the embodiments of the present specification and its application or uses.

[0043] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0044] To solve the above technical problems, the embodiments of the present disclosure provide an image search method, which realizes intelligent search of images based on the text corresponding to the input information, and combines two types of matching results to determine the image search results. One type is the matching result determined based on the similarity between the feature vector of the text corresponding to the input information and the feature vector of the image in the database, and the other type is the matching result determined based on the similarity between the element information of the text corresponding to the input information and the element information of the image, thereby realizing accurate search of images and improving user experience.

[0045] Figure 1 The flowchart of the image search method according to one embodiment of the present disclosure is shown. The method can be applied to an APP (application). The APP can be a client APP or a web APP. Figure 1 As shown, the method includes steps S110 to S120.

[0046] Step S110: receiving an image search request from a user based on input information.

[0047] The input information can be input by triggering the text input box of the APP interface, or by triggering the voice input control of the APP interface. When inputting through the voice input control, the obtained voice signal is converted into text as the input text.

[0048] Step S120, in response to the image search request, displaying image search results corresponding to the text corresponding to the input information; wherein the image search results are determined based on the first matching result and / or the second matching result, the first matching result is determined based on the similarity between the feature vector of the text corresponding to the input information and the feature vector of the image in the database, and the second matching result is determined based on the similarity between the element information of the text corresponding to the input information and the element information of the image.

[0049] In some embodiments, the feature vector of the text corresponding to the input information is obtained by inputting the text corresponding to the input information into a set multimodal model, and the feature vector of the image is obtained by inputting the corresponding image into the multimodal model.

[0050] The multimodal model can extract features of different modal information (e.g., images, text), and has the ability to capture the correlation of different modal information. The multimodal model can be trained based on the data set in the following manner. The data set includes a training set and a test set. Each sample in the training set includes an image and a corresponding annotated text. The images include images captured based on the Internet and images authorized for use by users. When the image is an image captured from the Internet, the corresponding annotated text is extracted based on the accompanying text information of the corresponding image obtained from the Internet. When the image is an image authorized for use by the user, the corresponding annotated text is manually annotated text. First, the training set is used to train the multimodal model to be trained, so that the multimodal model to be trained learns the correlation between the image and the corresponding annotated text in each training sample. Then, the trained multimodal model is tested using the test set to determine whether the multimodal model needs to continue training.

[0051] In some embodiments, the feature vector of the text corresponding to the input information is obtained by identifying the extended text, and the extended text is obtained by extending the text corresponding to the input information to a content that meets the user's intention. The extended text can provide more accurate and richer information, making the feature vector of the text obtained based on the extended text more accurate.

[0052] For example, the input text is expanded according to preset expansion rules and templates to obtain the expanded text that meets the user's intention. For another example, the input text is expanded using a natural language processing model to obtain the expanded text that meets the user's intention.

[0053] In some embodiments, the element information of the text corresponding to the input information is pre-classified information of the input information. The element information of the text includes at least one of a person, time, place, scene, and event. The element information of the text can be a keyword directly extracted from the text corresponding to the input information, or a keyword expanded based on the text corresponding to the input information.

[0054] The characters in the element information of the text include names representing different identities.

[0055] The time element information in the text includes the date and the specific time of the day.

[0056] The location in the element information of the text includes geographical name information, for example, ×× Province, ×× City.

[0057] The scene in the element information of the text includes object information and environment information. Object information includes animals, plants, and objects. Environment information includes natural environment and artificial environment. For example, natural environment includes sea, grassland, desert, blue sky, and dark clouds. For example, artificial environment includes oceanarium, amusement park, and gymnasium.

[0058] The events in the element information of the text are activity contents, such as exercise, eating, and sleeping.

[0059] In some embodiments, the element information of the image is a pre-classified image of the searched image, and the element information of the image includes at least one of person, time, place, scene, and event.

[0060] The characters in the element information of the image include names representing different identities. The criteria for identity classification can be classified in various ways according to different dimensions and fields. For example, when classified by age, the characters include minors, young people, middle-aged people, and elderly people. For example, when classified by gender, the characters include men and women. For example, when classified by occupation, the characters include doctors, teachers, lawyers, etc.

[0061] The time in the element information of the image is the shooting time of the image. The location in the element information of the image is the shooting location of the image. The shooting location is determined based on the latitude and longitude information collected when the image was shot.

[0062] The scene in the element information of the image includes object information and environment information. Object information includes animals, plants, and objects. Environment information includes natural environment and artificial environment. For example, natural environment includes sea, grassland, desert, blue sky, and dark clouds. For example, artificial environment includes oceanarium, amusement park, and gymnasium.

[0063] The events in the element information of the image are information corresponding to the actions performed by the characters in the image, such as exercising, eating, and sleeping.

[0064] In some embodiments, the characters, scenes, and events in the element information of the image are obtained by inputting the corresponding images into a set image processing model. Specifically, the image is segmented using the image processing model to obtain images corresponding to the characters and images corresponding to the scenes, character information is obtained from the images corresponding to the characters, scene information is obtained from the images corresponding to the scenes, and event information is obtained based on the correlation between the characters and scenes in the image. The image processing model can be a Convolutional Neural Network (CNN) model.

[0065] In some embodiments, before the corresponding image is input into the set image processing model to obtain the characters, scenes and events in the element information of the image, the image type is first determined. When the image type obtained by identification is inconsistent with any of the preset image types, the characters, scenes and events in the element information of the image are obtained by the image processing model set by the corresponding image input. When the image type obtained by identification is consistent with one of the preset image types, the operation of obtaining the characters, scenes and events in the element information of the image through the image processing model set by the corresponding image input is no longer performed. The preset image types include cartoon images, anime images, emoticon images, social media interfaces, and file display interfaces. Since valid information such as characters, scenes and events cannot be obtained from the images corresponding to the preset image types, when the image type obtained by identification is consistent with one of the preset image types, the operation of obtaining the characters, scenes and events in the element information of the image through the image processing model set by the corresponding image input is no longer performed, which can reduce unnecessary calculations and improve search efficiency.

[0066] In some embodiments, when the image type obtained by recognition is a file display interface, the element information of the image also includes the document type. The document type is a certificate, a contract, a notice, an announcement, or a PPT presentation. The document type can be determined based on OCR (Optical Character Recognition). Correspondingly, the element information of the text also includes the document type.

[0067] In some embodiments, the characters, scenes and events in the element information of the image are determined based on the first identification information and the second identification information. The first identification information is the information obtained by inputting the corresponding image into a set image processing model. The second identification information is the text information extracted from the corresponding image.

[0068] The characters, scenes and events in the element information of the image are information obtained by aggregating the first identification information and the second identification information.

[0069] The first identification information can be determined by segmenting the image using an image processing model to obtain an image corresponding to a person and an image corresponding to a scene, identifying person information from the image corresponding to the person, identifying scene information from the image corresponding to the scene, and identifying event information based on the correlation between the person and the scene in the image, and determining the first identification information based on the person information, scene information, and event information. The image processing model can be a Convolutional Neural Network (CNN) model.

[0070] The second identification information can be determined in the following manner: using a text recognition model to extract text information from the image, and then determining keywords based on the text information. The keywords determined based on the text information can be keywords directly extracted from the text information, or keywords expanded based on the text information. For example, the text information extracted from the image is "Please turn left to watch the dolphin show", and the second identification information can be "dolphin", "performance", and "aquarium", where "dolphin" and "performance" are keywords directly extracted from "Please turn left to watch the dolphin show", and "aquarium" is a keyword expanded based on "Please turn left to watch the dolphin show". The text recognition model can be a model corresponding to OCR.

[0071] In some embodiments, based on the similarity between the feature vector of the text and the feature vector of the image in the database, an image corresponding to a similarity value exceeding a first preset threshold is obtained as the first matching result.

[0072] In some embodiments, based on the similarity between the element information of the text and the element information of the image, an image corresponding to a similarity value exceeding a second preset threshold is obtained as the second matching result.

[0073] In some embodiments, the image search result is the intersection of the first matching result and the second matching result. It should be noted that, when the intersection of the first matching result and the second matching result is an empty set, the image search result is the union of the first matching result and the second matching result. This can avoid the problem of the image search result being none.

[0074] In some embodiments, displaying the image search results corresponding to the input information specifically includes: determining the display order of the image search results according to the size of the similarity value corresponding to the image in the image search results; and displaying the image search results corresponding to the input information according to the display order of the image search results. The greater the similarity value corresponding to the image in the image search results, the higher the display order. The greater the similarity value corresponding to the image in the image search results, the higher the match between the image and the input information, that is, the higher the match between the image and the image required by the user, and the image with a higher match with the user's needs is placed in a front display position, so that the user can capture the image with a higher match with the image required by the user in a timely manner, thereby improving the user experience.

[0075] Figure 2 FIG. 1 is a flow chart of an image search method according to an embodiment of the present disclosure. The image search method provided in this embodiment can be implemented by a server. Figure 2 As shown, the method includes steps S210 to S240.

[0076] Step S210: receiving an image search request from a user based on input information.

[0077] In response to the information input operation triggered by the user, the client generates an image search request and sends the image search request to the server.

[0078] The information input can be input by triggering the text input box of the APP interface, and can also be input by triggering the voice input control of the APP interface.

[0079] Step S220, determining a first matching result based on the similarity between the feature vector of the text corresponding to the input information and the feature vector of the image in the database, and determining a second matching result based on the similarity between the element information of the text corresponding to the input information and the element information of the image.

[0080] In some embodiments, before executing step S220, the method further includes: inputting the text corresponding to the input information into a set multimodal model to obtain a feature vector of the text. The feature vector of the image in the database is obtained by inputting the corresponding image into the multimodal model. Every time the server receives a new image, the image is input into the multimodal model to obtain a feature vector corresponding to the image. The multimodal model can refer to the above embodiment and will not be described in detail here.

[0081] In some embodiments, before executing step S220, the method further includes: expanding the text corresponding to the input information to obtain an extended text that meets the user's intention. Inputting the extended text into a set multimodal model to obtain a feature vector of the text. The extended text can provide more accurate and richer information, so that the feature vector of the text obtained based on the extended text is more accurate.

[0082] In some embodiments, before executing step S220, the method further includes: extracting element information of the text based on the input information. The element information of the text includes at least one of a person, time, place, scene, and event. The element information of the text can be a keyword directly extracted from the input information, or a keyword expanded based on the input information. The person, time, place, scene, and event in the element information of the text can refer to the above embodiment, and will not be described in detail here.

[0083] The element information of the image includes at least one of a person, time, place, scene, and event, which can be referred to in the above embodiment and will not be described in detail here.

[0084] In some embodiments, each time a new image is received by the server, the image processing model is used to obtain the characters, scenes and events in the element information of the image. Specifically, the image processing model is used to segment the image to obtain images corresponding to characters and images corresponding to scenes, character information is obtained from the images corresponding to characters, scene information is obtained from the images corresponding to scenes, and event information is obtained based on the correlation between characters and scenes in the image.

[0085] In some embodiments, before the server receives each new image and uses the image processing model to obtain the characters, scenes, and events in the element information of the image, the image type is first determined. When the image type obtained by identification is inconsistent with any of the preset image types, the image processing model is used to obtain the characters, scenes, and events in the element information of the image. When the image type obtained by identification is consistent with one of the preset image types, the image processing model is no longer used to obtain the characters, scenes, and events in the element information of the image. The preset image types include cartoon images, anime images, emoticon images, social media interfaces, and file display interfaces. Since valid information such as characters, scenes, and events cannot be obtained from images corresponding to the preset image types, when the image type obtained by identification is consistent with one of the preset image types, the operation of obtaining the characters, scenes, and events in the element information of the image through the image processing model set by the corresponding image input is no longer performed, which can reduce unnecessary calculations and improve search efficiency.

[0086] In some embodiments, when the image type obtained by recognition is a file display interface, the element information of the image also includes the document type. The document type is a certificate, a contract, a notice, an announcement, or a PPT presentation. The document type can be determined based on OCR (Optical Character Recognition). Correspondingly, the element information of the text also includes the document type.

[0087] In some embodiments, each time a new image is received by the server, the server inputs the image into a set image processing model to obtain first identification information, and extracts text information from the corresponding image as second identification information. Based on the first identification information and the second identification information, the characters, scenes, and events in the element information of the image are determined. The method for determining the first identification information and the second identification information can refer to the above embodiment, and no further details are given here.

[0088] In some embodiments, based on the similarity between the feature vector of the text and the feature vector of the image in the database, an image corresponding to a similarity value exceeding a first preset threshold is obtained as the first matching result.

[0089] In some embodiments, based on the similarity between the element information of the text and the element information of the image, an image corresponding to a similarity value exceeding a second preset threshold is obtained as the second matching result.

[0090] Step S230: determining image search results corresponding to the input information according to the first matching result and / or the second matching result.

[0091] In some embodiments, the image search result is the intersection of the first matching result and the second matching result. It should be noted that, when the intersection of the first matching result and the second matching result is an empty set, the image search result is the union of the first matching result and the second matching result. This can avoid the problem of the image search result being none.

[0092] Step S240: Return the image search results to the client for display.

[0093] In some embodiments, the method further includes: determining the display ranking of the image search results according to the size of the similarity value corresponding to the image in the image search results; generating the image search results according to the image to be displayed and the corresponding display ranking; and returning the image search results to the client for display. The larger the similarity value corresponding to the image in the image search results, the higher the display ranking.

[0094] The image search method provided by the present invention is described below with a specific embodiment. Figure 3 As shown, the image search method includes steps S301 to S314.

[0095] Step S301: In response to an image upload operation triggered by a user, the client sends the image to a server.

[0096] Step S302: the server receives the image.

[0097] Step S303: input the image into the multimodal model to obtain a feature vector of the image.

[0098] Step S304, input the received image into the set image processing model to obtain the first recognition information, extract text information from the corresponding image as the second recognition information, and determine the person, scene and event corresponding to the image based on the first recognition information and the second recognition information.

[0099] Step S305, obtaining the shooting time and shooting location of the image as the time and location corresponding to the image.

[0100] Step S306, determining element information of the image according to the characters, scenes, events, incidents and places corresponding to the image.

[0101] Step S307, storing the image, the feature vector of the image, and the element information of the image in a database.

[0102] The above steps S301 to S307 are processing steps before the image is stored in the database. For details, see Figure 4 .

[0103] Step S308: In response to the text input operation triggered by the user, the client generates an image search request and sends the image search request to the server.

[0104] Step S309: receiving an image search request from the user based on the input text.

[0105] Step S310, input the text into the set multimodal model to obtain the feature vector of the text, determine the similarity between the feature vector of the text and the feature vector of the image in the database, and determine the first matching result based on the similarity between the feature vector of the text and the feature vector of the image in the database.

[0106] Step S311, extracting text element information based on the input text, determining the similarity between the text element information and the image element information, and determining a second matching result according to the similarity between the text element information and the image element information.

[0107] Step S312: Determine the image search result corresponding to the text according to the first matching result and the second matching result.

[0108] Step S313: Return the image search results to the client for display.

[0109] Step S314: The client receives the image search results and displays them.

[0110] The above steps S308 to S314 are image search processing steps, and can be found in detail. Figure 5 .

[0111] It should be noted that the image search method provided in any of the above embodiments is applicable to network disk apps. The database in any of the above embodiments is images uploaded to the network disk with user authorization.

[0112] The present disclosure also provides an image search device. Figure 6 FIG. 4 shows a structural block diagram of an image search device according to some embodiments. Figure 6 As shown, the image search device 600 may include a receiving module 610 and a display module 620 .

[0113] The receiving module 610 is used to receive an image search request from a user based on input information.

[0114] The display module 620 is used to respond to an image search request and display image search results corresponding to the input information; wherein the image search results are determined based on a first matching result or / and a second matching result, the first matching result is determined based on the similarity between a feature vector of the text corresponding to the input information and a feature vector of an image in a database, and the second matching result is determined based on the similarity between element information of the text corresponding to the input information and element information of the image.

[0115] In some embodiments, the feature vector of the text is obtained by inputting the text corresponding to the input information into a set multimodal model, and the feature vector of the image is obtained by inputting the corresponding image into the multimodal model.

[0116] In some embodiments, the feature vector of the text corresponding to the input information is obtained by identifying the extended text, and the extended text is obtained by expanding the text to a content that meets the user's intention.

[0117] In some embodiments, the element information of the text corresponding to the input information is pre-classified information of the input information, including at least one of person, time, place, scene, and event, and the element information of the image is pre-classified information of the searched image, including at least one of person, time, place, scene, and event.

[0118] In some embodiments, the image search result is an intersection of the first matching result and the second matching result.

[0119] In some embodiments, the characters, scenes and events in the element information of the image are obtained by inputting the corresponding image into a set image processing model.

[0120] In some embodiments, the characters, scenes, and events in the element information of the image are determined based on first identification information and second identification information. The first identification information is the information obtained by inputting the corresponding image into a set image processing model, and the second identification information is the text information extracted from the corresponding image.

[0121] In some embodiments, when the identified image type is inconsistent with any of the preset image types, the characters, scenes, and events in the element information of the image are obtained through the image processing model set by the corresponding image input, where the preset image types include cartoon images, anime images, emoticon images, social media interfaces, and file display interfaces.

[0122] In some embodiments, when the identified image type is a file display interface, the element information of the image also includes a document type, wherein the document type is a certificate, a contract, a notice, an announcement, or a PPT presentation.

[0123] In some embodiments, the time in the element information of the image is the shooting time, and the location in the element information of the image is the shooting location.

[0124] The present disclosure also provides an image search device. Figure 7 FIG. 4 shows a structural block diagram of an image search device according to some embodiments. Figure 7 As shown, the image search device 700 may include a receiving module 710 , a matching result determination module 720 , an image search result determination module 730 and an image search result sending module 740 .

[0125] The receiving module 710 is used to receive an image search request from a user based on input information.

[0126] The matching result determination module 720 is used to determine the first matching result based on the similarity between the feature vector of the text corresponding to the input information and the feature vector of the image in the database, and to determine the second matching result based on the similarity between the element information of the text corresponding to the input information and the element information of the image.

[0127] The image search result determination module 730 is used to determine the image search result corresponding to the input information according to the first matching result and / or the second matching result.

[0128] The image search result sending module 740 is used to return the image search result to the client for display.

[0129] The present disclosure also provides an electronic device for implementing any of the above method embodiments. Figure 8 The structure block diagram of the electronic device 800 according to some embodiments is shown. The electronic device 800 may be a PC, a workstation, a notebook computer, a server, etc., which is not limited here.

[0130] like Figure 8 As shown, the electronic device 800 includes a processor 810 and a memory 820 for storing instructions executable by the processor 810. The processor 810 is configured to implement the image search method according to any embodiment of the present disclosure when executing the instructions stored in the memory 820.

[0131] The processor 810 is used to execute computer instructions, which can be written in an instruction set of an architecture such as x86, Arm, RISC, MIPS, SSE, etc. The memory 820 includes, for example, ROM (read-only memory), RAM (random access memory), a non-volatile memory such as a hard disk, etc., which is not limited here.

[0132] Fig. 9 FIG. 2 shows a structural block diagram of an electronic device according to some other embodiments. Fig. 9As shown, the electronic device 900 may include a display device 930, an interface device 940, a communication device 950, an input device 960, etc. in addition to a processor 910 and a memory 920.

[0133] The interface device 940 includes, for example, a USB interface, a bus interface, a network interface, etc. The communication device 950 is capable of wired or wireless communication, for example, and the communication device 950 may include at least one short-range communication module, for example, any module for short-range wireless communication based on short-range wireless communication protocols such as Hilink protocol, WiFi (IEEE 802.11 protocol), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, LiFi, etc. The communication device 950 may also include a remote communication module, for example, any module for WLAN, GPRS, 2G / 3G / 4G / 5G remote communication. The display device 930 can display the application interface. The input device 960 may include a touch screen, a keyboard, a mouse, a microphone, a camera, etc., which are not limited here.

[0134] The present disclosure also provides a non-volatile computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the image search method provided in any of the above embodiments is implemented.

[0135] The present disclosure also provides a computer program product, which, when instructions in the computer program product are executed by a processor, implements the image search method provided in any of the above embodiments.

[0136] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For the device embodiment, its related parts can be referred to the partial description of the method embodiment.

[0137] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0138] The embodiments of the present specification may be systems, methods and / or computer program products. The computer program product may include a computer-readable storage medium carrying computer instructions for causing a processor to implement various aspects of the embodiments of the present specification.

[0139] A computer-readable storage medium may be a tangible device that can hold and store computer instructions used by a computer instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which computer instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0140] The computer instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device through a network layer, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network layer may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network layer adapter card or network layer interface in each computing / processing device receives the computer instructions from the network layer and forwards the computer instructions for storage in a computer-readable storage medium in each computing / processing device.

[0141] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to the multiple embodiments of this specification. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a computer instruction, and a part of a module, a program segment or a computer instruction contains one or more executable computer instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0142] The embodiments of the present specification have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image search method, characterized in that: include: Receiving an image search request from a user based on input information; In response to the image search request, image search results corresponding to the input information are displayed; wherein the image search results are determined based on a first matching result or / and a second matching result, the first matching result is determined based on the similarity between a feature vector of the text corresponding to the input information and a feature vector of an image in a database, and the second matching result is determined based on the similarity between element information of the text corresponding to the input information and element information of the image.

2. The method according to claim 1, characterized in that The feature vector of the text corresponding to the input information is obtained by inputting the text corresponding to the input information into a set multimodal model, and the feature vector of the image is obtained by inputting the corresponding image into the multimodal model.

3. The method according to claim 1, characterized in that The element information of the text corresponding to the input information is pre-classified information of the input information, including at least one of person, time, place, scene, and event; the element information of the image is pre-classified information of the searched image, including at least one of person, time, place, scene, and event.

4. The method according to claim 1, characterized in that: The characters, scenes and events in the element information of the image are obtained by inputting the corresponding images into a set image processing model.

5. The method according to claim 1, characterized in that The characters, scenes and events in the element information of the image are determined based on first identification information and second identification information. The first identification information is information obtained by inputting the corresponding image into a set image processing model, and the second identification information is text information extracted from the corresponding image.

6. An image search method, characterized in that: include: Receiving an image search request from a user based on input information; Determine a first matching result based on the similarity between a feature vector of the text corresponding to the input information and a feature vector of an image in a database, and determine a second matching result based on the similarity between element information of the text corresponding to the input information and element information of the image; Determine, according to the first matching result and / or the second matching result, an image search result corresponding to the input information; The image search results are returned to the client for display.

7. An image search device, characterized in that: include: A receiving module, used for receiving an image search request from a user based on text corresponding to the input information; A display module is used to display image search results corresponding to the text corresponding to the input information in response to the image search request; wherein the image search results are determined based on a first matching result or / and a second matching result, the first matching result is determined based on the similarity between the feature vector of the text corresponding to the input information and the feature vector of the image in the database, and the second matching result is determined based on the similarity between the element information of the text and the element information of the image.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the computer program is used to control the processor to operate so as to execute the method according to any one of claims 1 to 6.

9. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images