Visual media search method and device, storage medium and program product

Abstract

The embodiment of the invention provides a visual media searching method and device, a storage medium and a program product. The method is suitable for the electronic equipment and comprises the steps that after a search operation on a search statement input to a search box is received, a first candidate set, a second candidate set and a third candidate set are determined, the search statement comprises a time entity and a first keyword, the time entity corresponds to a first time search range, and the first keyword corresponds to a second time search range; the first candidate set is a set matched with the first keyword, the second candidate set is a set matched with a first time search range, the third candidate set is a set matched with a second time search range, and the second time search range is obtained by expanding based on a time entity; when an intersection exists between the first candidate set and the second candidate set, displaying a search result set; and when the first candidate set and the second candidate set do not have the intersection, displaying the recommendation set. According to the technical scheme provided by the embodiment of the invention, the probability of hitting the search expectation of the user can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of terminal, and in particular to a visual media search method and device, a storage medium and a program product. BACKGROUND

[0002] With the popularity of intelligent terminals, more and more users use intelligent terminals, such as mobile phones, to take photos and videos, and store the taken photos and videos in the gallery of an electronic device, so as to record the details of life. In addition, users can also download pictures, take screenshots of the interface of the mobile phone, and store the downloaded pictures and screenshots in the gallery of the electronic device.

[0003] In order to facilitate users to manage and view pictures and videos in the terminal, the gallery application or other similar applications of the terminal device are configured with a picture management function and a picture search function. For example, the gallery application in the terminal can classify the pictures and videos in the terminal according to the time and location of taking the pictures and videos, and generate corresponding albums. Users can view related pictures and videos by searching for time and location information. SUMMARY

[0004] Aspects of the present application provide a visual media search method, device, storage medium and program product, which can recall the same series of photos or videos taken by the user continuously before and after the start time point or the end time point of the time search range. In addition, in the case that the user inputs an incorrect time entity resulting in an empty search result, the application recommends pictures and videos that are more in line with the actual search needs of the user, so as to improve the search hit rate.

[0005] In a first aspect, a visual media search method is provided, which is suitable for an electronic device and includes the following steps:

[0006] displaying a first interface, the first interface including a search box;

[0007] after receiving a search operation of a search statement input into the search box, determining a first candidate set, a second candidate set and a third candidate set, wherein the search statement includes a time entity and a first keyword, the time entity corresponds to a first time search range, the first candidate set is a visual media set matching the first keyword, the second candidate set is a visual media set matching the first time search range, and the third candidate set is a visual media set matching a second time search range, the second time search range being obtained by expanding the time entity;

[0008] determining whether the first candidate set and the second candidate set have an intersection;

[0009] When the first candidate set and the second candidate set have an intersection, a search results set is displayed, which includes the intersection of the first candidate set and the second candidate set;

[0010] When the first candidate set and the second candidate set have no intersection, a recommended set is displayed, which includes the intersection of the first candidate set and the third candidate set.

[0011] Optionally, the intersection of the first candidate set and the second candidate set can be determined as the search results set.

[0012] Optionally, the intersection of the first candidate set and the third candidate set is determined as the recommendation set.

[0013] Optionally, when the first candidate set and the second candidate set do not intersect, a recommended set and a first prompt message are displayed, which are used to indicate that the recommended set is recommended content.

[0014] Optionally, when the first candidate set and the second candidate set do not intersect, a recommended set and a second prompt message are displayed, which are used to indicate that the search query has no matching results.

[0015] Optionally, the search results set may also include a first candidate visual media file, which does not belong to the intersection of the first candidate set and the second candidate set, and the interval between the acquisition time of the first candidate visual media file and the first time search range is less than a preset threshold.

[0016] Optionally, the intersection of the first candidate set and the third candidate set includes a second candidate visual media file, and the interval between the acquisition time of the second candidate visual media file and the first time search range is greater than a preset threshold. In other words, the interval between the acquisition time of the second candidate visual media file and the first time search range is greater than the interval between the acquisition time of the first candidate visual media file and the first time search range.

[0017] The interval between the acquisition time and the first time search range refers to the interval between the acquisition time and the first time point, which is the one closest to the acquisition time among the start and end times of the first time search range.

[0018] The technical solution provided in this application can recall photos or videos taken consecutively before and after the start or end time point of a user's search within a time range, thereby meeting the user's search expectations and improving the user's search experience. Furthermore, if the user enters an incorrect time entity, resulting in empty search results, images and videos whose capture time falls within an extended time search range can be recommended to the user, increasing the probability of matching the user's search expectations and thus improving the user's search experience.

[0019] In one embodiment provided in the first aspect, the above method may further include:

[0020] After receiving a search operation on the search statement entered into the search box, the time entity and the first keyword are semantically identified;

[0021] When the year of the time entity is identified, the first time search range is determined based on the year;

[0022] When the year of the time entity is not identified, N first time search ranges are determined based on the most recent N years, where N is greater than 1 and is an integer.

[0023] In one alternative implementation, when the year of a time entity is identified, the time range corresponding to the time entity in that year is determined as the first time search range.

[0024] In one optional implementation, when the year of the time entity is not identified, the N time ranges corresponding to the time entity in the most recent N years are determined as N first time search ranges.

[0025] In this embodiment, semantic recognition is performed on time entities to identify whether they include a year. When a time entity includes a year expressed numerically (e.g., time entity "July 2023") or a year expressed non-numerically (e.g., time entity "July of last year"), the year of the time entity can be identified through semantic recognition. When a time entity does not include a year expressed numerically or a year expressed non-numerically (e.g., time entity "July"), the year of the time entity cannot be identified through semantic recognition.

[0026] In one embodiment provided in the first aspect, the above method may further include:

[0027] When the year is not expressed as a number, the previous year and / or the next year are extended to obtain the second time search range;

[0028] When the year is expressed numerically or the year in which the time entity is not identified, the second time search range is obtained by expanding the previous and / or next time range of the first time search range.

[0029] In one alternative implementation, when the year in the time entity is a non-numeric expression, the preceding year and / or following year are extended, and the year in the time entity is replaced with the preceding year to obtain a second time search range, and / or the year in the time entity is replaced with the following year to obtain a second time search range.

[0030] In one alternative implementation, the year in the time entity is replaced with the previous year, and the time range corresponding to the replaced time entity is determined as the second time search range.

[0031] Optionally, the year in the time entity can be replaced with the following year, and the time range corresponding to the replaced time entity can be determined as the second time search range.

[0032] In one alternative implementation, when the year is expressed numerically or the year for which the time entity is not identified, the preceding time range of the first time search range is extended and the preceding time range is determined as the second time search range, and / or the following time range of the first time search range is extended and the following time range is determined as the second time search range.

[0033] Optionally, the time difference between the start time of the preceding time range and the start time of the first time search range is less than or equal to a first time difference. For example, the time difference between the start time of the preceding time range and the start time of the first time search range is the duration corresponding to the first time search range.

[0034] Optionally, the time difference between the start time of the latter time range and the start time of the first time search range is less than or equal to the second time difference. For example, the time difference between the start time of the latter time range and the start time of the first time search range is the duration corresponding to the first time search range.

[0035] Optionally, the first search scope corresponds to a duration of less than or equal to one month.

[0036] Optional, the first time difference is 1 month.

[0037] Optionally, the end time of the previous time range is the start time of the first time search range.

[0038] Optionally, the start time of the latter time range is the end time of the first time search range.

[0039] In this embodiment, when a user uses non-numerical expressions such as "this year," "last year," or "the year before last" in the time entity, the year in the time entity is expanded before and after to obtain a second time search range. In other words, the technical solution provided in this application can recommend images and videos that better match the user's actual psychological needs, even when the search results are empty due to incorrect year information input by the user, thereby improving the user's search experience.

[0040] Furthermore, when users use numerical expressions such as "2024" or "2020" in the time entity, or when the time entity does not include a year, the first time search range is expanded both forward and backward to obtain a second time search range. In other words, the technical solution provided in this application can recommend images and videos that better match the user's actual psychological needs, thus improving the user search experience, even when the search results are empty due to incorrect time element input (the time element belongs to a lower level than the year, such as "August" or "May Day").

[0041] In one embodiment provided in the first aspect, the method may further include: the first candidate set includes a plurality of first visual media files; the method may further include:

[0042] The multiple first visual media files are clustered according to their acquisition time to obtain K clusters, where K ≥ 1 and is an integer.

[0043] The target cluster is determined based on whether the collection time range partially overlaps with the first time search range;

[0044] The search results set is determined based on the target cluster.

[0045] Optionally, a clustering algorithm is used to cluster the multiple first visual media files according to their acquisition time to obtain K clusters.

[0046] In this solution, a clustering algorithm is used to group visual media files with similar capture times into clusters. This means that the capture times of the visual media files within each cluster are relatively close. Thus, the clustering algorithm can retrieve photos or videos taken consecutively before or after the start or end time points of a user's search range.

[0047] Optionally, the start time of the acquisition time range for each cluster is the acquisition time of the earliest acquired visual media file in that cluster, and the end time is the acquisition time of the latest acquired visual media file in that cluster; there is no overlap between the acquisition time ranges of the K clusters.

[0048] Optionally, the target cluster includes visual media files whose acquisition time is within the time search range and visual media files whose acquisition time is not within the time search range.

[0049] The number of target clusters can be one or more.

[0050] In an alternative implementation, the union of one or more target clusters can be determined as the search result.

[0051] Considering that there may be visual media files that have not been successfully clustered within the time search range, in another possible implementation, the union of one or more target clusters with the first intersection can be determined as the search results set, where the first intersection refers to the intersection of the first candidate set and the second candidate set.

[0052] In one embodiment provided in the first aspect, the start time of the first time search range is a first time point and the end time is a second time point;

[0053] The step of determining the target cluster based on whether the collection time range partially overlaps with the first time search range includes:

[0054] When the start time of the collection time range of the kth cluster in the K clusters is between the first time point and the second time point and the end time is greater than the second time point, it is determined that the collection time range of the kth cluster partially overlaps with the first time search range.

[0055] When the end time of the collection time range of the kth cluster is between the first time point and the second time point and the start time is less than the first time point, it is determined that the collection time range of the kth cluster partially overlaps with the first time search range.

[0056] When the start time of the collection time range of the kth cluster is less than the first time point and the end time is greater than the second time point, it is determined that the collection time range of the kth cluster partially overlaps with the first time search range; k is an integer, 1≤k≤K;

[0057] Clusters whose collection time range partially overlaps with the first time search range among the K clusters are identified as target clusters.

[0058] The time between the first time point and the second time point refers to a time greater than the first time point and less than the second time point.

[0059] In one implementation provided in the first aspect, the second candidate set and the third candidate set are determined simultaneously.

[0060] In this embodiment, the second and third candidate sets can be determined simultaneously. That is, the second and third candidate sets can be determined simultaneously during a single traversal of multiple visual media files stored in the electronic device. As is well known, when the number of visual media files stored in an electronic device is large, the traversal time is long. The technical solution provided in this embodiment can reduce the number of traversals of multiple visual media files stored in the electronic device, thereby reducing search latency.

[0061] Secondly, a visual media search method is provided, applicable to electronic devices, including:

[0062] After receiving the first search statement input by the user, the first search results interface is displayed. The first search statement includes a first time entity and a first keyword. The first time entity corresponds to a first time search range. The first search results interface includes a first image and a second image. The first image and the second image match the first keyword. The acquisition time of the first image is within the first time search range. The acquisition time of the second image is not within the first time search range. The interval between the acquisition time of the second image and the first time range is a first time interval.

[0063] After receiving the second search statement input by the user, a second search result interface is displayed. The second search statement includes the second time entity and the first keyword. The second time entity corresponds to the second time search range. The second search result interface includes a first prompt message and a recommended image. The first prompt message is used to indicate that the recommended image is recommended content. The recommended image matches the first keyword. The interval between the collection time of the recommended image and the second time search range is the second time interval.

[0064] The second time interval is greater than the first time interval.

[0065] The technical solution provided in this application can recall photos or videos taken by the user before or after the start or end time point of the time search range. In addition, if the user enters an incorrect time entity, resulting in empty search results, the solution can recommend images and videos that better match the user's actual search needs, thereby improving the search hit rate.

[0066] In one embodiment provided in the second aspect, the second search result interface further includes a second prompt message, which is used to indicate that the second search statement has no matching results.

[0067] This serves as a notification to the user that the recommended content is not a match for the second search query.

[0068] Optionally, the first time interval is the interval between the acquisition time of the second image and the first time point, where the first time point is the one closest to the acquisition time of the second image among the start and end times of the first time search range.

[0069] Optionally, the second time interval is the interval between the collection time of the recommended image and the second time point, where the second time point is the one closest to the collection time of the recommended image between the start and end times of the second time search range.

[0070] In one embodiment provided in the second aspect, the first time search range and the second time search range do not overlap; the time difference between the start time of the first time search range and the start time of the second time search range is less than or equal to the first time difference; the second search results interface also includes the first image.

[0071] Optionally, the first time difference is one month.

[0072] In other words, when the search results are empty due to the user entering an incorrect time entity, the system can recommend relevant images within a preset time range whose collection time is near the time search range corresponding to the time entity.

[0073] Thirdly, this application provides an electronic device, including: a memory, a processor, and a display, wherein,

[0074] The memory is used to store programs;

[0075] The display is used to show the search page;

[0076] The processor, coupled to the memory and the display, is used to execute the program stored in the memory to implement the method described in any of the preceding descriptions.

[0077] Fourthly, this application provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer, is capable of implementing the methods described in any of the above-mentioned embodiments.

[0078] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the above-mentioned embodiments. Attached Figure Description

[0079] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0080] Figure 1A A set of interface diagrams showing the search interface of a mobile phone entering a gallery application, provided as an embodiment of this application;

[0081] Figure 1B This application provides a first set of interface images involved in searching a gallery application, as an embodiment of the present application.

[0082] Figure 1C A first interface image is provided in a gallery application for searching, as an embodiment of this application;

[0083] Figure 1D A set of interface diagrams related to the negative one screen provided in one embodiment of this application;

[0084] Figure 1E An interface diagram related to searching in the negative one screen is provided for one embodiment of this application;

[0085] Figure 2A This is a schematic diagram of the structure of an electronic device provided in yet another embodiment of this application;

[0086] Figure 2B A software structure block diagram of an electronic device provided in yet another embodiment of this application;

[0087] Figure 3A A second set of interface images is provided in a gallery application for searching, as an embodiment of this application;

[0088] Figure 3B A third set of interface images related to searching in a gallery application, provided as an embodiment of this application;

[0089] Figure 3C A fourth set of interface images related to searching in a gallery application, provided as an embodiment of this application;

[0090] Figure 3D A second interface image is provided in a gallery application for searching, as an embodiment of this application;

[0091] Figure 3E A third interface image involved in searching a gallery application is provided as an embodiment of this application;

[0092] Figure 3F A fourth interface diagram related to searching in a gallery application, provided as an embodiment of this application;

[0093] Figure 4 An interaction diagram of an index construction method provided in an embodiment of this application;

[0094] Figure 5 An interactive diagram illustrating a visual media search method provided in an embodiment of this application. Detailed Implementation

[0095] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0096] Hereinafter, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature.

[0097] First, the terminology used in the embodiments of this application will be explained. It should be understood that this explanation is for the purpose of providing a clearer understanding of the embodiments of this application and does not necessarily constitute a limitation on the embodiments of this application.

[0098] Visual media files: refer to images or videos.

[0099] Semantic Subjects: Named Entity Recognition (NER) technology can identify entities with specific meanings in text, such as names of people and places. In this scheme, the entities with specific meanings identified are referred to as semantic subjects.

[0100] Visual content-related and visual content-independent: Visual content refers to the target objects presented in a visual media file and the relationships between these objects. Computer vision enables computers to possess abilities similar to human vision, including perceiving, understanding, analyzing, and interpreting visual content. Currently, Generative Pre-trained Transformer 4 (GPT-4) can support inputting images into the model and outputting human natural language to describe important information in the image.

[0101] In the context of visual media search in this solution, data that requires a model's natural image understanding to retrieve from visual media files is termed "visual content-related." In other words, visual content refers to data that needs to be obtained through a natural image understanding model. Data that is related to visual media files but can be obtained without requiring a model's image understanding capabilities is termed "visual content-independent," such as data that the terminal device can acquire and save when collecting visual media files, including the shooting location, shooting time, name, and file attributes.

[0102] For example, in "Photos taken in Beijing this year", "this year" (shooting time), "Beijing" (shooting location), and "photo" (file attribute) are all data that the terminal device can obtain and save when collecting visual media files. Therefore, "this year", "Beijing", and "photo" are not related to the visual content. In "The sky taken in Beijing this year", "sky" needs to be understood by the model's image understanding ability to understand the images or image frames of the visual media file. Therefore, "sky" is related to the visual content.

[0103] Text semantic vectors are vectors obtained by feeding text into a text encoder, representing the semantic features of the text. The text encoder can employ models such as the Transformer, commonly used in Natural Language Processing (NLP), and this approach is not limited to these. In this approach, the text semantic vector obtained from a sentence is called the sentence semantic vector, the text semantic vector obtained from the semantic subject of the sentence is called the subject semantic vector, and the text semantic vector obtained from the label is called the label semantic vector. The sentence semantic vector of a sentence can fully represent the semantic features of that sentence.

[0104] Visual semantic vectors can be obtained by feeding images or image frames from visual media files into an image encoder. Common convolutional neural network (CNN) models or vision transformer (VIT) models can be used, but this approach does not impose any restrictions.

[0105] Density-based clustering algorithms describe the density of a sample set based on a neighborhood, where (first parameter ε, second parameter MinPts) describes the density of the sample distribution within the neighborhood. The first parameter ε describes the neighborhood radius of a data point; the second parameter MinPts describes the minimum number of data points in the neighborhood of a data point. A representative algorithm is Density-Based Spatial Clustering of Applications with Noise (DBSCAN). DBSCAN is a representative density-based clustering algorithm that can divide regions with sufficiently high density into clusters and can discover clusters of arbitrary shapes in noisy spatial databases.

[0106] Vector similarity: Describes the degree of similarity between two vectors (e.g., between a sentence semantic vector and a visual semantic vector). In this embodiment, the visual media matching the search query can be determined by comparing the similarity between the sentence semantic vector and the visual semantic vector. Generally, vector similarity can be calculated using the cosine similarity formula; however, it can also be calculated using other methods.

[0107] Mobile phones, tablets and other electronic devices use gallery applications to manage visual media files such as pictures and videos taken by users (hereinafter referred to as: visual media).

[0108] Taking a mobile phone as an example, after a photo is taken, the gallery app can obtain and save attributes unrelated to the visual content, such as the shooting location, shooting time, and photo name. Subsequently, while the phone is charging and the screen is off, the photo is input into a natural image understanding model. The model generates and saves tags and visual semantic vectors for the photo. These tags can be seen as attributes related to the visual content of the photo; for example, tags could be "sky," "cat," "dog," etc. Optionally, the gallery app can create an index for the photo based on its attributes and visual semantic vectors. After the index is created, the gallery app can provide users with corresponding search services. For example, users can search for images or videos in the gallery app by entering keywords. For example, users can enter keywords such as "Beijing," "sky," or "National Day" in the search box provided by the gallery app. The gallery app matches the user's entered keywords with the index of visual media such as images and videos in the gallery app to obtain search results. For example, users can enter search terms such as "the sky during last year's National Day" or "fireworks during the 2023 Spring Festival" in the search box provided by the photo library app to search for images or videos. The photo library app matches the user's search terms with its index of images, videos, and other visual media to obtain search results. Search terms, compared to keywords, can more comprehensively describe the images or videos the user wants, thus improving search accuracy.

[0109] The following description, with reference to the accompanying diagram, illustrates the interface involved in the search process of the image gallery application:

[0110] like Figure 1A As shown in (a), the mobile phone can display a main interface 101, which can also be referred to as the desktop. The main interface 101 may include an icon 102 for the gallery application. The mobile phone can receive user actions on the icon 102. For example, this action could be a click. In response to this action, the mobile phone launches the gallery application and displays... Figure 1AInterface 103, as shown in (b) of the figure, can be a photo album interface. Optionally, in response to a user's operation on icon 102, the phone launches the gallery application and may first display the photo gallery interface (not shown in the figure). The photo gallery interface may include a navigation bar and thumbnails of photos in the gallery or a large image of a photo. The navigation bar may include an "Album" option. In response to a user's operation on the "Album" option in the photo gallery interface, the aforementioned interface 103 is displayed.

[0111] like Figure 1A As shown in (b), interface 103 includes multiple albums, including "All Photos" album containing 2023 photos, "Camera" album containing 1502 photos and videos, "Screenshot and Screen Recording" album containing 102 photos and videos, "My Favorites" album containing 48 photos and videos, "One Record, Multiple Views" album containing 34 photos and videos, "Video Editing" album containing 65 videos, "Custom Albums" album containing 57 photos and videos, and "Shared Albums" album containing 100 photos and videos.

[0112] like Figure 1A As shown in (b), the interface 103 may include a search box 104. The mobile phone can receive the user's click on the search box 104, and in response to this operation, the mobile phone can display as shown in Figure (b). Figure 1A Interface 105, shown in (c), can be referred to as a search interface. Interface 105 includes a search box 106 for users to input keywords or search queries. Interface 105 may also include multiple albums categorized according to different dimensions of photos stored on the phone. These dimensions can include time, people, and objects. For example, under the time dimension, the phone categorizes its photos into three time periods: "This Month," "Last Month," and "This Year," resulting in three albums: "This Month," "Last Month," and "This Year." The "This Month" album includes photos or videos taken this month, the "Last Month" album includes photos or videos taken last month, and the "This Year" album includes photos or videos taken this year. As another example, under the people dimension, the phone categorizes its photos by different people, resulting in multiple albums for each person, such as the four albums for different people shown in interface 105. For example, in the dimension of "objects," the phone categorizes its photos into "Landscape," "Animal," "Documents," and "Architecture," resulting in four albums: "Landscape," "Animal," "Documents," and "Architecture." It should be noted that other categorization dimensions can also be used; no specific restrictions are placed here. In interface 105, users can view these categorized albums without entering keywords.

[0113] Optionally, the interface 105 may also include search history 107 and a "clear" option 108. Search history 107 includes keywords or search terms previously entered by the user, such as "flowers," "this year's sky," and "cat." The phone can receive the user's click on the "clear" option 108, and in response to this click, the phone displays... Figure 1A The interface 109 shown in (d) differs from interface 108 in that it does not display search history and the "clear" option.

[0114] Mobile phone receiving user Figure 1A When the user enters the search query "sky photographed in August 2018" in the search box 106 of interface 105 shown in (c), the phone displays the following in response: Figure 1B Interface 110 is shown in (a). Interface 110 may include a search box 306 and partial search results for a search query. The search box 111 includes a search query, and the partial search results may include thumbnails of eight photos. In this example, the search results are found because the phone stores images whose index matches the search query; that is, the phone stores images of the sky taken in August 2018.

[0115] Optionally, interface 110 may also include a "More" option 112 corresponding to the search results. In response to the user's action on the "More" option 112, the phone displays as follows: Figure 1B Interface 113, shown in (b) above, includes all search results for the search query within the gallery application. Users can also swipe up on interface 113 to view images that are not currently displayed.

[0116] Optionally, the interface 110 may also include a prompt message 114, which indicates that the phone has found 239 images based on the search query.

[0117] The search query "sky photographed in August 2018" includes the time entity "August 2018" and the primary keyword "sky". Among the images retrieved via mobile search, the visual content of each image matches the primary keyword "sky", and each image was taken in August 2018.

[0118] Mobile phone receiving user Figure 1A When the user enters the search query "sky photographed in July 2018" in the search box 106 of interface 105 shown in (c), the phone displays the following in response: Figure 1CThe interface 120 shown is a search box 121 and a prompt message 122. The search box 121 contains a search query, and the prompt message 122 indicates to the user that the search results are empty or that no image matches the search query. In this example, no search results were found because the phone does not have an image indexed to match the search query; that is, the phone does not have the sky image collected in July 2018 stored on it.

[0119] In this embodiment, all images taken by the user outside the time search range defined by the time entity are filtered out, and only images taken by the user within the time search range that match the first keyword are displayed to the user. If the user does not find any images matching the first keyword within the time search range, the user is prompted that the search query has no matching results. In practical applications, Figure 1A Interfaces 103, 105, and 109 are included. Figure 1B The interfaces involved and Figure 1C The interface in question is provided by the gallery app.

[0120] Currently, mobile phones also feature a negative one screen and a pull-down search interface. The negative one screen can be understood as the leftmost split screen on an electronic device, used to provide users with search, quick services, and other functions. It can also display notifications to be pushed to the user, such as app notifications, real-time trending searches, segment selections, and trip information. The pull-down search interface is displayed in response to a pull-down action performed by the user on the main screen. This interface provides users with search, app suggestions, and other functions. For example,... Figure 1D Interface 136.

[0121] The following will use the negative one screen as an example. When a user needs to view the negative one screen on their phone, they can swipe the phone screen to make the electronic device display the negative one screen.

[0122] For example, refer to Figure 1D As shown in (a), the mobile phone can receive a first operation performed by the user on interface 130 (which may be referred to as the desktop). For example, this first operation may be as follows: Figure 1D The rightward swipe operation shown in (a) is illustrated. In response to this first operation, the phone can display as shown... Figure 1D The negative one screen 131 shown in (b) may include: a search box 132, quick services 133, default cards 134, recommended cards 134, etc. Quick services 133 may be shortcuts to a page or function of an application, such as: scan, payment code, ride code, etc.; default cards may be: gallery cards, remaining battery cards, etc.; recommended cards may be recommended application cards.

[0123] The mobile phone receives the user's click on the search box 132 on the negative one screen 131 and displays it as follows. Figure 1D Interface 136 is shown in (c). Interface 136 may include: a search box 137. The search box 137 may display trending news titles, such as "Tianjin Marathon". Interface 136 may also include: application suggestions and search history 138. Application suggestions include: icons of suggested applications. Interface 136 may also include: search history 138 and its corresponding "clear" option 139. In response to the user's triggering operation on the "clear" option 139, the search history 138 and the "clear" option 139 will no longer be displayed on interface 136.

[0124] Mobile phone receiving user Figure 1D When the user enters the search query "sky photographed in August 2018" in the search box 137 of interface 136 shown in (c), the following will be displayed in response to the user's input of the search query "sky photographed in August 2018". Figure 1E Interface 140 is shown in (a). Interface 140 includes a search box 141 and a preview area 143. The search box 141 contains a search query, and the preview area 143 contains partial search results from the gallery application for that search query. In response to a user's trigger action on the preview area 143, the user is directed to the search results details interface provided by the gallery application, such as... Figure 1B The interface 113 shown in (b) allows the user to flip through pages to view all search results for the search query.

[0125] Optionally, interface 140 may also include a "Search in Apps" option 144. In response to a user's trigger action on the "Search in Apps" option 144, the phone displays as follows: Figure 1B Interface 110 is shown in (a).

[0126] Optionally, the interface 140 may also display an online search option 142. In response to the user's triggering operation of the online search option 142, the mobile phone displays a search webpage (not shown) and displays the online search results on the search webpage.

[0127] In practical applications, users' descriptions of time are relatively vague, and sometimes they may even make mistakes due to memory confusion. Therefore, the search results from the above search solutions are sometimes difficult to meet users' expectations, resulting in a poor user search experience.

[0128] For example, a user traveled to another city on September 30, 2022, and returned home on October 9, 2022. During this period, the user took many photos. For instance, in August 2023, the user wanted to view the beautiful scenery taken during this trip. Suppose the user entered the search query "scenery taken during the 2022 National Day holiday." If the time filter window is set to "October 1, 2022 to October 7, 2022" based directly on the search query "2022 National Day," then scenery photos taken on September 30, 2022, October 8, 2022, and October 9, 2022 would be filtered out, which clearly does not meet the user's expectations. As another example, in early February 2023, the user wanted to view the beautiful scenery taken during this trip. Suppose the user experienced memory confusion and thought it was still 2022, so the user entered the search query "scenery taken during this year's National Day holiday." This would result in no matching photos being found, which also clearly does not meet the user's expectations.

[0129] To address or partially address the aforementioned technical problems, this application provides a visual media search method. This method can be applied to electronic devices, including mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and other terminal devices.

[0130] For example, Figure 2A A schematic diagram of the structure of electronic device 200 is shown. Electronic device 200 may include processor 210, external memory interface 220, internal memory 221, universal serial bus (USB) interface 230, charging management module 240, power management module 241, battery 242, antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, audio module 270, speaker 270A, receiver 270B, microphone 270C, headphone jack 270D, sensor module 280, button 290, motor 291, indicator 292, camera 293, display screen 294, and subscriber identification module (SIM) card interface 295, etc.

[0131] The sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, a barometric pressure sensor 280C, a magnetic sensor 280D, an accelerometer sensor 280E, a distance sensor 280F, a proximity sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, etc.

[0132] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0133] Processor 210 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0134] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0135] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0136] In some embodiments, the processor 210 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0137] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0138] Electronic device 200 implements display functions through a GPU (Graphics Processing Unit), a display screen 294, and an application processor. The GPU connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0139] Display screen 294 is used to display images, videos, etc. Display screen 294 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 294, where N is a positive integer greater than 1.

[0140] Electronic device 200 can perform shooting functions through ISP, camera 293, video codec, GPU, display screen 294 and application processor.

[0141] Camera 293 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 293, where N is a positive integer greater than 1.

[0142] Video codecs are used to compress or decompress digital video. Electronic device 200 may support one or more video codecs. Thus, electronic device 200 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0143] An NPU (Neural Processing Unit) is a neural network (NN) computing processor that, by borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, rapidly processes input information and can continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0144] The external storage interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200. The external memory card communicates with the processor 210 through the external storage interface 220 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0145] Internal memory 221 can be used to store computer executable program code, which includes instructions. Internal memory 221 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 200 (such as audio data, phonebook, etc.). Furthermore, internal memory 221 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 210 executes various functional applications and data processing of electronic device 200 by running instructions stored in internal memory 221 and / or instructions stored in memory disposed in the processor.

[0146] Figure 2B This is a software structure block diagram of an electronic device 200 according to an embodiment of this application. The software system of the electronic device 200 can adopt a layered architecture, which divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. Taking the Android system as an example, in some embodiments, the Android system can be divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0147] like Figure 2B As shown, the application layer may include applications such as a gallery service module, a search module, a multimodal understanding module, a natural language understanding module, and a camera application.

[0148] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0149] The system library can include multiple functional modules, such as a surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), and 2D graphics engines (e.g., SGL). The media libraries support playback and recording of various common audio and video formats, as well as still image files. The media libraries support multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0150] The kernel layer is the layer between hardware and software.

[0151] The visual media search method provided in this application can be applied to applications such as image library applications and file management applications. The following description uses a mobile phone as an electronic device and images as the visual media, along with accompanying drawings, to illustrate the interface involved in the visual media search method provided in this application.

[0152] Hypothetically, a user took some pictures at the beach on June 29 to July 10, 2023, July 2 to July 8, 2022, and July 20 to July 28, 2020.

[0153] For example, a user performs a search in 2024 for the query "a walk on the beach in July 2023". Figure 3A Interface 310, shown in (a), includes the search query "strolling on the beach in July 2023" in its search box. The time entity in the search query is "July 2023," and the first keyword is "sea." Interface 310 also includes thumbnails of search result images that match the first keyword "sea."

[0154] The search range for the time entity "July 2023" is from July 1, 2023 to July 31, 2023. The image corresponding to thumbnail A was taken (i.e., collected) on June 30, 2023; the image corresponding to thumbnail B was taken on July 1, 2023, both within the search range; and the image corresponding to thumbnail L was taken on July 2, 2023, also within the search range. Although the image corresponding to thumbnail A was taken before July 1, 2023, it is displayed as a search result image because its taking time is closer to that of the image corresponding to thumbnail B.

[0155] Optionally, interface 310 may display only thumbnails of partial search results for the search query. Interface 310 may also include a "More" option 312, which, in response to a user's triggering action on the "More" option 312 on interface 310, displays images such as... Figure 3A Interface 311 shown in (b) is used to display thumbnails of all search results images for the search query “walking on the beach in July 2023”.

[0156] It should be noted that, as an optional implementation, the search results for a search statement that includes a time entity may include images whose shooting time is earlier than the start time of the time search range represented by the time entity, and / or images whose shooting time is later than the end time of the time search range represented by the time entity.

[0157] For example, the thumbnails mentioned above are scaled-down thumbnails of the corresponding images. Furthermore, when the visual media being searched is video, the video thumbnail can be a scaled-down thumbnail of any image frame from that video.

[0158] As can be seen, when a user searches for photos or videos using a search query that includes both a time search range and keywords, the search solution provided in this application can find photos and videos whose collection time is outside the time search range but which match the keywords. This ensures that a series of photos or videos taken by the user before and after the time search range can be retrieved, thereby improving the user's search experience.

[0159] In the above embodiments, the time entity includes a year indicator expressed in numbers, such as 2023. In practical applications, in some scenarios, users will use year indicators expressed in non-numeric forms, such as the year before last, last year, this year, etc.

[0160] In another embodiment, for example, a user performs a search in 2024 for the phrase "walked on the beach last July". Figure 3BThe interface 320 shown includes a search box containing the search term "stroll on the beach last July". The time entity in the search term is "last July", and the first keyword is "sea". Interface 320 also includes thumbnails of search result images that match the first keyword "sea".

[0161] The time entity in the search query contains year indicators in different descriptive ways, such as "2023" and "last year". For the current time of the search, 2024, "2023" and "last year" have the same meaning. Therefore, the search results for the user input "a walk on the beach in July 2023" and "a walk on the beach in July last year" are the same, as shown below. Figure 3A and Figure 3B As shown.

[0162] In another embodiment, the time entity input by the user does not contain a year indicator, such as July or National Day. When the time entity input by the user does not contain a year indicator, relevant images from the most recent N years can be returned to the user. The value of N can be set according to actual needs, and this application does not impose a specific limitation on it. For example, N can be any positive integer greater than 0, such as 5, 10, 20, etc.

[0163] For example, a user performs a search query in 2024 for "a walk on the beach in July". Figure 3C Interface 330, shown in (a), includes the search query "a walk on the beach in July" in its search box. The time entity in the search query is "July," and the first keyword is "sea." Interface 330 also includes thumbnails of search result images that match the first keyword "sea."

[0164] Among them, the image corresponding to thumbnail A was taken on June 30, 2023; the image corresponding to thumbnail B was taken on July 1, 2023; and the image corresponding to thumbnail E was taken on July 21, 2020.

[0165] Optionally, interface 330 may display only thumbnails of partial search results for the search query. Interface 330 may also include a "More" option 332, which, in response to a user's triggering action on the "More" option 332 on interface 330, displays on the phone as follows: Figure 3C Interface 331, shown in (b), is used to display thumbnails of all search results images for the search query "a walk on the beach in July". For example, the image corresponding to thumbnail F was taken on July 6, 2022.

[0166] In other words, when a user enters the search term "stroll on the beach in July", the phone returns images related to "the sea" that the user took around July in the last N years.

[0167] When the user enters a time entity that matches the search expectation, Figure 3A , Figure 3B as well as Figure 3C In the example shown, the search result image matches the "first keyword" in the search statement, and the time of the search result image is either within the search time range corresponding to the time entity in the search statement, or close to the search time range.

[0168] However, in some real-world scenarios, users may enter incorrect time entities, resulting in no search results being matched. This clearly does not meet the user's search expectations and negatively impacts the search experience. To improve the user's search experience, the search solution provided in this application recommends relevant content to the user when no search results are found, thus matching the user's search needs.

[0169] For example, the photos on the phone that match "sea" were all taken by the user between June 29 and July 10, 2023; July 2 and July 8, 2022; and July 20 and July 28, 2020, respectively. If the user executes the search query "stroll on the beach in August 2023" in 2024, such as... Figure 3D The interface 340 shown contains the search query "August 2023 beach walk" in its search box. The time entity in the search query is "August 2023," and the first keyword is "sea." Since there are no matching images of "sea" taken in August 2023 on the phone, no results are found, resulting in an empty search result.

[0170] Optionally, the interface 340 also includes a prompt message 341, which indicates that there are no matching results for "A walk on the beach in August 2023".

[0171] Optionally, the interface 340 may also include recommended content, which may include: thumbnails of recommended images, recommended images that match the "first keyword" in the search query, and each recommended image being taken in July 2023.

[0172] For example, the image corresponding to thumbnail B was taken on July 1, 2023, and the image corresponding to thumbnail L was taken on July 2, 2023. The taking times of these two images are not close to the time search range corresponding to the time entity in the search query, which is from August 1, 2023 to August 31, 2023.

[0173] CombinationFigure 3A and Figure 3D There is no overlap between the time search ranges corresponding to the time entities "August 2023" and "July 2023". The time difference between "August 2023" and "July 2023" is one month. The recommended image set for the search query "August 2023 beach walk" is a subset of the search image set for the search query "July 2023 beach walk". Figure 3D In the example shown, when there are no matching results for "August 2023 beach walk" or the search results for "August 2023 beach walk" are empty, the user is returned an image taken by the user in July 2023 that matches "sea".

[0174] Optionally, when there are no matching results for "August 2023 beach walk" or the search results for "August 2023 beach walk" are empty, recommend to the user images that match "sea" taken by the user in July 2023 and / or images that match "sea" taken by the user in September 2023.

[0175] In this example, since the year indicator "2023" in the time entity of the search query is expressed in numbers, it can be assumed that the year indicator is accurately expressed by the user. Therefore, when making recommendations, the year in which the recommended image was taken is the year indicated by the year indicator, except that the specific time of taking the image is one month before and / or one month after "July".

[0176] In another embodiment, a user executes the search query "walking on the beach in July this year" in January 2024, such as... Figure 3E The interface 350 shown includes a search box containing the search term "stroll on the beach in July this year," where the time entity in the search term is "July this year," and the first keyword is "sea." Since there are no images on the phone that match "sea" and were taken in or around July 2024, no results are found, meaning the search results are empty. Optionally, interface 350 also includes a prompt message 351, which indicates that there are no matching results for "stroll on the beach in July this year."

[0177] Optionally, the interface 350 may also include recommended content, which may include: thumbnails of recommended images, recommended images that match the "first keyword" in the search query, and each recommended image being taken in July 2023.

[0178] For example, the image corresponding to thumbnail B was taken on July 1, 2023, and the image corresponding to thumbnail L was taken on July 2, 2023. The taking time of these two images is not close to the time search range from July 1, 2024 to July 31, 2024, corresponding to the time entity "July of this year" in the search query.

[0179] Figure 3E In the example shown, when there are no matching results for "walking on the beach in July this year" or the search results for "walking on the beach in July this year" are empty, the user is returned a picture taken by the user last July that matches "beach".

[0180] In this example, since the year indicator "this year" in the time entity of the search query is expressed as a non-numeric value, it can be assumed that the year indicator "this year" is expressed incorrectly by the user. Therefore, when making recommendations, the recommended images should be taken in the year before and / or the year after "this year", specifically in July.

[0181] In another embodiment, a user performs a search for the phrase "August beach walk" in 2024. For example... Figure 3F The interface 360 ​​shown includes the search query "August beach walk" in its search box. The time entity in the search query is "August," and the first keyword is "sea." Since there are no images on the phone that match "sea" and were taken in or around August of recent years, no results are found; the search results are empty. Optionally, interface 360 ​​also includes a notification message 361, indicating that there are no matching results for "August beach walk."

[0182] Optionally, the 360 ​​interface may also include recommended content, which may include thumbnails of recommended images and images that match the "first keyword" in the search query.

[0183] For example, thumbnail I was taken on July 21, 2020, thumbnail J was taken on July 5, 2022, and thumbnail K was taken on July 9, 2023. The taking time of these three images is not close to the time search range of August 1 to August 31 corresponding to the time entity "August" in the search statement.

[0184] Optionally, if no matching results are found for the user's input "August beach walk", the phone recommends pictures that match "sea" that the user took in July and / or September of recent years.

[0185] In this example, since the time entity "August" in the search query lacks a year indicator, it can be assumed that the user wants to search for relevant images taken in recent years. Therefore, the recommendation algorithm suggests images matching "sea" taken by the user in the month preceding and / or following "August" in the most recent N years.

[0186] It should be added that, in Figure 3A , Figure 3B as well as Figure 3C In the interface shown, the search result images displayed earlier have a higher match rate with the first keyword in the search query than the search result images displayed later. Figure 3D , Figure 3E as well as Figure 3F In the interface shown, the recommended images displayed earlier match the first keyword in the search query more closely than the recommended images displayed later.

[0187] Optionally, when the first keyword in the search query can be a semantic subject related to visual content (such as the Great Wall or the sky), the visual content of the search result image or the recommended image for the search query matches the first keyword.

[0188] Optionally, when the first keyword in the search query can be a semantic subject unrelated to visual content (e.g., location), the attributes of the search result image or the recommended image for the search query (e.g., location attributes used to indicate the collection location) match the first keyword.

[0189] The following section, with reference to the accompanying diagrams, describes the internal interaction process of the electronic device. Figure 4 An interaction diagram of the index construction stage provided in an embodiment of this application. For example... Figure 4 As shown, the electronic device includes: a gallery service module (i.e., a gallery application) 41, a search module 42, and a multimodal understanding module 43. The index building phase includes the following steps:

[0190] S401. Add and / or modify visual media and their properties.

[0191] In S401 above, for newly added visual media, the gallery application can automatically generate attributes unrelated to the visual content, including but not limited to: capture location, capture time, and visual media name. For example, in the case of a captured video or image, the capture location refers to the capture location, and the capture time refers to the capture time; in the case of a screenshot, the capture location refers to the screenshot location, and the capture time refers to the screenshot time; in the case of a downloaded video or image, the capture location refers to the download location, and the capture time refers to the download time.

[0192] Users can add new visual media by taking photos, downloading, or screenshotting. In addition, users can modify existing visual media. These modifications include, but are not limited to, enhancements, custom naming, and adding watermarks.

[0193] S402, Storing visual media and its properties.

[0194] The image library service module 41 can respond to the above-mentioned add or modify operations by storing visual media and its attributes locally on the electronic device. In practical applications, with the user's authorization, the electronic device can store the locally stored visual media and its attributes in the cloud to reduce the local storage pressure on the electronic device.

[0195] S403, Requesting visual semantic understanding of visual media.

[0196] To improve subsequent search efficiency and shorten search latency, visual semantic understanding of visual media can be performed in advance to obtain and store the visual semantic vectors of the visual media. In this way, the visual semantic vectors can be directly retrieved during subsequent searches.

[0197] Since visual semantic understanding requires a lot of computing resources, in order not to affect the user's use, the above step S403 can be performed when the phone is charging and the screen is off.

[0198] The image library service module 41 can request the multimodal understanding module 43 to perform visual semantic understanding on the newly added or modified visual media, and obtain the visual semantic vector of the visual media.

[0199] Among them, the multimodal understanding module 43 can perform visual semantic understanding of visual media based on the multimodal model to obtain the visual semantic vector of the visual media.

[0200] The multimodal model is used not only for visual semantic understanding of visual media to obtain visual semantic vectors, but also for semantic understanding of search statements to obtain sentence semantic vectors; semantic understanding of rewritten search statements to obtain sentence semantic vectors; semantic understanding of semantic subjects in search statements to obtain subject semantic vectors; and semantic understanding of tags in visual media to obtain tag semantic vectors. The multimodal model can be trained using training samples.

[0201] For example, a multimodal model can specifically be CLIP (Contrastive Language-Image Pre-training, a pre-trained model based on contrastive text-image pairs). The CLIP model can map visual media and text (i.e., search statements, rewritten search statements, semantic subjects, and tags) to a unified vector space to understand the relationships between different modal resources in both text and visual terms, thereby enabling image retrieval. Specifically, in this embodiment, the CLIP model can be used to match visual media files with text.

[0202] Multimodal models can map visual media and text into vectors of the same dimension. That is, the dimension of the visual semantic vector of visual media is the same as the dimension of the semantic vector of text (e.g., the sentence semantic vector of a search query). A multimodal model includes, as mentioned above, an image encoder and a text encoder.

[0203] S404. Returns the visual semantic vector of the visual media.

[0204] The multimodal understanding module 44 returns the visual semantic vector of the visual media to the image library service module 41.

[0205] S405, Store the visual semantic vector of the visual media.

[0206] The gallery service module 41 can store visual semantic vectors of visual media locally.

[0207] S406. Send the attribute information of the visual media and its visual semantic vector.

[0208] For example, the image library service module 41 can store the visual semantic vectors of the visual media returned by the multimodal understanding module 44, and then send the attributes of the visual media and their visual semantic vectors to the search module 42 in batches, so that the search module 42 can build an index of the visual media.

[0209] S407, Building Indexes

[0210] The index of visual media constructed by search module 42 may include: attributes of visual media and visual semantic vectors of visual media.

[0211] Optionally, the electronic device also includes: a natural language understanding module 44 (such as...) Figure 5 (As shown). Figure 5 An interactive diagram of the visual media search method provided in an embodiment of this application is shown.

[0212] The search phase includes the following steps:

[0213] S501, Receive search statement.

[0214] Search queries can include: a time entity and a primary keyword. The primary keyword can be one or more.

[0215] For example, in response to a user's action, such as Figure 1A The search box 106 in interface 105 shown in (c) is used to input and receive search statements.

[0216] S502, Send search query.

[0217] After receiving the search query input by the user, the image library service module 41 sends the search query to the search module 42 to perform the search.

[0218] S503, Request semantic subject recognition for the search statement.

[0219] The search module 42 requests the natural language understanding module 44 to perform semantic subject identification on the search statement to obtain the semantic subjects contained in the search statement. The natural language understanding module 44 performs semantic subject identification based on a natural language understanding model. Specifically, named entity recognition technology can be used to perform semantic subject identification on the search statement to obtain the semantic subjects contained in the search statement. In this embodiment, the semantic subject can also be referred to as an entity.

[0220] Named entity recognition (NAME) technology can identify time-related, location-related, and tag-related semantic entities in search queries. Time-related and location-related semantic entities are semantic entities unrelated to visual content; however, tags in visual media files require natural image understanding by the model to obtain data, therefore, tag-related semantic entities are semantic entities related to visual content. In practical applications, based on experience, several tags that users frequently use can be identified, such as "sky," "cat," "dog," "birthday," and "child." These tags describe visual content. Subsequently, NAME technology can match keywords (or search terms) in the search query with several pre-defined tags to determine whether a keyword belongs to a tag-related semantic entity.

[0221] For example, named entity recognition technology is used to identify the semantic subject of the search query "walking by the sea in July 2023". It is determined that "July 2023" belongs to the semantic subject related to time and "sea" belongs to the semantic subject related to visual content.

[0222] The time entity mentioned above refers to the semantic subject related to time. The first keyword mentioned above includes: semantic subjects related to location and / or semantic subjects related to visual content.

[0223] For ease of explanation, semantic subjects related to visual content will be referred to as visual semantic subjects, and semantic subjects unrelated to visual content will be referred to as non-visual semantic subjects.

[0224] S504, Return to semantic subject.

[0225] The natural language understanding module 44 returns the identified semantic subject to the search module 42.

[0226] S505, Request semantic understanding.

[0227] When the first keyword is a visual semantic subject, the search module 42 can send the visual semantic subject to the multimodal understanding module 43, so that the multimodal understanding module 43 can perform semantic understanding on the visual semantic subject and obtain the subject semantic vector of the visual semantic subject. For example, Figure 3A The search query “walking by the sea in July 2023” shows that “sea” is a visual semantic subject. The multimodal understanding module 43 is requested to perform semantic understanding on “sea” to obtain the subject semantic vector corresponding to “sea”.

[0228] Optionally, non-visual semantic subjects in the search statement are deleted to obtain a rewritten search statement. The rewritten search statement is then sent to the multimodal understanding module 43, which performs semantic understanding on the rewritten search statement to obtain its sentence semantic vector.

[0229] For example: the search term "stroll on the beach in July 2023", where "July 2023" is the semantic subject related to time, can be rewritten as "stroll on the beach".

[0230] In some cases, removing non-visual semantic subjects may leave some redundant stop words. For example, in the search query "sky photographed in Beijing this year," "this year" is a time-related semantic subject, and "Beijing" is a location-related semantic subject. After removing "this year" and "Beijing," the stop word "in" becomes redundant and therefore needs to be removed as well. Specifically, removing non-visual semantic subjects and their related stop words from the search query yields the rewritten search query. For example, the rewritten search query for "sky photographed in Beijing this year" is "photographed sky."

[0231] For details on the semantic understanding process, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0232] S506, Returns the text vector.

[0233] The multimodal understanding module 43 can return the sentence semantic vector of the rewritten search statement and the subject semantic vector corresponding to the visual semantic subject to the search module 42.

[0234] S507. Determine the search time range corresponding to the time entity in the search statement.

[0235] In one alternative implementation, semantic recognition can be performed on time entities, for example, recognizing the year of the time entity.

[0236] Depending on the identification results, the search time range corresponding to the time entity can be determined according to the following different situations:

[0237] (1) Identify the year of the time entity.

[0238] When a time entity includes a year (such as "2023" or "last year"), electronic devices can identify the year through semantic recognition.

[0239] For example, with Figure 3A Taking the search query "strolling on the beach in July 2023" as an example, the time entity is "July 2023", which includes the year "2023". The time search range corresponding to "July 2023" is determined from this year, which is "July 1, 2023 to July 31, 2023".

[0240] For example, with Figure 3B Taking the search query "a walk on the beach last July" as an example, the time entity is "last July", which includes the year "last year". The search range corresponding to "last July" is determined from this year, which is "July 1, 2023 to July 31, 2023".

[0241] For example, taking the time entity "May Day 2023" as an example, the time entity includes the year "2023". The number of time search ranges corresponding to "May Day 2023" is determined from the year. The time search range is "May 1 to May 5, 2023".

[0242] For example, the time entity is "second week of August 2023," which includes the year "2023." The search range for "second week of August 2023" is determined from this year, and this search range is "August 5, 2023 to August 11, 2023." It should be noted that when the time entity includes the year, the search range corresponding to the time entity is singular.

[0243] (2) Years of time entities not identified.

[0244] When the time entity does not include the year, electronic devices cannot identify the year through semantic recognition.

[0245] For example, with Figure 3C The search query "taking a walk on the beach in July as an example" uses "July" as the time entity, which does not include the year. The search range for "July" can be defined as the period from July 1st to July 31st of the most recent N years. For example, the value of N can be set according to actual needs; this application does not impose a specific limitation on it. For example, N can be any positive integer greater than 0, such as 5, 10, 20, etc. When N is 20, it represents searching for relevant images taken in July of the most recent 20 years.

[0246] For example, if the time entity is "May Day" (excluding the year), the search range for "May Day" can be defined as the period from May 1st to May 5th within the last N years. For instance, N = 20 represents searching for relevant images taken during the May Day holiday in the last 20 years.

[0247] In practical applications, there may be situations where, although a time entity does not include a year, the implied year can be determined through deduction. For example, "last week," "last month," and "first week of last month." Taking September 14, 2024 as an example, the implied year for the time entities "last week," "last month," and "first week of last month" is 2024. To address this, a year whitelist can be set up, recording time elements such as "last week" and "last month," and establishing a mapping relationship between "last week," "last month," and 2024. This way, if a time entity is subsequently identified as containing "last week" or "last month," the implied year can be determined as 2024 based on the mapping relationship in the year whitelist. The search range corresponding to the time entity can then be determined based on the implied year. The mapping relationships in the year whitelist can be updated according to the current time. For example, as time goes by, the current time is 2025, and the mapping relationships in the year whitelist need to be updated. That is, the relationship between "last week", "last month" and 2025 is established in the year whitelist, and the mapping relationship between "last week", "last month" and 2024 is deleted.

[0248] It should be noted that when a time entity implicitly contains a year, the time search range corresponding to the time entity is one; when a time entity neither contains nor implies a year, the time search range corresponding to the time entity is N.

[0249] Optionally, before identifying the year of a time entity, it is determined whether the time entity includes time elements recorded in the year whitelist. If the time entity includes time elements recorded in the year whitelist, the year corresponding to that time element in the year whitelist is determined as the implicit year of the time entity. If the time entity does not include time elements recorded in the year whitelist, the year of the time entity is identified.

[0250] S508, Time Generalization.

[0251] In one optional implementation, time generalization (also known as time expansion) can be performed based on the time entity to obtain a generalized time search range. Optionally, time generalization can be performed on the year of the identified time entity and its expression method (which may include numerical expression or non-numerical expression) according to different identification results, following different cases:

[0252] 1. Identify the year of the time entity and identify that the year of the time entity is not a number.

[0253] For example, time entities such as "last July", "this July", etc., include years expressed in non-numeric terms.

[0254] For this special case where years are expressed using non-numeric characters, the generalized time search range can be determined by expanding the year.

[0255] Optionally, the previous year can be extended to obtain a generalized time search range.

[0256] For example, suppose the year entered in the search query is 2024, and the time entity in the search query is "last August". Then, for "last year", the previous year obtained by expansion is 2023. Therefore, the generalized time search range is "August 2023".

[0257] For example, suppose the year entered in the search query is 2024, and the time entity in the search query is "last May Day". Then, for "last year", the previous year obtained by expansion is 2023. Therefore, the generalized time search range is "May Day 2023".

[0258] Among them, the generalization time search range obtained by expanding the previous year can be called the pre-generalization time search range.

[0259] Optionally, extending the year to the next year results in a generalized time search range.

[0260] For example, suppose the year entered in the search query is 2024, and the time entity in the search query is "last August". Then, for "last year", the next year obtained by expansion is 2025. Therefore, the generalized time search range is "August 2025".

[0261] For example, suppose the year entered in the search query is 2024, and the time entity in the search query is "last May Day". Then, for "last year", the next year obtained by expansion is 2025. Therefore, the generalized time search range is "May Day 2025".

[0262] The generalized time search range obtained by extending the next year can be called the post-generalized time search range.

[0263] 2. Identify the year of the time entity and identify that the year of the time entity is expressed in numbers.

[0264] For example, time entities such as "July 2023" and "May Day 2024" both contain year indicators expressed in numbers.

[0265] In this case, a generalized time search range can be obtained by expanding the time search range to include the preceding and / or following time range.

[0266] Optionally, the previous time range of the extended time search range is used to obtain the generalized time search range.

[0267] For example, the time search range corresponding to the time entity "July 2023" is July 2023, and the previous time range corresponding to the time search range July 2023 is June 2023.

[0268] For example, the time search range corresponding to the time entity "May Day 2024" is from May 1st to 5th, 2024, and the time range preceding the time search range from May 1st to 5th, 2024 is from April 26th to 30th, 2024.

[0269] Among them, the generalized time search range obtained by expanding the time search range to the previous time range can be called the pre-generalized time search range.

[0270] Optionally, the next time range of the extended time search range is obtained as a generalized time search range.

[0271] For example, the time search range corresponding to the time entity "July 2023" is July 2023, and the next time range corresponding to the time search range July 2023 is August 2023.

[0272] For example, the time search range corresponding to the time entity "May Day 2024" is from May 1st to 5th, 2024, and the next time range after the time search range of May 1st to 5th, 2024 is from May 6th to 10th, 2024.

[0273] Among them, the generalized time search range obtained by expanding the time search range to the next time range can be called the post-generalized time search range.

[0274] 3. Years for which no time entity was identified.

[0275] When the year of a time entity is not identified, the time search range corresponding to that time entity is N. For each time search range corresponding to that time entity, a generalized time search range can be obtained by expanding the previous and / or next time range of that time search range. For the specific expansion method, please refer to Case 2 above, which will not be detailed here.

[0276] For example, taking the time entity "July" as an example, its corresponding time search range is July in the most recent N years, and the extended generalized time search range is June and / or August in the most recent N years.

[0277] Furthermore, for time entities whose implicit years are determined through the aforementioned year whitelist, a generalized time search range can be obtained by expanding the time search range by the preceding and / or following time range. For specific expansion methods, please refer to Case 2 above, which will not be detailed here.

[0278] Optionally, a generalized whitelist can be created in advance, recording preset time elements such as "New Year's Day," "Spring Festival," "Valentine's Day," and "Qixi Festival." The whitelist also records the mapping relationships between "New Year's Day" and "Spring Festival," and between "Valentine's Day" and "Qixi Festival." The system identifies whether a time entity includes a preset time element from the generalized whitelist. If so, based on the mapping relationships in the whitelist, the target time element corresponding to that preset time element can be determined. The preset time element in the time entity can then be replaced with the target time element, thus obtaining the generalized time search range.

[0279] For example, the time entity "New Year's Day 2023" is replaced with Spring Festival according to the generalized whitelist, resulting in the generalized time search range "Spring Festival 2023".

[0280] For example, the time entity "Valentine's Day 2023" is replaced with Qixi Festival (Chinese Valentine's Day) based on the generalized whitelist, resulting in the generalized time search range "Qixi Festival 2023".

[0281] In practical applications, the execution order of the time generalization steps based on the recognition results of the year and its expression of the time entity and the time generalization steps based on the generalization whitelist can be set according to actual needs, and this application embodiment does not make specific limitations on this.

[0282] In one example, the process can begin by performing time generalization based on the recognition results of the year and its expression for the identified time entity, followed by a step of time generalization based on a generalization whitelist. For instance, taking the time entity "New Year's Day 2024" as an example, time generalization can first be performed based on the recognition results of the year and its expression for the identified time entity, resulting in a generalized time search range of December 29th to 31st, 2023 and January 4th to 6th, 2024. Then, time generalization can be performed based on the generalization whitelist, resulting in a generalized time search range of February 10th to 17th, 2024.

[0283] In another example, the step of generalizing time based on a generalized whitelist can be performed first, and then the step of generalizing time based on the recognition result of the year of the identified time entity and its expression can be performed.

[0284] Optionally, when the time entity specifically points to a particular day (e.g., June 1, 2023), time generalization is not required. This is because when a user can accurately describe the date of a specific day, the user's time expression can be considered very accurate. Therefore, no generalization processing is needed, meaning there is no pre- or post-generalized time search range, and there is no need for subsequent recall based on the pre- or post-generalized time search range.

[0285] Optionally, time generalization is not required when the time entity points to a specific quarter (e.g., spring, summer, etc.).

[0286] S509. Determine the candidate set Q1 that matches the first keyword.

[0287] The candidate set Q1 refers to a set of one or more visual media that match the first keyword, that is, any visual media in the candidate set Q1 matches the first keyword.

[0288] Optionally, when the first keyword is a non-visual semantic subject, it can be determined whether the attributes of the visual media stored in the electronic device match the first keyword.

[0289] In one optional implementation, when the first keyword is a semantic subject related to a location (i.e., a location entity), it is determined whether the location where the visual media was collected is within the geographical range corresponding to the location entity. When the location where the visual media was collected is within the geographical range corresponding to the location entity, it is determined that the attribute of the visual media matches the first keyword, that is, the visual media matches the first keyword.

[0290] For example, the collection location of visual media 1 is **Elementary School in Haidian District, Beijing. The geographical range corresponding to the location entity d is Beijing. Since **Elementary School in Haidian District, Beijing is located in Beijing, the attributes of visual media 1 match the first keyword, that is, visual media 1 matches the first keyword.

[0291] Optionally, when the first keyword is a visual semantic subject, it can be determined whether the visual content of the visual media stored in the electronic device matches the first keyword.

[0292] In one optional implementation, it can be determined whether there is a tag in the visual media that matches the first keyword. If so, it is determined that the visual content of the visual media matches the first keyword. For example, the vector similarity between the text semantic vector of the tag and the text semantic vector of the first keyword can be calculated. If the vector similarity is greater than or equal to a preset similarity threshold, it is determined that the tag matches the first keyword, that is, the visual content of the visual media matches the first keyword.

[0293] In another optional implementation, it can be determined whether the visual semantic vector of the visual media matches the text semantic vector of the first keyword. For example, the vector similarity between the visual semantic vector of the visual media and the text semantic vector of the first keyword can be calculated. If the vector similarity is greater than or equal to a preset similarity threshold, it is determined that the visual semantic vector of the visual media matches the text semantic vector of the first keyword, that is, the visual content of the visual media matches the first keyword.

[0294] Optionally, when there are multiple first keywords, any visual media in the candidate set Q1 can be matched with multiple first keywords.

[0295] Optionally, the set of visual media that matches the rewritten search query can be defined as the candidate set Q1. That is, the candidate set Q1 refers to the set of one or more visual media that match the rewritten search query.

[0296] In this embodiment, any visual media in the candidate set Q1 matches the rewritten search statement. For example, the vector similarity between the visual semantic vector of the visual media and the sentence semantic vector of the rewritten search statement can be calculated. When the vector similarity is greater than a preset similarity threshold, it is determined that the visual content of the visual media matches the rewritten search statement, that is, the visual media matches the rewritten search statement.

[0297] S510. Determine the candidate set Q2 that matches the time search range.

[0298] The candidate set Q2 refers to a set of one or more visual media that match the time search range. In other words, the acquisition time of any visual media in the candidate set Q2 matches the time search range.

[0299] Optionally, when the time search range corresponding to a time entity is 1, matching the time search range means that the collection time falls within that time search range.

[0300] Optionally, when there are N time search ranges corresponding to a time entity, matching a time search range means that the acquisition time falls within one of the N time search ranges.

[0301] S511. Determine the candidate set Q3 that matches the generalization time search range.

[0302] The candidate set Q3 refers to a set of one or more visual media that match the generalization time search range. In other words, the acquisition time of any visual media in the candidate set Q3 matches the generalization time search range.

[0303] Optionally, the generalization time search range is 1. Matching the generalization time search range indicates that the acquisition time falls within the generalization time search range.

[0304] Optionally, the generalization time search range is V, where V is an integer greater than 1. Matching a generalization time search range means that the acquisition time falls within one of the V generalization time search ranges.

[0305] Optionally, when the generalization time search range includes both the pre-generalization time search range and the post-generalization time search range, the candidate set Q3 may include: a candidate subset Q31 that matches the pre-generalization time search range and a candidate subset Q32 that matches the post-generalization time search range.

[0306] Here, the candidate subset Q31 refers to a set of one or more visual media that match the pre-generalization time search range, that is, the acquisition time of any visual media in the candidate set Q31 matches the pre-generalization time search range.

[0307] Optionally, the pre-generalization time search range is 1. Matching the pre-generalization time search range indicates that the acquisition time falls within the generalization time search range.

[0308] Optionally, the pre-generalization time search range is V1, where V1 is an integer greater than 1. Matching the pre-generalization time search range means that the acquisition time falls within one of the V1 pre-generalization time search ranges.

[0309] Here, the candidate subset Q32 refers to a set of one or more visual media that match the post-generalization time search range; that is, the acquisition time of any visual media in the candidate subset Q32 matches the post-generalization time search range.

[0310] Optionally, the post-generalization time search range is 1. Matching the post-generalization time search range indicates that the acquisition time falls within the post-generalization time search range.

[0311] Optionally, the post-generalization time search range is V2, where V2 is an integer greater than 1. Matching the post-generalization time search range means that the acquisition time falls within one of the V2 post-generalization time search ranges.

[0312] To reduce search latency, steps S510 and S511 can be performed simultaneously.

[0313] S512. Determine whether there is an intersection between candidate set Q1 and candidate set Q2.

[0314] When candidate set Q1 and candidate set Q2 have an intersection, execute step S513; when candidate set Q1 and candidate set Q2 have no intersection, execute step S514.

[0315] S513. Use clustering algorithms to determine the search result set.

[0316] The candidate set Q1 includes P candidate visual media.

[0317] Optionally, step S514 above can be implemented using the following steps:

[0318] S11. Using a clustering algorithm, cluster the P candidate visual media according to their acquisition time to obtain K (K≥1) clusters.

[0319] There is no overlap between the K clusters.

[0320] In this way, images from the same series that were collected close together can be grouped into the same cluster.

[0321] For example, a picture taken by a user while walking on the beach on June 30, 2023 and a picture taken by the same user while walking on the beach on July 1, 2023 are taken at close intervals and are therefore grouped into the same cluster using the clustering algorithm described above.

[0322] The clustering algorithms mentioned above may include, but are not limited to: K-Means clustering algorithm, Mean shift clustering algorithm, and density-based clustering algorithm.

[0323] Taking density-based clustering algorithms as an example, in semantic search scenarios involving time entities, setting a fixed first parameter ε and second parameter MinPts is unreasonable. For instance, if a user searches for "photos of trips taken in 2022," the time range is a whole year; while if a user searches for "photos of trips taken during the National Day holiday in 2022," the time range is a few days. The same first parameter ε and second parameter MinPts should not be set for these two cases. To improve the rationality of clustering, the following steps can be used to determine the first parameter ε and the second parameter MinPts:

[0324] (1) Determine the first parameter involved in the density-based clustering algorithm based on the time search range represented by the time entity.

[0325] The first parameter is positively correlated with the duration defined by the time search range.

[0326] The first parameter can be determined based on the duration defined by the time search range. For example, the start time point of the time search range (which can be understood as the start timestamp) is T. start The end time (which can be understood as the end timestamp) is T. end The duration of the time search range is: T end -T start The first parameter can be calculated using the following formula:

[0327] ∈=α*(T end -T start (1)

[0328] Wherein, α is a coefficient that can be adjusted manually, and its size can be set according to actual needs. This application does not impose specific limitations on it.

[0329] (2) Determine the second parameter involved in the density-based clustering algorithm based on the ratio of the number of candidate visual media in the candidate set Q2 to the total search time.

[0330] The second parameter can be calculated using the following formula:

[0331]

[0332] Where M is the number of visual media in the candidate set Q2, M≥1; where N is the number of time search ranges represented by the time entity, N is an integer greater than or equal to 1.

[0333] Wherein, β is a coefficient that can be adjusted manually, and its size can be set according to actual needs. This application does not impose specific limitations on it.

[0334] In this embodiment, the first parameter ∈ and the second parameter MinPts involved in the density-based clustering algorithm are dynamically adjusted according to the duration defined by the time search range represented by the time entity. This ensures the rationality of clustering and improves the accuracy of the final search results.

[0335] S12. Based on whether the collection time range partially overlaps with the time search range, select G (G≥1) target clusters from K clusters.

[0336] Determine the acquisition time range for each of the K (K≥1) clusters. The acquisition time range for each cluster is from the acquisition time of the earliest acquired visual media in that cluster to the acquisition time of the latest acquired visual media in that cluster. The acquisition time ranges of the K clusters do not overlap.

[0337] For example, when there is only one time search range, partial overlap between the collection time range of the target cluster and the time search range means that a part of the collection time range corresponding to the cluster overlaps with the time search range, while the other part does not overlap with the time search range.

[0338] When the time search range is one, the following steps can be used to determine the target cluster:

[0339] S121a. When the start time of the collection time range of the kth cluster in the K clusters is between the first time point and the second time point and the end time is greater than the second time point, it is determined that the collection time range of the kth cluster partially overlaps with the time search range.

[0340] The first time point is the start time of the time search range, and the second time point is the end time of the time search range.

[0341] S122a. When the end time of the collection time range of the kth cluster is between the first time point and the second time point and the start time is less than the first time point, it is determined that the collection time range of the kth cluster partially overlaps with the time search range.

[0342] S123a. When the start time of the collection time range of the kth cluster is less than the first time point and the end time is greater than the second time point, it is determined that the collection time range of the kth cluster partially overlaps with the time search range.

[0343] Where k is an integer, 1≤k≤K.

[0344] S124a. The clusters in the K clusters whose collection time range and time search range partially overlap are identified as the target clusters.

[0345] In this way, G (G≥1) target clusters are selected from K clusters.

[0346] For example, when there are N time search ranges, the partial overlap between the collection time range of the target cluster and the time search range means that a part of the collection time range corresponding to the cluster overlaps with these N time search ranges, while the other part does not overlap with these N time search ranges.

[0347] When the time search range is N, the following steps can be used to determine the target cluster:

[0348] S121b: When the start time of the collection time range of the kth cluster in the K clusters is between the first time point and the second time point and the end time is greater than the second time point, it is determined that the collection time range of the kth cluster partially overlaps with the time search range.

[0349] Wherein, the first time point is the start time of the nth time search range, and the second time point is the end time of the nth time search range. Here, n is an integer, 1≤n≤N.

[0350] S122b: When the end time of the collection time range of the kth cluster is between the first time point and the second time point and the start time is less than the first time point, it is determined that the collection time range of the kth cluster partially overlaps with the nth time search range.

[0351] S123b. When the start time of the collection time range of the kth cluster is less than the first time point and the end time is greater than the second time point, it is determined that the collection time range of the kth cluster partially overlaps with the nth time search range.

[0352] Where k is an integer, 1≤k≤K.

[0353] S124b: Determine the target cluster as the cluster whose collection time range partially overlaps with the nth time search range among the K clusters.

[0354] In this way, G (G≥1) target clusters are selected from K clusters.

[0355] For example, if the time search range is from October 1, 2022 to October 7, 2022, and the collection time range of the cluster is from September 30, 2022 to October 1, 2022, and these two ranges partially overlap (i.e., October 1, 2022), then this cluster is identified as the target cluster.

[0356] S13. Determine the search result set for the search statement based on the intersection J1 of candidate set Q1 and candidate set Q2 and the G target clusters.

[0357] For example, the intersection J1 and the union of the G clusters can be used to determine the search results set for the search query.

[0358] For example, such as Figure 3A , 3B As shown in 3C, the three time entities "July 2023", "July of last year", and "July" all correspond to the time search range "July 1, 2023 to July 31, 2023", such as Figure 3A Interface 310 shown in (a) Figure 3B Interface 320 shown in (a) andFigure 3C In the interface 330 shown in (a), the image corresponding to thumbnail A (taken on June 30, 2023) and the image corresponding to thumbnail B (taken on July 1, 2023) are clustered in the same cluster because their shooting times are close. Since the collection time range of this cluster partially overlaps with the time search range "July 1, 2023 to July 31, 2023", the image corresponding to thumbnail A in this cluster will be recalled as the search result image.

[0359] As can be seen, the technical solution provided by the embodiments of this application can ensure that photos taken before and after the time search range are displayed to the user, ensuring the continuity of search results and improving the user's search experience.

[0360] Optionally, if the target cluster does not exist, the intersection J1 can be used as the search result set for the search query.

[0361] S514. Determine the recommended set.

[0362] Optionally, the intersection J2 of candidate sets Q1 and Q3 can be determined; the intersection J2 can be defined as the recommendation set. The visual media in the intersection J2 match the first keyword and the generalization time search range. The visual media in the recommendation set can be called recommended visual media.

[0363] Optionally, the intersection J2 includes: the intersection J21 of candidate set Q1 and candidate subset Q31 and the intersection J22 of candidate set Q1 and candidate subset Q32. That is, the recommendation set includes: intersection J21 and intersection J22.

[0364] For example, as Figure 3D Taking the search query "A walk on the beach in August 2023" in the search box of the interface 340 as an example, the recommended image displayed on the interface matches the first keyword "sea" and the shooting time is within the generalized time search range of "July 1, 2023 to July 31, 2023" corresponding to the time entity "August 2023".

[0365] For example, as Figure 3E Taking the search query "strolling on the beach in July this year" in the search box of the interface 350 shown as an example, if a user enters this search query in 2024, the recommended images displayed on the interface match the first keyword "sea" and the shooting time is within the generalized time search range "July 1, 2023 to July 31, 2023" corresponding to the time entity "July this year".

[0366] For example, as Figure 3FTaking the search query "August beach walk" in the search box of the 360 ​​interface as an example, if a user enters this search query in 2024, the recommended images displayed on the interface match the first keyword "sea" and the shooting time is within a generalized time search range corresponding to the time entity "August" from "July 1, 2023 to July 31, 2023".

[0367] S514, Visual Media Sort.

[0368] (1) Search results set

[0369] In one optional implementation, the visual media in the search results set can be sorted according to the acquisition time to obtain the display order of the visual media. The visual media displayed earlier were acquired earlier than the visual media displayed later.

[0370] In another optional implementation, the visual media in the search results set can be sorted according to the matching degree between the visual media and the first keyword to obtain the display order of the visual media. The visual media displayed earlier in the order have a higher matching degree with the first keyword than the visual media displayed later in the order.

[0371] In another optional implementation, the overall matching degree of the visual media can be determined based on the matching degree between the visual semantic vector of the visual media and the sentence semantic vector of the rewritten search statement, the matching degree between the visual media and the first keyword, and the matching degree between the collection time of the visual media and the time entity. The visual media in the search results set are then sorted according to their overall matching degree to obtain the display order. The overall matching degree of visual media displayed earlier is greater than that of visual media displayed later. The overall matching degree of the visual media can be obtained by weighted summation of the above multiple matching degrees. The weights used in the weighted summation can be set according to actual needs, and this embodiment does not specifically limit this.

[0372] The matching degree between the acquisition time of the visual media and the time entity is determined based on whether the acquisition time of the visual media falls within the time search range represented by the time entity. For example, when the acquisition time of the visual media falls within the time search range represented by the time entity, the matching degree between the acquisition time of the visual media and the time entity is determined to be a first value (e.g., 1); when the acquisition time of the visual media does not fall within the time search range represented by the time entity, the matching degree between the acquisition time of the visual media and the time entity is determined to be a second value (e.g., 0.8).

[0373] (2) Recommendation Collection

[0374] In one optional implementation, the visual media in the recommended set can be sorted according to the acquisition time to obtain the display order of the visual media. The visual media displayed earlier were acquired earlier than the visual media displayed later.

[0375] In another alternative implementation, the recommendation set includes an intersection J21 and an intersection J22. The display order of visual media in intersection J21 precedes the display order of visual media in intersection J22.

[0376] For multiple visual media within intersection J21 or multiple visual media within intersection J22, one of the following methods can be used to achieve sorting:

[0377] Method 1: Based on the matching degree between multiple visual media and the first keyword, sort the visual media in the set to obtain the display order. The visual media displayed earlier in the order have a higher matching degree with the first keyword than the visual media displayed later in the order.

[0378] Method 2: Determine the overall matching degree of visual media based on the matching degree between the visual semantic vector of the visual media and the sentence semantic vector of the rewritten search statement, the matching degree between the visual media and the first keyword, and the matching degree between the collection time of the visual media and the time entity. Then, rank the multiple visual media according to their overall matching degree to obtain the display order. Visual media displayed earlier have a higher overall matching degree than visual media displayed later.

[0379] S516. Send search results set / recommendation set.

[0380] The search module 42 returns the search results set / recommendation set to the image library service module 41.

[0381] For example, the search module 42 can send the search result set, the display order of each visual medium in the search result set, and the indication information used to indicate that the search result set is a search result to the gallery service module 41.

[0382] For example, the search module 42 can send the recommendation set, the display order of each visual medium in the recommendation set, and the indication information used to indicate that the recommendation set is recommended content to the gallery service module 41.

[0383] S517, Display.

[0384] For example, when the search module 42 sends a search result set, the image library service module 41 can display thumbnails of the visual media on the interface according to the search result set and the display order of each visual media in the search result set, such as... Figure 3A ,3B , Figure 3C As shown.

[0385] For example, when the search module 42 sends a recommendation set, the image library service module 41 displays a prompt message on the interface indicating that the search query has no matching results, and displays thumbnails of the visual media on the interface according to the recommendation set and the display order of each visual media in the recommendation set, such as... Figure 3D , 3E , Figure 3F As shown.

[0386] exist Figure 5 In the illustrated embodiment, steps S508 and S511 are executed before step S512. To reduce search latency, steps S510 and S511 can be executed simultaneously.

[0387] Optionally, step S512 can be executed first, and after determining that candidate set Q1 and candidate set Q2 have an intersection, steps S508 and S511 can be executed. In this application, steps S510 and S511 are not executed simultaneously, but sequentially.

[0388] It should be added that, in order to better protect user privacy and security and to meet the principle of minimizing user data, we try to avoid reporting user data to the cloud side and complete the entire search process on the client side.

[0389] Furthermore, this application provides an electronic device, including: a memory, a processor, and a display, wherein the memory is used to store a program; the processor is coupled to the memory and the display, and is used to execute the program stored in the memory to implement the above-described visual media search method.

[0390] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer, can implement one or more steps of any of the above-described visual media search methods.

[0391] Computer-readable storage media can be non-transitory computer-readable storage media.

[0392] Another embodiment of this application provides a computer program product containing instructions. When the computer program product is executed by a computer, it can perform one or more steps of any of the methods described above.

[0393] In this embodiment, the electronic device, computer-readable storage medium, and computer program product are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0394] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0395] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0396] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0397] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0398] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A visual media search method, applicable to electronic devices, characterized in that, include: Display a first interface, which includes a search box; After receiving a search operation on the search statement entered into the search box, a first candidate set, a second candidate set, and a third candidate set are determined. The search statement includes a time entity and a first keyword. The time entity corresponds to a first time search range. The first candidate set is a set of visual media that matches the first keyword. The second candidate set is a set of visual media that matches the first time search range. The third candidate set is a set of visual media that matches the second time search range. The second time search range is obtained by expanding upon the time entity. Determine whether the first candidate set and the second candidate set have any intersection; When the first candidate set and the second candidate set have an intersection, a search results set is displayed, which includes the intersection of the first candidate set and the second candidate set; When the first candidate set and the second candidate set have no intersection, a recommended set is displayed, which includes the intersection of the first candidate set and the third candidate set.

2. The method according to claim 1, characterized in that, Also includes: After receiving a search operation on the search statement entered into the search box, the time entity and the first keyword are semantically identified; When the year of the time entity is identified, the first time search range is determined based on the year; When the year of the time entity is not identified, N first time search ranges are determined based on the most recent N years, where N is greater than 1 and is an integer.

3. The method according to claim 2, characterized in that, Also includes: When the year is not expressed as a number, the previous year and / or the next year are extended to obtain the second time search range; When the year is expressed numerically or the year in which the time entity is not identified, the second time search range is obtained by expanding the previous and / or next time range of the first time search range. The time difference between the start time of the preceding time range and the start time of the first time search range is less than or equal to the first time difference; the time difference between the start time of the following time range and the start time of the first time search range is less than or equal to the second time difference.

4. The method according to any one of claims 1 to 3, characterized in that, The first candidate set includes multiple first visual media files; The method further includes: The multiple first visual media files are clustered according to their acquisition time to obtain K clusters, where K ≥ 1 and is an integer. The target cluster is determined based on whether the collection time range partially overlaps with the first time search range; The search results set is determined based on the target cluster.

5. The method according to claim 4, characterized in that, The start time of the collection time range for each cluster is the collection time of the earliest collected visual media file in that cluster, and the end time is the collection time of the latest collected visual media file in that cluster. There is no overlap in the collection time ranges of the K clusters.

6. The method according to claim 4, characterized in that, The start time of the first time search range is the first time point, and the end time is the second time point; The step of determining the target cluster based on whether the collection time range partially overlaps with the first time search range includes: When the start time of the collection time range of the kth cluster in the K clusters is between the first time point and the second time point and the end time is greater than the second time point, it is determined that the collection time range of the kth cluster partially overlaps with the first time search range. When the end time of the collection time range of the kth cluster is between the first time point and the second time point and the start time is less than the first time point, it is determined that the collection time range of the kth cluster partially overlaps with the first time search range. When the start time of the collection time range of the kth cluster is less than the first time point and the end time is greater than the second time point, it is determined that the collection time range of the kth cluster partially overlaps with the first time search range; k is an integer, 1≤k≤K; Clusters whose collection time range partially overlaps with the first time search range among the K clusters are identified as target clusters.

7. The method according to any one of claims 1 to 3, characterized in that, The second and third candidate sets were determined simultaneously.

8. A visual media search method, applicable to electronic devices, characterized in that, include: After receiving the first search statement input by the user, the first search results interface is displayed. The first search statement includes a first time entity and a first keyword. The first time entity corresponds to a first time search range. The first search results interface includes a first image and a second image. The first image and the second image match the first keyword. The acquisition time of the first image is within the first time search range. The acquisition time of the second image is not within the first time search range. The interval between the acquisition time of the second image and the first time range is a first time interval. After receiving the second search statement input by the user, a second search result interface is displayed. The second search statement includes the second time entity and the first keyword. The second time entity corresponds to the second time search range. The second search result interface includes a first prompt message and a recommended image. The first prompt message is used to indicate that the recommended image is recommended content. The recommended image matches the first keyword. The interval between the collection time of the recommended image and the second time search range is the second time interval. The second time interval is greater than the first time interval.

9. The method according to claim 8, characterized in that, The second search results interface also includes a second prompt message, which indicates that the second search query has no matching results.

10. The method according to claim 8, characterized in that, The first time interval is the interval between the acquisition time of the second image and the first time point, and the first time point is the one closest to the acquisition time of the second image among the start and end times of the first time search range; The second time interval is the interval between the time of collection of the recommended image and the second time point, where the second time point is the one closest to the collection time of the recommended image between the start and end times of the second time search range.

11. The method according to claim 8, characterized in that, The first time search range and the second time search range do not overlap; The time difference between the start time of the first time search range and the start time of the second time search range is less than or equal to the first time difference; The second search results interface also includes the first image.

12. The method according to claim 11, characterized in that, The first time difference is 1 month.

13. An electronic device, characterized in that, include: Memory, processor, and display, among which, The memory is used to store programs; The display is used to show the search page; The processor, coupled to the memory and the display, is used to execute the program stored in the memory to implement the method of any one of claims 1 to 12.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a computer, it can implement the method of any one of claims 1 to 12.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 12.

Images