Method, device and equipment for searching images in photo album and storage medium

Abstract

The invention provides a photo album image searching method and device, equipment and a storage medium, and the method comprises the steps: obtaining description information related to a to-be-searched image, transmitting the description information to a semantic recognition model server, receiving a first image tag from the semantic recognition model server, and transmitting the first image tag to a server; a target image label matched with the first image label is determined from second image labels corresponding to each image in the photo album, the second image labels are obtained by recognizing each image through a preset target recognition model, a to-be-searched image is determined based on the target image labels, and the to-be-searched image is searched according to the to-be-searched image. The image recognition model can be deployed on the end side to recognize the tag of the image, the semantic recognition model server is called to recognize the search intention to obtain the corresponding tag, and the to-be-searched image is further obtained through tag matching, so that model miniaturization processing does not need to be performed on the end side to reduce the model performance; therefore, the image searching effect in the photo album is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for searching images in an album. Background Art

[0002] Smart image search refers to accurately searching for the desired images from the album with the help of various models, which is a further improvement of the functions of album applications. In related technologies, large polymorphic models are usually miniaturized for deployment on the terminal side, and the search function for images in the album is realized through natural language search. However, the miniaturization of large polymorphic models will inevitably lead to a decrease in model capabilities, thus affecting the effect of image search in the album. Summary of the invention

[0003] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0004] The first aspect of the present disclosure proposes a method for searching images in an album, including: obtaining description information related to an image to be searched, and sending the description information to a semantic recognition model server; receiving a first image tag from the semantic recognition model server; determining a target image tag that matches the first image tag from second image tags corresponding to each image in the album, wherein the second image tag is obtained by recognizing each image using a preset target recognition model; and determining the image to be searched based on the target image tag.

[0005] The second aspect of the present disclosure proposes an image search device in an album, including: an acquisition module, used to acquire description information related to an image to be searched, and send the description information to a semantic recognition model server; a receiving module, used to receive a first image tag from the semantic recognition model server; a matching module, used to determine a target image tag that matches the first image tag from the second image tag corresponding to each image in the album, wherein the second image tag is obtained by recognizing each image using a preset target recognition model; and a determination module, used to determine the image to be searched based on the target image tag.

[0006] The third aspect of the present disclosure proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for searching images in an album proposed in the first aspect of the present disclosure is implemented.

[0007] The fourth aspect embodiment of the present disclosure proposes a non-temporary computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method for searching images in an album proposed in the first aspect embodiment of the present disclosure is implemented.

[0008] The fifth aspect of the present disclosure provides a computer program product. When an instruction processor in the computer program product executes, the method for searching images in an album as provided in the first aspect of the present disclosure is performed.

[0009] In this embodiment, by obtaining descriptive information related to the image to be searched, and sending the descriptive information to the semantic recognition model server, and receiving the first image tag from the semantic recognition model server, and determining the target image tag matching the first image tag from the second image tag corresponding to each image in the album, wherein the second image tag is obtained by a preset target recognition model for each image recognition, and the image to be searched is determined based on the target image tag, the image recognition model can be deployed on the terminal side to identify the label of the image, and the semantic recognition model server is called to identify the search intent to obtain the corresponding label, and the image to be searched is further obtained through label matching. Therefore, there is no need to perform model miniaturization processing on the terminal side to reduce model performance, thereby improving the effect of image search in the album.

[0010] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0012] Figure 1 is a flowchart of a method for searching images in an album proposed in an embodiment of the present disclosure;

[0013] Figure 2 This is a schematic diagram of the photo album APP interface on the terminal device side proposed in the embodiment of the present disclosure;

[0014] Figure 3 is a schematic diagram of a process of identifying a second image label by a target recognition model proposed in an embodiment of the present disclosure;

[0015] Figure 4 is a schematic diagram of a process of identifying a first image label by a GPT model proposed in an embodiment of the present disclosure;

[0016] Figure 5 is a flowchart of a method for searching images in an album according to another embodiment of the present disclosure;

[0017] Figure 6 is a flowchart of a method for searching images in an album according to another embodiment of the present disclosure;

[0018] Figure 7 is a schematic diagram of an image search architecture in an album proposed in an embodiment of the present disclosure;

[0019] Figure 8 is a structural schematic diagram of an image search device in an album proposed in another embodiment of the present disclosure;

[0020] Fig. 9 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0021] Embodiments of the present disclosure are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present disclosure, and are not to be construed as limitations of the present disclosure. On the contrary, the embodiments of the present disclosure include all changes, modifications, and equivalents that fall within the spirit and connotation of the appended claims.

[0022] It should be noted that the executor of the method for searching images in an album of this embodiment may be a device for searching images in an album, which may be implemented by software and / or hardware, and which may be configured in an electronic device, which may include but is not limited to a terminal, a server, etc.

[0023] Figure 1 FIG. 1 is a flow chart of a method for searching images in an album according to an embodiment of the present disclosure. Figure 1 As shown, the method for searching images in the album includes:

[0024] S101: Obtain description information related to the image to be searched, and send the description information to the semantic recognition model server.

[0025] The image search method in the photo album of the embodiment of the present disclosure can be executed on the terminal device side, such as a smart phone, a tablet computer, a pad, and any other device that supports a photo album application (Application, APP), wherein the photo album APP can be a system photo album of the terminal device, or it can also be a photo album of a third-party application, without limitation.

[0026] The image that the user needs to search in the album is called a to-be-searched image, and the to-be-searched image can be any image without limitation.

[0027] The description information is used to describe the content of the image to be searched, and the description information may be in the form of text or voice, for example, and there is no limitation on this. Figure 2 is a schematic diagram of the photo album APP interface on the terminal device side proposed in the embodiment of the present disclosure, such as Figure 2As shown, in this embodiment, an interface for the image search function can be configured in the album APP, and the user can enter the description information of the image to be searched in the interface.

[0028] For example, when a user needs to search for panda images in an album (i.e., images to be searched), the user can enter description information in the interface, such as text "Search for panda images" or voice "Search for panda images". In some embodiments, when the description information entered by the user is voice, the terminal device can also use a preset voice-to-text function to convert the voice into corresponding text for subsequent processing, which is not limited.

[0029] In order to accurately search for the image to be searched in the album, the disclosed embodiment may also predetermine the image tag corresponding to each image in the album. The image tag may be called a second image tag. The second image tag is used to describe the content of each image. The second image tag may be, for example, divided into: people, buildings, scenery (which may be further subdivided into mountains, deserts, grasslands, etc.), animals (which may be further subdivided into fish, cats, birds, etc.), vehicles, and any other possible tags, without limitation. It is understandable that each image in the album may correspond to one or more second image tags, without limitation.

[0030] For example, the album stores image 1 (including a house), image 2 (including a person), image 3 (including a panda), image 4 (including a pet cat), image 5 (including a panda), etc. In this case, it can be determined that the second image label corresponding to image 1 is, for example, "building", the second image label corresponding to image 2 is "person", the second image label corresponding to image 3 is "panda", the second image label corresponding to image 4 is "pet cat", and the second image label corresponding to image 5 is "panda".

[0031] In some embodiments, a target recognition model for identifying image labels may be pre-trained and deployed on the terminal device side. Figure 3 is a schematic diagram of a process of identifying a second image label by a target recognition model proposed in an embodiment of the present disclosure, such as Figure 3 As shown, the target recognition model of this embodiment can be, for example, a YOLOV8 model, and the YOLOV8 model can be deployed in a neural network forward computing framework configured in the terminal device, that is, an NCNN framework. During the application initialization process, the terminal device can call the target recognition model to identify each image in the album to output the corresponding second image label; further, the terminal device stores the mapping relationship between each image and the second image label in a local database. Among them, this embodiment does not specifically limit the training process of the target recognition model.

[0032] Figure 4is a schematic diagram of the process of the GPT model proposed in the embodiment of the present disclosure to identify the first image label, such as Figure 4 As shown, after the description information is obtained, the terminal device can further send the description information (text / voice information) to the semantic recognition model server.

[0033] Specifically, the terminal device can send the text / voice description information to the semantic recognition model server via an http request. The semantic recognition model server is configured with a semantic recognition model, which can be, for example, a generative pre-trained Transformer model (GPT). The GPT model is used to semantically understand the text / voice information input by the user to output the image label expected by the user, which is called the first image label. The GPT model can be, for example, a trained model, or it can call an existing GPT model, such as a chat GPT model, without limitation.

[0034] For example, the terminal device can send the description information "panda image" text to the GPT model server via an http request, and the GPT model can perform semantic understanding of "search panda image", and the first image label output can be "panda".

[0035] In some embodiments, in order to make the first image tag output by the GPT model more relevant to the image in the album, this embodiment may also send a second image tag set as a reference while sending the description information to the GPT model server, such as Figure 4 As shown, that is, the description information (text / voice information) and the second image tag set (tag set) are sent to the GPT model server. Thus, the GPT model identifies the first image tag in the description information with reference to the second image tag set, so the first image tag is more closely related to the second image tag, which can improve the accuracy of the subsequent matching process.

[0036] In practical applications, there may be a situation where the description information input by the user cannot be recognized as the first image label by the GPT model. For example, the description information "Search for panda images" input by the above user may be recognized by the GPT model as the user wants panda images. Therefore, the GPT model may directly obtain panda images from the entire network for output, resulting in the inability to output the first image label. In view of this, the present embodiment can process the description information so that the processed information can clearly allow the GPT model to fully understand the meaning of the description information input by the user and output the expected first image label. Specifically, the present embodiment can add preset prompt information to the description information, such as "what is the label" and "what is the keyword". For example, after adding the prompt information to the above description information, the form is, for example, "what is the label in "search for panda images". Thus, the GPT model can understand the requirement to output the label, that is, the first image label. Among them, for the GPT model, the prompt information can be continuously adjusted so that the GPT model can better understand the needs of the user.

[0037] It is understandable that, in the process of searching for the image to be searched, the user can input description information multiple times to communicate with the GPT model, so as to gradually obtain more accurate first image labels.

[0038] In this embodiment, since the second image tag is identified by using the locally deployed target recognition model, and the description information is identified by the GPT model server, the process does not need to upload the image to the GPT model server, and is therefore more private.

[0039] S102: Receive a first image tag from a semantic recognition model server.

[0040] like Figure 4 As shown, after the GPT model recognizes and obtains the first image tag, the first image tag is further transmitted to the terminal device side as a return. In this case, the terminal device of this embodiment can receive the first image tag.

[0041] S103: Determine a target image tag that matches the first image tag from the second image tags corresponding to each image in the album.

[0042] Furthermore, the terminal device of this embodiment can obtain the second image tag corresponding to each image from the local database, and compare the second image tag corresponding to each image with the first image tag respectively to determine the tag in the second image tag that matches the first image tag. This tag is called the target image tag, that is, the target image tag belongs to one or more second image tags.

[0043] In some embodiments, a second image tag that is the same as or similar to a first image tag may be referred to as a target image tag.

[0044] For example, the first image label determined above is "panda", and the second image label matching the first image label, such as the second image label "panda" corresponding to image 3, the second image label "pet cat" corresponding to image 4, and the second image label "panda" corresponding to image 5, then "panda" in image 3, "pet cat" in image 4, and "panda" in image 5 are all used as target image labels.

[0045] It can be understood that when the second image label and the first image label of each image are multiple labels, all the second image labels of each image need to be matched with the first image labels respectively, and all the second image labels that are successfully matched are used as the target image labels.

[0046] S104: Determine an image to be searched based on the target image tag.

[0047] That is to say, the image to be searched is determined according to the target image label.

[0048] In some embodiments, the images corresponding to the target image tags may be used as the images to be searched, that is, the images to be searched in this embodiment include image 3, image 4, and image 5, or some images may be selected as the images to be searched, and there is no limitation to this.

[0049] Furthermore, the terminal device performs subsequent applications on the image to be processed, for example, displaying the image to be searched.

[0050] In some other embodiments, the photo album of the terminal device may also support a picture search function.

[0051] Specifically, the user can select an image in the album as a reference image. In this case, the terminal device can call the target recognition model to identify the reference image to determine its image label, which is called the reference image label; further, the terminal device can match the reference image label with the second image label respectively to determine the target image related to the reference image, that is, to use the image corresponding to the second image label that matches the reference image label as the target image.

[0052] For example, if the reference image selected by the user is image 3, the reference image label includes "panda"; further, a second image label that matches the reference image label is determined, that is, the second image label needs to be the same as or similar to "panda", then this implementation can determine that the "panda" of image 5 matches, so the image corresponding to the label (i.e., image 5) can be used as the target image.

[0053] In this embodiment, by obtaining descriptive information related to the image to be searched, and sending the descriptive information to the semantic recognition model server, and receiving the first image tag from the semantic recognition model server, and determining the target image tag matching the first image tag from the second image tag corresponding to each image in the album, wherein the second image tag is obtained by a preset target recognition model for each image recognition, and the image to be searched is determined based on the target image tag, the image recognition model can be deployed on the terminal side to identify the label of the image, and the semantic recognition model server is called to identify the search intent to obtain the corresponding label, and the image to be searched is further obtained through label matching. Therefore, there is no need to perform model miniaturization processing on the terminal side to reduce model performance, thereby improving the effect of image search in the album.

[0054] Figure 5 FIG. 1 is a flow chart of a method for searching images in an album according to another embodiment of the present disclosure. Figure 1 As shown, the method for searching images in the album includes:

[0055] S501: Determine the images to be identified in the album and the available computing resources based on preset rules.

[0056] In practical applications, different target recognition models have different recognition capabilities (recognition speed, recognition accuracy, etc.), and the computing resources (cup resources, memory resources) required in the recognition process are also different. It is understandable that the target recognition model with stronger recognition capabilities requires more computing resources.

[0057] In view of this, in order to improve the accuracy of second image label recognition, the disclosed embodiment may deploy at least two models for identifying the second image label on the terminal device side, each model is called a candidate recognition model, and the embodiment may select a candidate recognition model as a target recognition model according to the computing resources of the terminal device. For example, the candidate recognition models of the present embodiment include YOLOV8-S and YOLOV8-N, wherein YOLOV8-S has better recognition capability (recognition speed, recognition accuracy) than YOLOV8-N, and requires more computing resources.

[0058] In the process of identifying the second image tag of each image, this embodiment first determines the image to be identified in the album and the available computing resources based on preset rules.

[0059] Among them, the images in the album that need to be labeled are called images to be identified, and the available computing resources represent the computing resources that can be used by the target recognition model. Among them, the images to be identified and the available computing resources can be determined by triggering multiple rules, and there is no restriction on this.

[0060] In some embodiments, a time interval (e.g., 24 hours) for determining the images to be identified in the album may be preset, that is, the terminal device determines the images to be identified every 24 hours. In this case, the terminal device may monitor the current time, and when the current time reaches the preset time interval, determine the images without identification tags in the album as the images to be identified, and determine the available computing resources of the terminal device at the current time.

[0061] In some other embodiments, when a terminal device detects that a new image is added to an album, the terminal device determines the new image as an image to be identified, and determines the available computing resources of the terminal device at the time when the new image is added.

[0062] S502: Determine a candidate recognition model that can be satisfied by available computing resources and has the best recognition capability as a target recognition model.

[0063] For example, if the currently available computing resources only meet the requirements of YOLOV8-N, YOLOV8-N will be used as the target recognition model; for another example, if the available computing resources meet the computing resources of both YOLOV8-N and YOLOV8-S models, the candidate recognition model with the best recognition capability (ie, YOLOV8-S) will be selected as the target recognition model.

[0064] S503: Using the target recognition model to recognize a second image label of the image to be recognized.

[0065] Furthermore, the target recognition model is used to recognize the second image label of the image to be recognized. The specific recognition process is shown in the above embodiment and will not be described again here.

[0066] Therefore, in this embodiment, when the available computing resources meet the demand, a target recognition model with stronger recognition ability is selected to recognize the second image label, thereby improving the recognition effect of the second image label.

[0067] S504: Obtain description information related to the image to be searched, and send the description information to the semantic recognition model server.

[0068] S505: Receive a first image tag from a semantic recognition model server.

[0069] S506: Determine a target image tag that matches the first image tag from the second image tags corresponding to each image in the album.

[0070] S507: Determine an image to be searched based on the target image tag.

[0071] The specific description of S504-S507 refers to the above embodiment and will not be repeated here.

[0072] In this embodiment, by obtaining description information related to the image to be searched, and sending the description information to the semantic recognition model server, and receiving the first image tag from the semantic recognition model server, and determining the target image tag that matches the first image tag from the second image tag corresponding to each image in the album, wherein the second image tag is obtained by the preset target recognition model for each image recognition, and the image to be searched is determined based on the target image tag, it is possible to deploy an image recognition model on the terminal side to identify the label of the image, and call the semantic recognition model server to identify the search intent to obtain the corresponding label, and further obtain the image to be searched through label matching, so there is no need to perform model miniaturization processing on the terminal side to reduce model performance, thereby improving the effect of image search in the album. In addition, in this embodiment, when the available computing resources meet the requirements, a target recognition model with stronger recognition ability is selected to recognize the second image tag, thereby improving the recognition effect of the second image tag.

[0073] Figure 6 FIG. 1 is a flow chart of a method for searching images in an album according to another embodiment of the present disclosure. Figure 6 As shown, the method for searching images in the album includes:

[0074] S601: Obtain description information related to the image to be searched, and send the description information to the semantic recognition model server.

[0075] S602: Receive a first image tag from a semantic recognition model server.

[0076] The specific description of S601 - S602 refers to the above embodiment and will not be repeated here.

[0077] S603: Calculate the similarity between the second image label and the first image label respectively, and sort the second image labels according to the similarity.

[0078] In the embodiment of the present disclosure, in the process of determining the target image label matching the first image label, the similarity between the second image label and the first image label of each image may be calculated respectively, for example, the cosine distance is calculated as the similarity.

[0079] Furthermore, the second image tags are sorted according to the similarity.

[0080] In some embodiments, the images may be sorted in descending order of similarity. For example, if the first image label is "panda", the second image label sorting result of this implementation is, for example, "panda" (image 3), "panda" (image 5), "pet cat", "person" (image 2), "building" (image 1).

[0081] S604: Select a preset number of second image tags with high similarity as target image tags according to the sorting result.

[0082] That is to say, a preset number of second image labels with the highest similarity are selected as target image labels, wherein the preset number can be flexibly set according to the actual application scenario. For example, the preset number in this implementation is 3, and the first three second image labels with the highest similarity are taken as target image labels, namely: "panda" (image 3), "panda" (image 5), and "pet cat".

[0083] S605: Sort the images corresponding to the target image labels based on the confidences corresponding to the target image labels.

[0084] In some embodiments, the object recognition model may also output the confidence level corresponding to the second image label when recognizing the second image label of each image.

[0085] In the process of determining the image to be searched based on the target image label, the present embodiment can sort the images corresponding to the target image label according to the confidence level corresponding to the target image label, for example, sorting them in descending order of confidence level.

[0086] For example, among the above three target image labels, the confidence of "panda" (image 3) is 90%, the confidence of "panda" (image 5) is 85%, and the confidence of "pet cat" (image 4) is 70%. The ranking results are: image 3, image 5, image 4.

[0087] In some embodiments, when there are target image labels with the same confidence, the order of the images corresponding to the target image labels with the same confidence is determined according to the number of labels of the images corresponding to the target image labels with the same confidence. It is understandable that the image with fewer labels is more accurate, so its order is higher.

[0088] For example, the confidence of "panda" (image 3) and "panda" (image 5) is 90%, and the confidence of "pet cat" (image 4) is 70%. In this case, the present embodiment can determine the number of second image labels of image 3 and image 5. For example, if image 3 only contains pandas, then its second image label is 1; for another example, if image 5 contains pandas and people, then its second image labels are 2; in this case, the sorting result can be determined as: image 3, image 5, image 4.

[0089] S606: Selecting a preset number of images with high confidence as images to be searched based on the ranking results.

[0090] Furthermore, the sorting result selects a preset number of images with high confidence as the images to be searched, for example, selects two images with the highest confidence as the images to be searched.

[0091] Therefore, the embodiment of the present disclosure can determine the target image label based on the similarity, and further determine the image to be searched based on the execution degree, thereby improving the accuracy of searching the image to be searched.

[0092] In this embodiment, by obtaining description information related to the image to be searched, and sending the description information to the semantic recognition model server, and receiving the first image tag from the semantic recognition model server, and determining the target image tag that matches the first image tag from the second image tag corresponding to each image in the album, wherein the second image tag is obtained by the preset target recognition model for each image recognition, and the image to be searched is determined based on the target image tag, it is possible to deploy an image recognition model on the terminal side to identify the label of the image, and call the semantic recognition model server to identify the search intent to obtain the corresponding label, and further obtain the image to be searched through label matching, so there is no need to perform model miniaturization processing on the terminal side to reduce model performance, thereby improving the effect of image search in the album. In addition, the disclosed embodiment can determine the target image tag based on similarity, and further determine the image to be searched based on the execution degree, thereby improving the accuracy of the search of the image to be searched.

[0093] In one specific example, Figure 7 is a schematic diagram of an image search architecture in an album proposed in an embodiment of the present disclosure, such as Figure 7 As shown, the image search architecture in the photo album of this embodiment includes the photo album APP, database, target recognition model and GPT model server on the terminal device side. Among them, the photo album APP can provide functional modules such as image preview, history query, image search, text and speech recognition, matching algorithm, and interactive interface design, and the target recognition model and GPT model can interact with the photo album APP.

[0094] like Figure 8 As shown, the image search device 80 in the photo album includes:

[0095] The acquisition module 801 is used to acquire description information related to the image to be searched, and send the description information to the semantic recognition model server;

[0096] A receiving module 802 is used to receive a first image tag from a semantic recognition model server;

[0097] A matching module 803 is used to determine a target image tag that matches the first image tag from the second image tag corresponding to each image in the album, wherein the second image tag is obtained by a preset target recognition model for each image; and

[0098] The determination module 804 is used to determine the image to be searched based on the target image tag.

[0099] In some embodiments, the target recognition model is one of at least two candidate recognition models, and each candidate recognition model has different recognition capabilities and required computing resources. The device 80 also includes a recognition module, which is specifically used to: determine the image to be recognized in the album and the available computing resources based on preset rules; determine the candidate recognition model with the best recognition capability that can be satisfied by the available computing resources as the target recognition model; and use the target recognition model to recognize the second image label of the image to be recognized.

[0100] In some embodiments, the identification module is specifically used to: in response to the current time reaching a preset time interval, determine the images with unrecognized tags in the album as images to be identified, and determine the available computing resources at the current time; or in response to the addition of new images to the album, determine the newly added images as images to be identified, and determine the available computing resources at the time of the newly added images.

[0101] In some embodiments, the determination module 804 is specifically configured to: sort the images corresponding to the target image tags based on the confidences corresponding to the target image tags; and select a preset number of images with high confidences as images to be searched based on the sorting results.

[0102] In some embodiments, the determination module 804 is specifically used to: when there are target image labels with the same confidence, determine the order of images corresponding to the target image labels with the same confidence according to the number of labels of the images corresponding to the target image labels with the same confidence.

[0103] In some embodiments, the matching module 803 is specifically used to: respectively calculate the similarity between the second image tag and the first image tag, and sort the second image tags according to the similarity; and select a preset number of second image tags with high similarity as target image tags according to the sorting result.

[0104] In some embodiments, the acquisition module 801 is specifically used to: send the description information and the second image tag set to the semantic recognition model server, so that the semantic recognition model recognizes the first image tag in the description information with reference to the second image tag set.

[0105] In some embodiments, the device further comprises a processing module, which is used to: add preset prompt information to the description information, wherein the prompt information is used to prompt the semantic recognition model to recognize the first image label.

[0106] In some embodiments, the apparatus 80 further includes a search module, specifically configured to: determine a reference image tag corresponding to the reference image; and match the reference image tag with the second image tag respectively to determine a target image related to the reference image.

[0107] In this embodiment, by obtaining descriptive information related to the image to be searched, and sending the descriptive information to the semantic recognition model server, and receiving the first image tag from the semantic recognition model server, and determining the target image tag matching the first image tag from the second image tag corresponding to each image in the album, wherein the second image tag is obtained by a preset target recognition model for each image recognition, and the image to be searched is determined based on the target image tag, the image recognition model can be deployed on the terminal side to identify the label of the image, and the semantic recognition model server is called to identify the search intent to obtain the corresponding label, and the image to be searched is further obtained through label matching. Therefore, there is no need to perform model miniaturization processing on the terminal side to reduce model performance, thereby improving the effect of image search in the album.

[0108] In order to implement the above embodiments, the present disclosure further proposes a non-temporary computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for searching images in an album as proposed in the above embodiments of the present disclosure.

[0109] In order to implement the above embodiments, the present disclosure further proposes a computer program product. When the instruction processor in the computer program product is executed, the method for searching images in the album proposed in the above embodiments of the present disclosure is executed.

[0110] Fig. 9 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. Fig. 9 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0111] like Fig. 9 As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0112] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnection (PCI) bus.

[0113] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0114] The memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Fig. 9 not shown, usually called a "hard drive"). Although Fig. 9 Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a compact disc read only memory (CD-ROM), a digital versatile disc read only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the various embodiments of the present disclosure.

[0115] A program / utility 50 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described in the present disclosure.

[0116] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via an input / output (I / O) interface 22. In addition, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0117] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the methods mentioned in the above embodiments.

[0118] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0119] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0120] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present disclosure belong.

[0121] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0122] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0123] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0124] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0125] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present disclosure. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present disclosure.

Claims

1. A method for searching images in an album, characterized in that: include: Obtaining description information related to the image to be searched, and sending the description information to the semantic recognition model server; Receiving a first image tag from the semantic recognition model server; Determine a target image tag matching the first image tag from the second image tag corresponding to each image in the album, wherein the second image tag is obtained by recognizing each image using a preset target recognition model; as well as The image to be searched is determined based on the target image tag.

2. The method according to claim 1, characterized in that The target recognition model is one of at least two candidate recognition models, and each of the candidate recognition models has different recognition capabilities and required computing resources. Before determining the target image tag matching the first image tag from the second image tag corresponding to each image in the album, the method further includes: Determine the images to be identified in the album and the available computing resources based on preset rules; Determine a candidate recognition model that can be satisfied by the available computing resources and has the best recognition capability as the target recognition model; and The object recognition model is used to recognize a second image label of the image to be recognized.

3. The method according to claim 2, characterized in that The determining the image to be identified in the album and the currently available computing resources based on the preset rules includes: In response to the current time reaching a preset time interval, determining an image in the album without a recognized tag as the image to be recognized, and determining available computing resources at the current time; or In response to the newly added image in the album, the newly added image is determined as the image to be identified, and available computing resources at the time of the newly added image are determined.

4. The method according to claim 1, characterized in that: The determining the image to be searched based on the target image tag comprises: sorting the images corresponding to the target image labels based on the confidences corresponding to the target image labels; and A preset number of images with high confidence levels are selected as the images to be searched based on the ranking results.

5. The method according to claim 4, characterized in that The step of sorting the images corresponding to the target image labels based on the confidences corresponding to the target image labels comprises: In the case where there are target image labels with the same confidence, the order of the images corresponding to the target image labels with the same confidence is determined according to the number of labels of the images corresponding to the target image labels with the same confidence.

6. The method according to claim 1, characterized in that The step of determining a target image tag matching the first image tag from the second image tag corresponding to each image in the album comprises: respectively calculating similarities between the second image tags and the first image tags, and sorting the second image tags according to the similarities; and A preset number of second image tags with high similarity are selected as the target image tags according to the sorting result.

7. The method according to claim 1, characterized in that The sending of the description information to the semantic recognition model server includes: The description information and the second image tag set are sent to a semantic recognition model server, so that the semantic recognition model identifies the first image tag in the description information with reference to the second image tag set.

8. The method according to claim 1, characterized in that Before sending the description information to the semantic recognition model server, the method further includes: Preset prompt information is added to the description information, wherein the prompt information is used to prompt the semantic recognition model to recognize the first image label.

9. The method according to claim 1, characterized in that: The method further comprises: determining a reference image label corresponding to the reference image; and The reference image labels are matched with the second image labels respectively to determine a target image related to the reference image.

10. A device for searching images in a photo album, characterized in that: The device comprises: An acquisition module, used for acquiring description information related to the image to be searched, and sending the description information to the semantic recognition model server; A receiving module, configured to receive a first image tag from the semantic recognition model server; a matching module, configured to determine a target image tag matching the first image tag from a second image tag corresponding to each image in the album, wherein the second image tag is obtained by recognizing each image using a preset target recognition model; and A determination module is used to determine the image to be searched based on the target image tag.

11. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for searching images in an album as described in any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for searching images in an album as described in any one of claims 1 to 9 is implemented.

13. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the method for searching images in an album as described in any one of claims 1 to 9.

Images