Label determination method and apparatus

Abstract

The application discloses a label determination method and device, and belongs to the technical field of image processing. The method comprises the following steps: in response to a first input for a first image in an image set, determining a first label of the first image; determining an image similarity between the first image and a second image in the image set according to characteristic information of the first image and characteristic information of the second image; and determining a second label of the second image according to the image similarity and the first label.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to a label determination method and apparatus. Background Technology

[0002] Nowadays, in image collections on devices such as mobile phones and computers, each image in the collection is usually tagged to facilitate the classification, searching, and batch management of images in the collection.

[0003] However, the tags automatically assigned to images by devices are usually quite simple. For example, they might be tagged as plants, vegetables, clothing, or buildings based on image type. This kind of tagging doesn't support more granular searching and management by users. If users want to add more specific and personalized tags, they need to manually tag each image, which is too complex and costly. Summary of the Invention

[0004] The purpose of this application is to provide a label determination method and apparatus that can solve the technical problem of high cost in existing image labeling methods.

[0005] In a first aspect, embodiments of this application provide a label determination method, the method comprising:

[0006] In response to a first input to a first image in a set of images, a first label for the first image is determined;

[0007] Based on the characteristic information of the first image and the characteristic information of the second image in the image set, the image similarity between the first image and the second image is determined;

[0008] A second label for the second image is determined based on the image similarity and the first label.

[0009] Secondly, embodiments of this application provide a label determining device, the device comprising:

[0010] The first determining module is configured to determine a first label of the first image in response to a first input to a first image in the image set;

[0011] The second determining module is used to determine the image similarity between the first image and the second image based on the characteristic information of the first image and the characteristic information of the second image in the image set.

[0012] The third determining module is used to determine the second label of the second image based on the image similarity and the first label.

[0013] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the method provided in the first aspect.

[0014] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method provided in the first aspect.

[0015] Fifthly, embodiments of this application provide a chip, which includes a processor and a communication interface, the communication interface and the processor being coupled together, the processor being used to run programs or instructions to implement the method provided in the first aspect.

[0016] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method as provided in the first aspect.

[0017] In the label determination method and apparatus of this application, a first label for the first image can be determined in response to a first input to a first image in an image set; the image similarity between the first image and the second image can be determined based on the characteristic information of the first image and the characteristic information of the second image in the image set; and a second label for the second image can be determined based on the image similarity and the first label. In this way, a user can manually label only a portion of the images in the image set, and then the electronic device can automatically propagate the label of the first image to similar second images based on the image similarity by calculating the image similarity between different images in the image set, reducing the workload of manual labeling and lowering the cost of image labeling. Attached Figure Description

[0018] Figure 1 This is a schematic flowchart of a label determination method provided in one embodiment of this application;

[0019] Figure 2 This is one of the schematic diagrams of the interface display of an electronic device provided in one embodiment of this application;

[0020] Figure 3 This is a second schematic diagram of the interface display of an electronic device provided in one embodiment of this application;

[0021] Figure 4 This is the third schematic diagram of the interface display of an electronic device provided in one embodiment of this application;

[0022] Figure 5 This is the fourth schematic diagram of the interface display of an electronic device provided in one embodiment of this application;

[0023] Figure 6 This is the fifth schematic diagram of the interface display of an electronic device provided in one embodiment of this application;

[0024] Figure 7 This is a schematic diagram of the structure of a tag determining device provided in another embodiment of this application;

[0025] Figure 8 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application;

[0026] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0028] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0029] To address the aforementioned technical problems, this application provides a label determination method. The label determination method provided by this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0030] like Figure 1 As shown, Figure 1 This is a flowchart illustrating a label determination method provided in one embodiment of this application. This application provides a label determination method, which may include:

[0031] S101, in response to a first input for a first image in the image set, determine a first label for the first image;

[0032] In this embodiment, the first image can be any image from the image set. This first image can be an image with a tag or an image without a tag. The user can select the first image from the image set on the electronic device and then input a first input to determine the first tag of the first image. For example, the user can input the tag "beach" for the first image, and the electronic device will use "beach" as the first tag for this first image. The image set refers to a group of images managed uniformly on the electronic device. For example, the image set could be a photo album application on the electronic device, or an image set application within a file manager on the electronic device.

[0033] Specifically, the first input can be a user's touch input to the electronic device, such as a touch click, a swipe, or other operation. The first input can also be an input operation performed by the user using a mouse, remote control, keyboard, or other input device connected to the first electronic device. Optionally, the first input can indicate a control in a notification message. Unless otherwise specified, the input to the electronic device described below can be a touch click, a swipe, or other operation, or an input operation performed by the user using a mouse, remote control, keyboard, or other input device connected to the electronic device.

[0034] In some embodiments, the first input includes a first sub-input and a second sub-input;

[0035] S101 includes:

[0036] Display the first image;

[0037] In response to a first sub-input to the first image, a label editing control for the first image is displayed;

[0038] In response to a second sub-input to the label editing control, the phrase corresponding to the second sub-input is determined as the first label.

[0039] In this embodiment, the first input can be divided into a first sub-input and a second sub-input. Before entering the first input for the first image in the image set, the user can first open the image set and select the image identifier of the first image in the image set. In this way, the first image will be displayed in the image set.

[0040] After a first image is displayed in the image set of the electronic device, the user can perform a first sub-input on the first image. This first sub-input is used to access the tag editing page for that first image, which includes at least one tag editing control. For example, the first sub-input could be a double-click on the first image, or it could be a long press on the first image for 3 seconds. In response to the first sub-input, the electronic device displays at least one tag editing control on the tag editing page.

[0041] After the label editing control is displayed on the electronic device, the user can enter the label editing control as a phrase to be labeled on the first image, which is the second sub-input. The electronic device responds to the second sub-input and adds the phrase entered by the user as the first label to the first image in the image set.

[0042] As an alternative implementation method, Figure 2 The display includes a first image 201, on which the user can make a first sub-input, and then the display interface of the electronic device will display something like this. Figure 3 The label editing interface shown in the diagram generates and displays a thumbnail 301 of the first image, and displays three label input controls 302 (Label 1, Label 2, and Label 3), a confirmation control 303, and a cancellation control 304. The user can input a phrase in at least one label input control via a second sub-input, and then confirm the input via the confirmation control. Each phrase entered in a label input control is then identified as a first label. If the user cancels the input via the cancellation control, the display interface of the electronic device switches back to normal. Figure 2 The first image shown.

[0043] As another alternative implementation method, Figure 4 The first image to be displayed also includes a label editing control 401. The user can make initial input to the label editing control, and the electronic device's display interface will then redirect to the first image. Figure 3 The label editing interface shown.

[0044] In this way, users can manually tag images in an image collection, allowing them to precisely control the tag content and ensure that the tags are highly relevant to the image content, thereby improving the accuracy of photo classification and retrieval.

[0045] In some embodiments, S101 includes:

[0046] Identify the N third elements included in the first image, where N is a positive integer greater than 1;

[0047] Determine P fourth elements from the N third elements, where P is a positive integer less than or equal to N, and the fourth elements are those elements whose pixel points are greater than a pixel point threshold.

[0048] The first tag corresponding to each of the fourth elements is determined based on the content of the fourth element.

[0049] In this embodiment, elements in an image refer to identifiable independent objects or salient regions, such as people, animals, plants, and items. After acquiring the first image, the electronic device can identify N third elements contained in the image using image recognition technology. Then, among these N third elements, P fourth elements with larger areas in the first image are further selected based on the number of pixels contained in each element. The electronic device can then identify the main content of these P fourth elements and assign a corresponding first label to each fourth element based on its main content. The first label of the P fourth elements is also the first label of the first image.

[0050] For example, such as Figure 5 As shown, N is 5 and P is 3. Figure 5 The system includes a first image 501. In response to a first input to the first image in the image set, the electronic device can use a pre-set image recognition model to extract content from the first image, identifying and detecting five third elements. The electronic device can then sort these five third elements from largest to smallest based on their area proportion or the number of pixels they contain in the first image, selecting the three fourth elements with the largest area proportion or the largest number of pixels: green beans, corn, and chili peppers. A bounding box is then displayed around each fourth element. Based on the content of these three fourth elements, the electronic device can label the green beans, corn, and chili peppers as "vegetables."

[0051] Furthermore, after the electronic device automatically assigns a first tag to each fourth element, the user can modify the content of the first tag of any fourth element. For example... Figure 6 As shown, after the electronic device labels the three fourth elements—green beans 603, corn 602, and chili peppers 601—as "vegetables," the user can change the first label of the chili pepper from "vegetable" to "favorite food." Furthermore, when the user edits this first label, the corresponding body box will be highlighted, indicating which element's label content is being modified.

[0052] Using the methods described above, electronic devices can accurately identify and label important elements that occupy significant areas in images, thereby improving the efficiency of labeling.

[0053] S102, determine the image similarity between the first image and the second image based on the characteristic information of the first image and the characteristic information of the second image in the image set;

[0054] In this embodiment, after adding a first label to the first image through the first input, any unlabeled second image in the image set can be obtained. The image similarity between the first and second images is calculated based on the characteristic information of the first and second images, and a determination is made based on the image similarity to add a label to the second image. Here, characteristic information refers to key data or features in the image used to describe and distinguish its content. Characteristic information may include the image's vectors, color histogram, texture, and keypoints.

[0055] For example, when the feature information is an image vector, the cosine similarity between the first image and the second image can be calculated based on the vectors of the first image and the second image. Specifically, feature vector A of the first image and feature vector B of the second image in the image set can be extracted, and then the cosine similarity between the two vectors A and B can be calculated using the following cosine similarity formula, which serves as the image similarity between the two images.

[0056] The formula for cosine similarity is as follows:

[0057]

[0058] Where Similarity is the image similarity, which is any constant between 0 and 1. The larger the Similarity, the more similar the first image and the second image are. A represents the vector of the first image, B represents the vector of the second image, and A·B represents the dot product of vectors A and B.

[0059] S103, determine the second label of the second image based on the image similarity and the first label.

[0060] In this embodiment, after determining the first label of the first image and the image similarity between the first and second images, the second label of the second image can be determined based on the image similarity and the first label. Since a higher image similarity indicates that the content of the first and second images is more similar, a high image similarity means that the content of the first and second images is not significantly different. Therefore, the first label of the first image can be used to mark the second image, and the first label can also be used as the second label of the second image.

[0061] In this embodiment, in response to a first input to a first image in an image set, a first label for the first image can be determined; based on the vector of the first image and the vector of a second image in the image set, an image similarity between the first image and the second image can be determined; and based on the image similarity and the first label, a second label for the second image can be determined. In this way, a user can manually label only a subset of images in the image set. Then, the electronic device can automatically propagate the label of the first image to similar second images based on the image similarity by calculating the image similarity between different images in the image set, reducing the workload of manual labeling and lowering the cost of image labeling.

[0062] In some embodiments, the image similarity includes a first similarity, which is the overall similarity between the first image and the second image;

[0063] In the case of determining a second label based on a first similarity in image similarity and a first label of a first image, step S103 includes:

[0064] If the first similarity between the first image and the second image is greater than or equal to the first similarity threshold, the first label is determined as the second label.

[0065] In this embodiment, the first image can be converted into a first vector and the second image can be converted into a second vector. Then, the cosine similarity between the first vector and the second vector is calculated as the first similarity between the first image and the second image. Then, the first similarity is compared with a preset first similarity threshold.

[0066] If the first similarity is greater than or equal to the first similarity threshold, the second image is considered to have a high overall similarity to the first image, and all first labels on the first image can be assigned to the second image. If the first similarity is less than the first similarity threshold, the second image is considered to have a low overall similarity to the first image, and the first label cannot be added to the second image based on the first similarity.

[0067] For example, the first vector of the first image is [0.8, 0.2, 0.3]. If the second vector of a second image, "Photo 1," is [0.1, -0.2, 0.2], and the vector of another second image, "Photo 2," is [0.9, 0.3, 0.8], the first similarity threshold can be set to 0.8. The cosine similarity between the first image and "Photo 1" is calculated to be 0.23, which does not exceed the set first similarity threshold, so no label is applied. If the cosine similarity between the first image and "Photo 2" is 0.93, which exceeds the set first similarity threshold of 0.8, then the system automatically assigns the first label of the first image to the second image, "Photo 2."

[0068] Using the above method, users do not need to label every image in the image set. The labels can be automatically propagated in the image set based on the overall similarity between the images, thereby improving the coverage of the labels.

[0069] In some embodiments, the image similarity includes a second similarity, which is the similarity between elements in the first image and elements in the second image;

[0070] In the case where the second label is determined based on the second similarity in the image similarity and the first label of the first image, S103 includes:

[0071] If the first similarity between the first image and the second image is less than the first similarity threshold and greater than or equal to the second similarity threshold, T first elements in the first image and T second elements in the second image are determined, wherein each first element corresponds to a first label and T is a positive integer;

[0072] If the second similarity between the i-th first element and the j-th second element is greater than the third similarity threshold, the first label corresponding to the i-th first element is determined as the second label of the j-th second element, where i and j are both positive integers less than or equal to T.

[0073] In this embodiment, if the first similarity is less than the first similarity threshold, it can be considered that the second image and the first image have low overall similarity, and therefore the first label cannot be added to the second image based on the first similarity.

[0074] A second similarity threshold can be preset, which is lower than a first similarity threshold. If the first similarity threshold is lower than the second similarity threshold, the first image and the second image are considered to have extremely low similarity, and the second image cannot be labeled. However, if the first similarity threshold is lower than the first similarity threshold but greater than or equal to the second similarity threshold, the first image and the second image are considered to have moderate overall similarity, and the similarity between elements in the first image and elements in the second image can be further compared.

[0075] Specifically, if the first similarity between the first image and the second image is less than a first similarity threshold and greater than or equal to a second similarity threshold, then the content in the first image can be extracted, and T first elements included in the first image can be identified and detected. Similarly, T second elements included in the second image can be identified and detected. For example, the top T elements with the most pixels in all elements of the first image can be selected as first elements, and the top T elements with the most pixels in all elements of the second image can be selected as second elements.

[0076] Then, the second similarity between each first element and each second element is compared, and a third similarity threshold is preset. If the second similarity between the i-th first element and the j-th second element is greater than the third similarity threshold, then the similarity between the i-th first element and the j-th second element is considered to be high, and the first label corresponding to the i-th first element is added to the j-th second element; if the second similarity between the first element and the second element is not greater than the third similarity threshold, then no label is applied to the second image.

[0077] The above method enables refined label propagation. In other words, when the overall similarity between images is low but still within a certain range, more detailed label propagation can be achieved by comparing the similarity of specific elements in the images.

[0078] In some embodiments, after S101, the method further includes:

[0079] In response to a user’s first input to a first image in the image set, a first label weight corresponding to a first label of the first image is determined;

[0080] Construct a first mapping relationship between the first image, the first label, and the weight of the first label, and generate a first inverted record based on the first mapping relationship;

[0081] A tag library for the image set is constructed based on the first inverted record.

[0082] In this embodiment, after receiving the user's first input on the first image, not only can the first label of the first image be determined, but also the first label weight corresponding to the first label can be determined. The first label weight is used to characterize the degree of correlation between the first label and the first image.

[0083] After determining the first label and its weight, a first mapping relationship can be constructed between the first image, the first label, and its weight, and a first inverted index can be generated based on this mapping relationship. Then, all inverted indexes can be stored in a pre-built empty database, thereby generating a label library for the image set.

[0084] In this embodiment, the inverted index is a data structure used to quickly find all images corresponding to a given tag and their weights. When a user adds tags to images in the image set, the electronic device generates an inverted index based on these tags and tag weights, and creates an inverted index for the tags. The inverted index is a data structure used to map tags to a list of images containing those tags. Therefore, the user can quickly find images containing a given tag within the image set by using the inverted index.

[0085] In some embodiments, after determining the first label weight corresponding to the first label of the first image, the method further includes:

[0086] The product of the first label weight and the image similarity is determined as the second label weight corresponding to the second label in the second image;

[0087] Construct a second mapping relationship between the second image, the second label, and the weights of the second label, and generate a second inverted record based on the second mapping relationship;

[0088] Add the second inverted record to the tag library of the image set.

[0089] In this embodiment, the first label on the first image is propagated to the second image. The product of the image similarity between the first and second images and the first label weight can also be used as the second label weight. Similarly, a second mapping relationship can be constructed between the second image, the second label, and the second label weight to generate a second inverted index, which is then stored in the label library.

[0090] As an optional embodiment, since the first label weight is used to characterize the relevance between the first label and the first image, and the first label is input by the user, it can be assumed that the relevance between the first label and the first image is high. Therefore, the first label weight can be set to 1. Taking "beach" as an example, the first label weight of "beach" on the first image is 1. If the image similarity between the first image and the second image is 0.93, then the second label weight is 0.93. The first inverted record is "first image--beach--1", and the second inverted record is "second image--beach--0.93".

[0091] By using the above method, the weight of the label in the second image can be automatically determined by multiplying the image similarity and the weight of the first label, making it more accurately reflect the relevance of the label in different images. At the same time, more inverted records are automatically generated and updated to ensure that the label library can be quickly updated and respond to retrieval requests, improving the system's search efficiency.

[0092] In some embodiments, after determining the second label of the second image based on the image similarity and the first label, the method further includes:

[0093] Receive user input regarding search intent;

[0094] Based on the search intent information, a search is performed in the image set to obtain multiple third images from the image set;

[0095] Display the multiple third images.

[0096] In this embodiment, the retrieval intent information is used to characterize the user's retrieval needs for images in the image set. The retrieval intent information can be voice information, text information, or action information. The electronic device can search the image set based on the user's input retrieval intent information to find multiple third images that match the retrieval intent information. These multiple third images are then displayed.

[0097] In this way, users can efficiently search within an image collection and quickly find images that interest them.

[0098] In some embodiments, the step of searching the image set according to the search intent information to obtain multiple third images in the image set includes:

[0099] Determine the search text corresponding to the search intent information;

[0100] If a third tag matching the search text exists in the image set, the third image corresponding to the third tag is retrieved from the image set.

[0101] If no third tag matching the search text exists in the image set, multiple third images from the image set are obtained through at least one of vector recall and keyword recall:

[0102] The keyword recall is as follows: determine the third tag that matches the keywords in the search text, and retrieve the third image corresponding to the third tag;

[0103] The vector recall is as follows: obtain the image vectors converted from each image in the image set, and the text vectors converted from the search text, retrieve the first image vector whose third similarity to the text vector is greater than the fourth similarity threshold, and determine the image corresponding to the first image vector as the third image.

[0104] In this embodiment, if the retrieval intent information is text information, it can be directly determined as retrieval text; if the retrieval intent information is voice information or other forms of information, it can be converted into retrieval text in text form.

[0105] Then, all tags in the tag library of the image set can be traversed to determine if there is a tag that matches the search text. If a third tag that matches the search text exists, all images corresponding to that third tag can be used as the recalled third images. Specifically, if the search text is a phrase, and the tag contains a phrase that is exactly the same as the search text or a synonym of the search text, then the tag can be considered to match the search text. Otherwise, it can be considered that there is no third tag that matches the search text.

[0106] If no third tag matching the search text exists in the image set, the electronic device can retrieve the third image from the image set using at least one of vector recall and keyword recall methods.

[0107] In the keyword retrieval process, keyword extraction techniques can be used to extract keywords from the search text. For example, if the search text is "find photos of Yangyang running on the beach," then the keywords "Yangyang," "beach," and "running" can be extracted. These keywords are then input into the search database for inverted index retrieval. When a third tag matching any keyword exists, all images corresponding to that tag can be retrieved as the third image.

[0108] In addition, the image set can contain a pre-defined vector library, which includes the image vectors converted from each image in the image set. During vector recall, the search text can be converted into a text vector, ensuring that the text vector and the image vector have the same dimension. Then, the text vector can be matched with each image vector to obtain a third similarity between the text vector and each image vector. If the third similarity between an image vector and a text vector is greater than a pre-defined fourth similarity threshold, the image corresponding to that image vector can be used as the third image for recall.

[0109] The above method combines tag matching, keyword recall, and vector recall to ensure accurate retrieval of the images the user needs. Even without tags that perfectly match the search text, the system can still find relevant images through keyword and vector matching, increasing the flexibility of the search.

[0110] In some embodiments, displaying the plurality of third images includes:

[0111] The number of first tags corresponding to the third tags of each third image and the first value corresponding to each third image are determined; wherein, if the image vector corresponding to the third image matches the text vector, the first value corresponding to the third image is 1, and if the image vector corresponding to the third image does not match the text vector, the first value corresponding to the third image is 0.

[0112] The sum of the number of first tags and the first value of each of the third images is determined as the recall number of the third image;

[0113] The multiple third images are sorted in descending order of their recall counts to obtain a sorting result;

[0114] The multiple third images are displayed according to the sorting results.

[0115] In this embodiment, after retrieving multiple third images through at least one of keyword recall and vector recall, these multiple third images can be displayed. To enable users to find the desired image more accurately and quickly, the display order of the multiple third images can be arranged based on the relevance of each third image to the search text.

[0116] Specifically, since the search text may include multiple keywords, each keyword will be searched separately in the tag library. For any third image, the number of its corresponding tag matches and the number of keywords in the search text is the number of the first tag of the third image corresponding to the third image, which is the first recall number of the third image through keyword retrieval. For example, if the third image can be recalled by the three keywords in the search text, then the number of the first tag of the third image corresponding to the third image is 3, and the first recall number of the third image through keyword retrieval is 3.

[0117] Furthermore, for any third image, if the third similarity between the image vector of the third image and the text vector of the search text is greater than a pre-set fourth similarity threshold, the third image can be recalled through vector recall. In this case, the first value corresponding to the third image is 1; otherwise, the first value corresponding to the third image is 0.

[0118] Therefore, the sum of the number of first labels and the first value of the third image is the total recall of the third image; after determining the total recall of each third image, multiple third images can be sorted and displayed in descending order of recall.

[0119] For example, taking the search text "Yangyang running on the beach" as an example, keyword extraction technology can extract three keywords: "Yangyang," "beach," and "running." If the keyword "Yangyang" retrieves Photo 1 and Photo 3, the keyword "beach" retrieves "Photo 5," and the keyword "running" retrieves "Photo 1," and the text vector retrieves "Photo 3 and Photo 1," then Photo 1 was retrieved 3 times, Photo 3 was retrieved 2 times, and Photo 5 was retrieved 1 time. Therefore, the photo ranking result is "Photo 1, Photo 3, Photo 5."

[0120] In this embodiment, a higher number of recalls means a higher degree of relevance between the third image and the searched text. Therefore, sorting based on the number of recalls allows the third images with higher relevance to be displayed earlier.

[0121] In some embodiments, the method further includes:

[0122] If the plurality of third images includes K fourth images with the same recall count, the recall method for each of the fourth images is determined, where K is a positive integer;

[0123] When the recall method includes keyword recall, M third tags corresponding to the fourth image are determined, as well as M third tag weights corresponding to the M third tags. The sum of the M third tag weights is determined as the first score of the fourth image, where M is a positive integer.

[0124] When the recall method includes vector recall, the vector similarity between the second image vector converted from the fourth image and the text vector is determined as the second score;

[0125] The sum of the first score and the second score of each of the fourth images is determined as the total score;

[0126] The K fourth images are sorted in descending order of their total scores.

[0127] In this embodiment, if the recall counts of each of the multiple third images are different, then the multiple third images can be sorted in descending order of recall count. However, if there are K fourth images with the same recall count among the multiple third images, then each fourth image needs to be scored to obtain a total score for each fourth image, and then the K fourth images are sorted in descending order of total score.

[0128] Specifically, for any fourth image, if the recall method for the fourth image includes keyword recall, then M third tags that match the keywords among the tags included in the fourth image can be identified, and the sum of these M third tags is determined as the first score of the fourth image.

[0129] Furthermore, for any fourth image, if the recall method for the fourth image includes vector recall, then the vector similarity between the converted second image vector and the text vector can be directly determined as the second score, and the sum of the first score and the second score is the total score.

[0130] For example, given two fourth images, Photo 1 and Photo 2, and the search text is "beach at the seaside," the keywords include "seaside and beach." Photo 1 is retrieved using the keywords "seaside" and "beach," with a label weight of 1 for "seaside" and 0.5 for "beach." Therefore, Photo 1 has a first score of 1.5. Since its second score is 0, Photo 1's total score is 1.5. Photo 2, however, is retrieved solely using the keyword "beach," with a label weight of 0.8. Furthermore, Photo 2 can be retrieved using its vector; the similarity between the second image vector and the text vector is 0.8. Therefore, Photo 2 has a first score of 0.8 and a second score of 0.8, resulting in a total score of 1.6. Thus, the ranking would be "Photo 2, Photo 1."

[0131] In this way, when the number of recalls is the same, the relevance between the image and the search text can be characterized based on the image score, and the images can be sorted based on the score, so that images with higher relevance can be displayed at the top.

[0132] Figure 7 This is a schematic diagram of the structure of a tag determining device provided in another embodiment of this application, as shown below. Figure 7 As shown, the label determining device may include:

[0133] The first determining module 701 is configured to determine a first label of the first image in response to a first input to a first image in the image set;

[0134] The second determining module 702 is used to determine the image similarity between the first image and the second image based on the characteristic information of the first image and the characteristic information of the second image in the image set.

[0135] The third determining module 703 is used to determine the second label of the second image based on the image similarity and the first label.

[0136] In this application, in response to a first input to a first image in an image set, a first label for the first image can be determined; based on the vector of the first image and the vector of a second image in the image set, an image similarity between the first image and the second image can be determined; and based on the image similarity and the first label, a second label for the second image can be determined. In this way, a user can manually label only a subset of images in the image set. Then, the electronic device can automatically propagate the label of the first image to similar second images based on the image similarity by calculating the image similarity between different images in the image set, reducing the workload of manual labeling and lowering the cost of image labeling.

[0137] In another alternative example, the label determining device further includes:

[0138] The fourth determining module is used to determine the first label weight corresponding to the first label of the first image in response to the user's first input to the first image in the image set;

[0139] The first generation module is used to construct a first mapping relationship between the first image, the first label, and the weight of the first label, and to generate a first inverted record based on the first mapping relationship;

[0140] The first construction module is used to construct a tag library for the image set based on the first inverted record.

[0141] In another alternative example, the label determining device further includes:

[0142] The fifth determining module is used to determine the product of the first label weight and the image similarity as the second label weight corresponding to the second label in the second image;

[0143] The second generation module is used to construct a second mapping relationship between the second image, the second label, and the weight of the second label, and to generate a second inverted record based on the second mapping relationship;

[0144] An add module is used to add the second inverted record to the tag library of the image set.

[0145] In another alternative example, the image similarity includes a first similarity, which is the overall similarity between the first image and the second image;

[0146] The third determining module 703 also includes:

[0147] The first determining unit is configured to determine the first label as the second label of the second image when the first similarity between the first image and the second image is greater than or equal to the first similarity threshold.

[0148] In another alternative example, the image similarity includes a second similarity, which is the similarity between elements in the first image and elements in the second image;

[0149] The third determining module 703 also includes:

[0150] The second determining unit is configured to determine T first elements in the first image and T second elements in the second image when the first similarity between the first image and the second image is less than a first similarity threshold and greater than or equal to a second similarity threshold, wherein each first element corresponds to a first label and T is a positive integer;

[0151] The third determining unit is used to determine the first label corresponding to the i-th first element as the second label of the j-th second element when the second similarity between the i-th first element and the j-th second element is greater than the third similarity threshold, where i and j are both positive integers less than or equal to T.

[0152] In another alternative example, the first determining module 701 further includes:

[0153] A first display unit is used to display the first image;

[0154] The second display unit is configured to display a label editing control of the first image in response to a first sub-input to the first image;

[0155] The fourth determining unit is configured to, in response to a second sub-input to the label editing control, determine the phrase corresponding to the second sub-input as the first label.

[0156] In another alternative example, the first determining module 701 further includes:

[0157] The fifth determining unit is used to determine the N third elements included in the first image, where N is a positive integer greater than 1;

[0158] The sixth determining unit is used to determine P fourth elements from the N third elements, where P is a positive integer less than or equal to N, and the fourth elements are elements whose pixel points are greater than a pixel point threshold.

[0159] The seventh determining unit is used to determine the first tag corresponding to each of the fourth elements based on the content of the fourth elements.

[0160] In another optional example, the label determining device further includes:

[0161] The receiving module is used to receive the user's search intent information;

[0162] The retrieval module is used to search the image set according to the retrieval intent information and obtain multiple third images in the image set;

[0163] The display module is used to display the multiple third images.

[0164] In another optional example, the retrieval module also includes:

[0165] The eighth determining unit is used to determine the search text corresponding to the search intent information;

[0166] A retrieval unit is configured to retrieve a third image corresponding to the third tag in the image set if a third tag matching the retrieval text exists in the image set.

[0167] The recall unit is configured to, when no third tag matching the search text exists in the image set, obtain multiple third images from the image set using at least one of vector recall and keyword recall methods:

[0168] The keyword recall is as follows: determine the third tag that matches the keywords in the search text, and retrieve the third image corresponding to the third tag;

[0169] The vector recall is as follows: obtain the image vectors converted from each image in the image set, and the text vectors converted from the search text, retrieve the first image vector whose third similarity to the text vector is greater than the fourth similarity threshold, and determine the image corresponding to the first image vector as the third image.

[0170] In another optional example, the retrieval module is further configured to determine the number of first tags corresponding to the third tags of each of the third images, and the first value corresponding to each of the third images; wherein, if the image vector corresponding to the third image matches the text vector, the first value corresponding to the third image is 1, and if the image vector corresponding to the third image does not match the text vector, the first value corresponding to the third image is 0.

[0171] The sum of the number of first tags and the first value of each of the third images is determined as the recall number of the third image;

[0172] The multiple third images are sorted in descending order of their recall counts to obtain a sorting result;

[0173] The multiple third images are displayed according to the sorting results.

[0174] In another alternative example, the label determining device is also used for:

[0175] If the plurality of third images includes K fourth images with the same recall count, the recall method for each of the fourth images is determined, where K is a positive integer;

[0176] When the recall method includes keyword recall, M third tags corresponding to the fourth image are determined, as well as M third tag weights corresponding to the M third tags. The sum of the M third tag weights is determined as the first score of the fourth image, where M is a positive integer.

[0177] When the recall method includes vector recall, the vector similarity between the second image vector converted from the fourth image and the text vector is determined as the second score;

[0178] The sum of the first score and the second score of each of the fourth images is determined as the total score;

[0179] The K fourth images are sorted in descending order of their total scores.

[0180] The tag determining device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific type of device.

[0181] The tag determining device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0182] The label determining device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0183] Optionally, such as Figure 8 As shown, this application embodiment also provides an electronic device 100, including a processor 110, a memory 119, and a program or instructions stored in the memory 119 and executable on the processor 110. When the program or instructions are executed by the processor 110, they implement the various processes of the above-described tag determination method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0184] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0185] Please refer to the following: Figure 9 , Figure 9 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. The electronic device 100 includes, but is not limited to, components such as: a radio frequency unit 121, a network module 122, an audio output unit 123, an input unit 124, a sensor 125, a display unit 126, a user input unit 127, an interface unit 128, a memory 129, and a processor 120.

[0186] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 120 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0187] The processor 120 is configured to determine a first label for the first image in response to a first input to the first image in the image set;

[0188] The processor 120 is configured to determine the image similarity between the first image and the second image based on the characteristic information of the first image and the characteristic information of the second image in the image set.

[0189] Processor 120 is configured to determine a second label for the second image based on the image similarity and the first label.

[0190] In this application, in response to a first input to a first image in an image set, a first label for the first image can be determined; based on the vector of the first image and the vector of a second image in the image set, an image similarity between the first image and the second image can be determined; and based on the image similarity and the first label, a second label for the second image can be determined. In this way, a user can manually label only a subset of images in the image set. Then, the electronic device can automatically propagate the label of the first image to similar second images based on the image similarity by calculating the image similarity between different images in the image set, reducing the workload of manual labeling and lowering the cost of image labeling.

[0191] In another alternative example, the processor 120 is further configured to determine a first label weight corresponding to a first label of the first image in response to a first input from a user to a first image in the image set;

[0192] Construct a first mapping relationship between the first image, the first label, and the weight of the first label, and generate a first inverted record based on the first mapping relationship;

[0193] A tag library for the image set is constructed based on the first inverted record.

[0194] In another alternative example, the processor 120 is also used for:

[0195] The product of the first label weight and the image similarity is determined as the second label weight corresponding to the second label in the second image;

[0196] Construct a second mapping relationship between the second image, the second label, and the weights of the second label, and generate a second inverted record based on the second mapping relationship;

[0197] Add the second inverted record to the tag library of the image set.

[0198] In another alternative example, the image similarity includes a first similarity, which is the overall similarity between the first image and the second image;

[0199] Processor 120 is also used for:

[0200] If the first similarity between the first image and the second image is greater than or equal to the first similarity threshold, the first label is determined as the second label of the second image.

[0201] In another alternative example, the image similarity includes a second similarity, which is the similarity between elements in the first image and elements in the second image;

[0202] Processor 120 is also used for:

[0203] If the first similarity between the first image and the second image is less than the first similarity threshold and greater than or equal to the second similarity threshold, T first elements in the first image and T second elements in the second image are determined, wherein each first element corresponds to a first label and T is a positive integer;

[0204] If the second similarity between the i-th first element and the j-th second element is greater than the third similarity threshold, the first label corresponding to the i-th first element is determined as the second label of the j-th second element, where i and j are both positive integers less than or equal to T.

[0205] In another alternative example, the display unit 126 is used to display the first image;

[0206] Processor 120 is configured to display a label editing control of the first image in response to a first sub-input to the first image;

[0207] Processor 120 is configured to, in response to a second sub-input to the label editing control, determine the phrase corresponding to the second sub-input as the first label.

[0208] In another alternative example, the processor 120 is configured to determine N third elements included in the first image, where N is a positive integer greater than 1;

[0209] Processor 120 is configured to determine P fourth elements among the N third elements, where P is a positive integer less than or equal to N, and the fourth elements are elements that include pixels with a pixel value greater than a pixel value threshold.

[0210] Processor 120 is configured to determine a first tag corresponding to each of the fourth elements based on the content of the fourth elements.

[0211] In another optional example, the user input unit 127 is used to receive retrieval intent information input by the user;

[0212] Processor 120 is configured to perform a search in the image set based on the search intent information to obtain multiple third images in the image set;

[0213] Display unit 126 is used to display the plurality of third images.

[0214] In another alternative example, the processor 120 is further configured to determine the search text corresponding to the search intent information;

[0215] If a third tag matching the search text exists in the image set, the third image corresponding to the third tag is retrieved from the image set.

[0216] If no third tag matching the search text exists in the image set, multiple third images from the image set are obtained through at least one of vector recall and keyword recall:

[0217] The keyword recall is as follows: determine the third tag that matches the keywords in the search text, and retrieve the third image corresponding to the third tag;

[0218] The vector recall is as follows: obtain the image vectors converted from each image in the image set, and the text vectors converted from the search text, retrieve the first image vector whose third similarity to the text vector is greater than the fourth similarity threshold, and determine the image corresponding to the first image vector as the third image.

[0219] In another optional example, the processor 120 is further configured to determine the number of first tags corresponding to each of the third images and the first value corresponding to each of the third images; wherein, if the image vector corresponding to the third image matches the text vector, the first value corresponding to the third image is 1, and if the image vector corresponding to the third image does not match the text vector, the first value corresponding to the third image is 0.

[0220] The sum of the number of first tags and the first value of each of the third images is determined as the recall number of the third image;

[0221] The multiple third images are sorted in descending order of their recall counts to obtain a sorting result;

[0222] The multiple third images are displayed according to the sorting results.

[0223] In another alternative example, the processor 120 is also used for:

[0224] If the plurality of third images includes K fourth images with the same recall count, the recall method for each of the fourth images is determined, where K is a positive integer;

[0225] When the recall method includes keyword recall, M third tags corresponding to the fourth image are determined, as well as M third tag weights corresponding to the M third tags. The sum of the M third tag weights is determined as the first score of the fourth image, where M is a positive integer.

[0226] When the recall method includes vector recall, the vector similarity between the second image vector converted from the fourth image and the text vector is determined as the second score;

[0227] The sum of the first score and the second score of each of the fourth images is determined as the total score;

[0228] The K fourth images are sorted in descending order of their total scores.

[0229] It should be understood that, in this embodiment, the input unit 124 may include a graphics processing unit (GPU) 1241 and a microphone 1242. The GPU 1241 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 126 may include a display panel 1261, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 127 includes at least one of a touch panel 1271 and other input devices 1272. The touch panel 1271 is also called a touch screen. The touch panel 1271 may include a touch detection device and a touch controller. Other input devices 1272 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0230] The memory 129 can be used to store software programs and various data. The memory 129 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 129 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 129 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0231] Processor 120 may include one or more processing units; optionally, processor 120 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 120.

[0232] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described tag determination method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0233] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0234] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described label determination method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0235] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0236] This application provides a computer program product that is stored in a storage medium and executed by at least one processor to implement the various processes of the label determination method embodiment described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0237] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0238] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0239] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A label identification method, characterized in that, include: In response to a first input to a first image in a set of images, a first label for the first image is determined; Based on the characteristic information of the first image and the characteristic information of the second image in the image set, the image similarity between the first image and the second image is determined; A second label for the second image is determined based on the image similarity and the first label; the image similarity includes a second similarity, which is the similarity between elements in the first image and elements in the second image; Determining the second label of the second image based on the image similarity and the first label includes: If the first similarity between the first image and the second image is less than the first similarity threshold and greater than or equal to the second similarity threshold, T first elements in the first image and T second elements in the second image are determined, wherein each first element corresponds to a first label, and T is a positive integer; the first similarity is the cosine similarity between the first vector of the first image and the second vector of the second image; If the second similarity between the i-th first element and the j-th second element is greater than the third similarity threshold, the first label corresponding to the i-th first element is determined as the second label of the j-th second element, where i and j are both positive integers less than or equal to T.

2. The method according to claim 1, characterized in that, The first input includes a first sub-input and a second sub-input; The step of determining a first label for a first image in response to a first input to a first image in the image set includes: Display the first image; In response to a first sub-input to the first image, a label editing control for the first image is displayed; In response to a second sub-input to the label editing control, the phrase corresponding to the second sub-input is determined as the first label.

3. The method according to claim 1, characterized in that, After determining the second label of the second image based on the image similarity and the first label, the method further includes: Receive user input regarding search intent; Based on the search intent information, a search is performed in the image set to obtain multiple third images from the image set; Display the multiple third images.

4. The method according to claim 3, characterized in that, The step of searching the image set according to the search intent information to obtain multiple third images in the image set includes: Determine the search text corresponding to the search intent information; If a third tag matching the search text exists in the image set, the third image corresponding to the third tag is retrieved from the image set. If no third tag matching the search text exists in the image set, multiple third images from the image set are obtained through at least one of vector recall and keyword recall: The keyword recall is as follows: determine the third tag that matches the keywords in the search text, and retrieve the third image corresponding to the third tag; The vector recall is as follows: obtain the image vectors converted from each image in the image set, and the text vectors converted from the search text, retrieve the first image vector whose third similarity to the text vector is greater than the fourth similarity threshold, and determine the image corresponding to the first image vector as the third image.

5. A label determining device, characterized in that, include: The first determining module is configured to determine a first label of the first image in response to a first input to a first image in the image set; The second determining module is used to determine the image similarity between the first image and the second image based on the characteristic information of the first image and the characteristic information of the second image in the image set. The third determining module is used to determine a second label for the second image based on the image similarity and the first label; the image similarity includes a second similarity, which is the similarity between elements in the first image and elements in the second image; The third determining module includes: The second determining unit is configured to determine T first elements in the first image and T second elements in the second image when the first similarity between the first image and the second image is less than a first similarity threshold and greater than or equal to a second similarity threshold, wherein each first element corresponds to a first label, and T is a positive integer; the first similarity is the cosine similarity between the first vector of the first image and the second vector of the second image; The third determining unit is used to determine the first label corresponding to the i-th first element as the second label of the j-th second element when the second similarity between the i-th first element and the j-th second element is greater than the third similarity threshold, where i and j are both positive integers less than or equal to T.

6. The apparatus according to claim 5, characterized in that, The first input includes a first sub-input and a second sub-input; The first determining module includes: A first display unit is used to display the first image; The second display unit is configured to display a label editing control of the first image in response to a first sub-input to the first image; The fourth determining unit is configured to determine the phrase corresponding to the second sub-input as the first tag in response to the second sub-input of the tag editing control.

7. The apparatus according to claim 5, characterized in that, The label determining device includes: The receiving module is used to receive the user's search intent information; The retrieval module is used to search the image set according to the retrieval intent information and obtain multiple third images in the image set; The display module is used to display the multiple third images.

8. The apparatus according to claim 7, characterized in that, The retrieval module also includes: The eighth determining unit is used to determine the search text corresponding to the search intent information; A retrieval unit is configured to retrieve a third image corresponding to the third tag in the image set if a third tag matching the retrieval text exists in the image set. The recall unit is configured to, when no third tag matching the search text exists in the image set, obtain multiple third images from the image set using at least one of vector recall and keyword recall methods: The keyword recall is as follows: determine the third tag that matches the keywords in the search text, and retrieve the third image corresponding to the third tag; The vector recall is as follows: obtain the image vectors converted from each image in the image set, and the text vectors converted from the search text, retrieve the first image vector whose third similarity to the text vector is greater than the fourth similarity threshold, and determine the image corresponding to the first image vector as the third image.

Images