Image classification method and device, electronic equipment and readable storage medium

Abstract

The application discloses an image classification method and device, electronic equipment and a readable storage medium, and belongs to the technical field of image processing. The image classification method comprises the following steps: acquiring a first keyword; acquiring coordinate information, wherein the coordinate information is the coordinates of a detection frame corresponding to a first image, and the first image is an image obtained by classifying the first keyword; performing feature extraction on a first region of a second image to obtain a first feature vector; and determining the classification to which the second image belongs based on the similarity between the coordinate information and the first feature vector.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and specifically relates to an image classification method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] After a user takes a picture, the image will be stored in the user's album for easy viewing.

[0003] To meet users' image viewing needs, users can manually add tags to images and then classify them based on the tags. At the same time, relevant manufacturers also provide some image classification models so that images can be classified using these models.

[0004] The classification method of manually adding labels to images by users is time-consuming, while the classification method using the classification model provided by the manufacturer can only classify images of a specified category and cannot perform custom classification. Summary of the Invention

[0005] The purpose of this application is to provide an image classification method, apparatus, electronic device, and readable storage medium that can customize classification categories according to user needs, thereby meeting the user's personalized classification requirements.

[0006] In a first aspect, embodiments of this application provide an image classification method, including: obtaining a first keyword; obtaining coordinate information, wherein the coordinate information is the coordinates of a detection box corresponding to a first image, and the first image is an image obtained by classification according to the first keyword; extracting features from a first region of a second image to obtain a first feature vector; and determining the classification of the second image based on the similarity between the coordinate information and the first feature vector.

[0007] Secondly, embodiments of this application provide an image classification apparatus, comprising: a first acquisition module for acquiring a first keyword; a second acquisition module for acquiring coordinate information, wherein the coordinate information is the coordinates of a detection box corresponding to a first image, and the first image is an image obtained by classification according to the first keyword; an extraction module for extracting features from a first region of a second image to obtain a first feature vector; and a processing module for determining the classification of the second image based on the similarity between the coordinate information and the first feature vector.

[0008] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions implementing the steps of the method as described in the first aspect when executed by the processor.

[0009] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, they implement the steps of the method as described in the first aspect.

[0010] Fifthly, embodiments of this application provide a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method as described in the first aspect.

[0011] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method as described in the first aspect.

[0012] In this embodiment, after obtaining the coordinate information and the first keyword input by the user, features are extracted from the first region of the second image to be classified to obtain a first feature vector. The first feature vector is then compared with the coordinate information to calculate the similarity result. This similarity result is then used to classify the category of the second image. In this process, since the coordinate information represents the coordinates of the detection box corresponding to the image classified according to the first keyword, it reflects the characteristics of the image classified according to the first keyword. By calculating the similarity between the first feature vector and the coordinate information, the similarity between the image classified according to the first keyword and the second image can be measured, and the second image can be classified based on this similarity. This process overcomes the problem of limited classification scenarios inherent in classification methods using manufacturer-provided classification models. Furthermore, while enabling custom image classification, it improves classification efficiency compared to manually adding labels to images. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating the image classification method in some embodiments of this application;

[0014] Figure 2 This is a schematic diagram illustrating the formation of the optimal user-defined keyword combination in some embodiments of this application;

[0015] Figure 3 This is a schematic diagram of the generation logic of the first feature vector in some embodiments of this application;

[0016] Figure 4 These are schematic block diagrams of image classification devices in some embodiments of this application;

[0017] Figure 5 These are schematic block diagrams of electronic devices according to some embodiments of this application;

[0018] Figure 6This is a schematic diagram of the hardware structure of an electronic device according to some embodiments of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0021] The image classification method, apparatus, electronic device, and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0022] In some embodiments of this application, such as Figure 1 As shown, an image classification method is proposed, including:

[0023] Step 102: Obtain the first keyword.

[0024] In some embodiments of this application, the first keyword may be a statement or word entered by the user in natural language, or text entered by the user through a device or terminal, or it may be determined based on information sent by other users.

[0025] For example, if other users send messages containing four words: travel, sightseeing, scenery, and waterfall, after receiving the messages, the user can manually select the first keyword from these four words, such as "sightseeing" and "travel".

[0026] For example, the first keyword can be one or more words, such as "play", "party", or "pet" or "family".

[0027] For example, the first keyword could be a sentence, such as "family travel together".

[0028] Step 104: Obtain coordinate information, which is the coordinate of the detection box corresponding to the first image. The first image is an image obtained by classifying according to the first keyword.

[0029] In the above embodiments, the first image may be an image stored on a server or an image stored locally on the device.

[0030] Specifically, the first image can be a single image or multiple images. If the first image is multiple images, the coordinate information is obtained by weighted summation of the coordinate information of multiple images.

[0031] For example, the coordinate information is obtained by summing the coordinate information of multiple images using the same weighting value.

[0032] In the above embodiments, the detection frame is the frame displayed on the first image when target detection is performed on the first image.

[0033] Step 106: Extract features from the first region of the second image to obtain the first feature vector.

[0034] Step 108: Determine the category to which the second image belongs based on the similarity between the coordinate information and the first feature vector.

[0035] In this embodiment, after obtaining the coordinate information and the first keyword input by the user, features are extracted from the first region of the second image to be classified to obtain a first feature vector. The first feature vector is then compared with the coordinate information to calculate the similarity result. This similarity result is then used to classify the category of the second image. In this process, since the coordinate information represents the coordinates of the detection box corresponding to the image classified according to the first keyword, it reflects the characteristics of the image classified according to the first keyword. By calculating the similarity between the first feature vector and the coordinate information, the similarity between the image classified according to the first keyword and the second image can be measured, and the second image can be classified based on this similarity. This process overcomes the problem of limited classification scenarios inherent in classification methods using manufacturer-provided classification models. Furthermore, while enabling custom image classification, it improves classification efficiency compared to manually adding labels to images.

[0036] In some embodiments of this application, before obtaining coordinate information, the method further includes: determining N keyword combinations based on a first keyword, where N is a positive integer greater than or equal to 1; obtaining coordinate information specifically includes: obtaining N average coordinates, wherein the N average coordinates correspond one-to-one with the coordinates of the detection box of the first image corresponding to the N keyword combinations; before determining the category of the second image based on the similarity between the coordinate information and the first feature vector, the method further includes: determining candidate keyword combinations based on the coordinate information of the first region and the N average coordinates; determining the category of the second image based on the similarity between the coordinate information and the first feature vector specifically includes: determining the category of the second image based on the similarity between the coordinate information corresponding to the candidate keyword combinations and the first feature vector.

[0037] In this embodiment, a first keyword is used to determine N keyword combinations. These N keyword combinations are then combined and expanded to generate a total of N keyword combinations. The category of the second image is then determined based on the image classification corresponding to each of these N keyword combinations. This process reduces the likelihood of situations where the first keyword is too obscure to find corresponding coordinate information.

[0038] In the above embodiment, after determining the N average coordinates that correspond one-to-one with the N keyword combinations, the N keyword combinations are filtered based on the coordinate information of the first region and the N average coordinates to obtain candidate keyword combinations.

[0039] In this process, by setting this filtering action, it is not necessary to calculate the similarity between the N average coordinates and the first feature vector one by one. It is only necessary to calculate the similarity between the coordinate information corresponding to the candidate keyword combination and the first feature vector to determine the category to which the second image belongs. In this process, the amount of calculation required to determine the category to which the second image belongs can be reduced, and at the same time, it is also easier to shorten the time required for image classification.

[0040] In some embodiments of this application, determining N keyword combinations based on a first keyword specifically includes: vectorizing each keyword in the keyword list to obtain a corresponding semantic vector, the keyword list including the first keyword; calculating the similarity between different keywords in the keyword list based on the semantic vector; obtaining the first number of times the first keyword and the second keyword can be combined, and the first number of keyword combinations that can be generated, the second keyword being a keyword in the keyword list other than the first keyword; and filtering the combined keyword combinations based on the similarity, the first number of times, and the first number to obtain N keyword combinations.

[0041] In this embodiment, the first keyword is vectorized so that the semantic vector obtained can be used to calculate the similarity. In this process, the similarity is taken into account when combining N keyword combinations. Therefore, the N keyword combinations can be more similar to the first keyword input by the user, thereby making the subsequent image classification more accurate.

[0042] In the above embodiment, by obtaining the first number and the first quantity, the user's classification needs can be reasonably inferred, thereby reasonably filtering out N keyword combinations.

[0043] For example, the similarity is calculated based on the following formula:

[0044]

[0045]

[0046] Where Keycos represents the similarity between different keywords, x i x j These are the semantic vectors obtained after vectorizing the first and second keywords, respectively, n. ij The total number of times these two keywords are entered by all users across the entire server.

[0047] Where, n ij Including the existence of only x i Only x exists j and the simultaneous occurrence of x i x j The total number of these cases, n i n j Then each of them has only x. i Keywords and x j The frequency of keywords.

[0048] n k This refers to the maximum number of times a user allows a keyword to be matched with other keywords, thus forming keyword combinations. This is also the maximum number of times the first keyword and the second keyword can be combined, which is the first number in this embodiment. α is the combination number adjustment factor, which needs to be given by the user. Num is the maximum number of combinations generated after the user inputs a certain number of keywords, which is also the first number in this embodiment. β is the combination number adjustment factor.

[0049] In this embodiment, the similarity between keywords is measured by formula (1), and formula (2) is used to determine the limit on the number of combinations of a single keyword with other keywords and the total number of combinations.

[0050] In some embodiments of this application, a trained Word2Vec natural language processing model is used to vectorize the keywords in the keyword list to obtain the corresponding semantic vectors.

[0051] Specifically, the first keyword entered by the user can be stored in a custom keyword list, and the keywords in the list can be vectorized using the Word2Vec natural language processing model, transforming them into a 1×4096 vector.

[0052] Word2Vec is a natural language processing technique used to represent words as mathematical vectors. Its main idea is to embed words into a low-dimensional vector space in order to capture the semantic relationships between words.

[0053] Word2Vec natural language processing models are typically trained on large text corpora, assigning a vector to each word by learning the distribution patterns of words in context. This makes words with similar semantic meanings appear close to each other in the vector space. Two main algorithms of Word2Vec are Continuous Bag of Words (CBOW) and Skip-gram.

[0054] In some embodiments of this application, the image classification method can be deployed on the server side. Specifically, users can define semantic keywords, or first keywords, through a mobile terminal application. Keywords can be added, edited, or deleted to clearly define the criteria for image classification.

[0055] For example, users can use text input boxes to freely enter or select desired primary keywords according to their personalized needs, such as personal photos, family, pets, locations, food, indoors, cars, and children. The personalized setting of the primary keyword allows users to customize image classification criteria based on their unique interests and needs. After editing the primary keyword, the entered primary keyword is uploaded to the server.

[0056] Specifically, the mobile application packages the user's primary keyword into a data packet and sends the packet to the server via a secure communication protocol. Once the server receives the data packet, it parses and stores the uploaded primary keyword, securely storing the user's personalized classification requirements in a database for use in subsequent image classification tasks. This ensures that the user-defined primary keyword can participate in calculations and decisions throughout the entire image classification process, achieving personalized image classification.

[0057] like Figure 2As shown, the first keyword is uploaded to the server in the form of a user-defined keyword list. In the server, it undergoes the following processing: Word2Vec vectorizes each keyword in the user-defined keyword list, calculates the cosine multiplication of the keyword vectors, calculates the number of user-defined N combinations, returns the maximum value of the cosine of the keyword vectors N groups, and outputs the optimal user-defined keyword combination.

[0058] In some embodiments of this application, determining candidate keyword combinations based on the coordinate information of the first region and N average coordinates specifically includes: calculating the distance value between the coordinate information of the first region and each average coordinate; and determining candidate keyword combinations from the N keyword combinations based on the distance values.

[0059] In this embodiment, the coordinate information of the first region and the distance value of each average coordinate are calculated to obtain N calculated distance values, where the distance values ​​can characterize the proximity of the first region to the detection frame. Therefore, after calculating the N distance values, the N distance values ​​can be sorted by size to select candidate keyword combinations.

[0060] In this process, N keyword combinations can be filtered by the user's sampling habits of the shooting object, and then the candidate keyword combinations can be obtained. This reduces the amount of calculation required to determine the category to which the second image belongs, and at the same time, ensures the accuracy of the candidate keyword combinations.

[0061] In some embodiments of this application, the coordinate information of the first region may be the coordinate value of the center point of the first region, the coordinate value of the edge vertex of the first region, or the coordinate value calculated based on the coordinate value of the edge vertex of the first region.

[0062] Specifically, the coordinate information of the first region and the distance value of each average coordinate are calculated based on formula (3):

[0063] Min((r fx -o k ) 2 +(r fy -o l ) 2 ), f=1, 2, 3,..., t (3)

[0064] Where, r fx r fy This represents the average coordinates of the user-captured image in the image server under the f-th candidate keyword combination, which is also the center coordinate of the detection box. [o k o l[] represents the center coordinates of the best ROI area for the current user's photo, which is also the coordinate information of the first area. The value of t is the same as that of N.

[0065] Based on formula (3), the minimum and maximum distances between the coordinates of the keyword content of the subordinate image and the center coordinates of the best ROI region of the current user's captured image can be calculated, thereby enabling the selection of candidate keyword combinations.

[0066] In some embodiments of this application, each keyword combination corresponds to multiple images. Target detection is performed on each image to obtain a corresponding detection box, which has a third coordinate value. The average coordinate value of the multiple third coordinate values ​​is used as the average coordinate corresponding to the keyword combination.

[0067] In some embodiments of this application, the YOLOv8 object detection algorithm is used to perform object detection on all images stored in the server, wherein the images stored in the server are also the first images in this application.

[0068] Specifically, the calculation of the N average coordinates is based on formula (4):

[0069] [r1, r2, ..., r t ] = Obj_reg(g1, g2, ..., g t (4)

[0070] Where, Obj_reg(g1, g2, ..., g t The coordinates of the bounding boxes after object detection on the image in the server are given. In g1, g represents the image, and 1, 2…t represent numerical values, indicating the t-th bounding box. t The first keyword is defined by the user, and then t keyword combinations are formed. The coordinates of the center of the target detection box of the image taken by the user in the image server under these t keyword combinations are also known as the N average coordinates.

[0071] In this embodiment, by extracting the sampling habits of a large number of users for a certain object, for example, if the server stores 10,000 pictures with the keyword "pet", the target object is detected by the target detection box, such as detecting pets. If it is found that these target objects, i.e., pets, are all located at the lower right corner coordinates of the picture, then the average coordinates of these lower right corner coordinates are the average coordinates of the detection box.

[0072] In some embodiments of this application, the classification of the second image is determined based on the similarity between coordinate information and the first feature vector. Specifically, if the similarity is greater than or equal to a similarity threshold, the second image is classified into the category corresponding to the first keyword.

[0073] In this embodiment, if the similarity is greater than or equal to the similarity threshold, the second image is considered to belong to the category of the first keyword; otherwise, if the similarity is greater than or equal to the similarity threshold, the second image is considered not to belong to the category of the first keyword.

[0074] In some embodiments of this application, the similarity between the coordinate information and the first feature vector can be Euclidean distance.

[0075] Specifically, when there are multiple candidate keyword combinations, based on the sorting results of the distance values ​​of each keyword combination in the first region and the coordinate information of the first region, the average coordinates corresponding to the candidate keyword combinations are sequentially compared with the first feature vector to calculate the similarity, and the second image is classified based on the comparison results with the similarity threshold.

[0076] In some embodiments of this application, the similarity threshold is specifically expressed as shown in formulas (5) and (6):

[0077]

[0078]

[0079] Where D is the distance between the image vector and the text vector, i.e., the similarity threshold, pic is the first feature vector, and word is the first feature vector. s This is the vector corresponding to the s-th keyword combination. Then it is the distance judgment factor, D min This is represented as the minimum distance between the first feature vector corresponding to all classified images in the server and the semantic vector corresponding to the keyword.

[0080] In some embodiments of this application, the method further includes: dividing the second image into regions to obtain M second regions, where M is a positive integer greater than or equal to 2; obtaining M first calculated values ​​corresponding one-to-one with the M second regions, wherein in each second region, the first calculated value is determined based on the maximum pixel value, minimum pixel value, maximum value of the color channel, and minimum value of the color channel in the second region; and taking the second region corresponding to the second calculated value as the first region; wherein the second calculated value is the maximum value among the M first calculated values.

[0081] In this embodiment, the ROI region, i.e. the first region, can be extracted from pixel value contrast and color saturation. During this process, the user's region of interest can be quickly selected from the second image, and then the region of interest can be used for image classification. In this process, the accuracy of image classification can be improved.

[0082] In image processing, computer vision, and medical image analysis, a Region of Interest (ROI) is a specific area or set of regions of interest to a user or algorithm. An ROI is typically defined as the portion of an image that contains information about a particular object, target, or feature. Using ROIs can reduce computational complexity, improve algorithm efficiency, and ensure that only important parts are considered. ROIs can be defined by manual annotation, algorithmic detection, or prior knowledge specific to the task.

[0083] Specifically, in the embodiments of this application, the second image is first resized to 648×648. Then, calculations are performed on the resized second image based on this 64×64 size, which can be understood as the APE operator. The specific calculation principle of the APE operator is as follows:

[0084]

[0085] δ+ε=1(8)

[0086] Where γ is the region delimitation threshold, Pix i A 64×64 pixel size covers the maximum pixel value within the image area. j Let δ be the minimum pixel value within the image area covered by the operator, δ be the pixel contrast factor, and ε be a value that sums with δ to 1. i Col j These represent the maximum and minimum values ​​of the color channels for all pixels within a 64×64 size area, respectively.

[0087] Among them, Pix i Pix j ,δ,ε,Col j Col i It can be read directly using tools like PyTorch.

[0088] As can be seen from formula (7), it is possible to check whether there is a high pixel value contrast and color saturation within the 64×64 size coverage area. That is, the first half of formula (7) is to calculate the contrast of pixel values, the maximum pixel value and the minimum pixel value, and the second half is to calculate a color saturation. In addition, when users take pictures, there is usually a blur effect in the non-focus area where they do not care. The changes in color and brightness will be smoothed and the contrast is low. Based on this, the first calculated value corresponding to each second area can be obtained through the above calculation.

[0089] After the calculation of the M first calculated values ​​is completed, the ROI region is determined, that is, the first region is determined.

[0090] Specifically, the extraction formulas for the first region are shown in formulas (9) and (10):

[0091] μ=Max(γ1, γ2, γ3,..., γ 342,225 (9)

[0092] [o k o l ] = X, Y (10)

[0093] Where μ is the maximum value in the set of region delimitation thresholds, Max(γ1, γ2, γ3, ..., γ 342，225 This indicates that for a 648×648 image, the threshold values ​​are calculated 342225 times, resulting in 342225 region definition thresholds. The maximum value is the largest value extracted from these 342225 different sub-regions of the image, which is then used as the Region of Interest (ROI). X and Y represent the coordinates of the center point of the second region in the image when the maximum region definition threshold is reached. k [o1] represents the coordinates of the center point of the optimal ROI region.

[0094] In this embodiment, the region definition threshold, i.e. the calculation of the first calculated value, can be realized by formula (9), and the ROI region that the user is most interested in taking pictures at this time, i.e. the first region, can be extracted by formula (10).

[0095] After determining the first region, as Figure 3 As shown, the Conv operator, with a size of 3×3, a spacing of 1, and a padding term of 1, can be used for feature extraction. After six iterations of Conv, LeakyReLU, and BatchNormal calculations, the FeatureMap is directly Flattened to form a 1×4096 vector, which is the first feature vector.

[0096] Among them, LeakyReLU is an activation function, Conv is a convolutional layer, BatchNormal is batch normalization, which is similar to ordinary data normalization, and Ooutput is the output port.

[0097] In this embodiment, extracting the first region reduces the computational burden while improving the accuracy of image classification. This allows users to find images of interest more quickly without having to manually browse the entire album, saving time and effort.

[0098] The image classification method provided in this application can be executed by an image classification device. This application uses an image classification device executing the image classification method as an example to illustrate the image classification device provided in this application.

[0099] In some embodiments of this application, such as Figure 4 As shown, an image classification device 400 is proposed, comprising: a first acquisition module 402 for acquiring a first keyword; a second acquisition module 404 for acquiring coordinate information, wherein the coordinate information is the coordinates of the detection box corresponding to the first image, and the first image is an image obtained by classification according to the first keyword; an extraction module 406 for extracting features from a first region of a second image to obtain a first feature vector; and a processing module 408 for determining the classification of the second image based on the similarity between the coordinate information and the first feature vector.

[0100] In this embodiment, after obtaining the coordinate information and the first keyword input by the user, features are extracted from the first region of the second image to be classified to obtain a first feature vector. The first feature vector is then compared with the coordinate information to calculate the similarity result. This similarity result is then used to classify the category of the second image. In this process, since the coordinate information represents the coordinates of the detection box corresponding to the image classified according to the first keyword, it reflects the characteristics of the image classified according to the first keyword. By calculating the similarity between the first feature vector and the coordinate information, the similarity between the image classified according to the first keyword and the second image can be measured, and the second image can be classified based on this similarity. This process overcomes the problem of limited classification scenarios inherent in classification methods using manufacturer-provided classification models. Furthermore, while enabling custom image classification, it improves classification efficiency compared to manually adding labels to images.

[0101] In some embodiments of this application, the processing module 408 is further configured to: determine N keyword combinations based on the first keyword, where N is a positive integer greater than or equal to 1; the second acquisition module 404 is specifically configured to: acquire N average coordinates, wherein the N average coordinates correspond one-to-one with the coordinates of the detection box of the first image corresponding to the N keyword combinations; the processing module 408 is further configured to: determine candidate keyword combinations based on the coordinate information of the first region and the N average coordinates; the processing module 408 is specifically configured to: determine the category to which the second image belongs based on the similarity between the coordinate information corresponding to the candidate keyword combinations and the first feature vector.

[0102] In some embodiments of this application, the processing module 408 is specifically used to: vectorize each keyword in the keyword list to obtain a corresponding semantic vector, the keyword list including a first keyword; calculate the similarity between different keywords in the keyword list based on the semantic vector; obtain the first number of times the first keyword and the second keyword can be combined, and the first number of keyword combinations that can be generated, the second keyword being a keyword in the keyword list other than the first keyword; and filter the combined keyword combinations based on the similarity, the first number of times, and the first number to obtain N keyword combinations.

[0103] In some embodiments of this application, the processing module 408 is specifically used to: calculate the coordinate information of the first region and the distance value of each average coordinate; and determine the candidate keyword combination from N keyword combinations based on the distance value.

[0104] In some embodiments of this application, the processing module 408 is specifically used to: classify the second image into the category corresponding to the first keyword when the similarity is greater than or equal to the similarity threshold.

[0105] In some embodiments of this application, the extraction module 406 is further configured to: divide the second image into regions to obtain M second regions, where M is a positive integer greater than or equal to 2; obtain M first calculated values ​​corresponding one-to-one with the M second regions, wherein, in each second region, the first calculated value is determined based on the maximum pixel value, the minimum pixel value, the maximum value of the color channel, and the minimum value of the color channel in the second region; and take the second region corresponding to the second calculated value as the first region; wherein, the second calculated value is the maximum value among the M first calculated values.

[0106] The image classification device 400 in this embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This embodiment does not specifically limit the specific type of device.

[0107] The image classification device 400 in this embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment does not specifically limit its use.

[0108] The image classification device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0109] Optionally, such as Figure 5 As shown, this application embodiment also provides an electronic device 500, including a processor 502 and a memory 504. The memory 504 stores a program or instructions that can run on the processor 502. When the program or instructions are executed by the processor 502, they implement the various steps of the above-described image classification method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0110] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0111] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0112] The electronic device 600 includes, but is not limited to, components such as: radio frequency unit 601, network module 602, audio output unit 603, input unit 604, sensor 605, display unit 606, user input unit 607, interface unit 608, memory 609, and processor 610.

[0113] Those skilled in the art will understand that the electronic device 600 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 610 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0114] Among them, such as Figure 6 As shown, the processor 610 is used to: obtain a first keyword; obtain coordinate information, the coordinate information being the coordinates of the detection box corresponding to the first image, the first image being an image obtained by classification according to the first keyword; extract features from a first region of the second image to obtain a first feature vector; and determine the classification of the second image based on the similarity between the coordinate information and the first feature vector.

[0115] In some embodiments of this application, before obtaining coordinate information, the processor 610 is further configured to: determine N keyword combinations based on a first keyword, where N is a positive integer greater than or equal to 1; obtain coordinate information, specifically the processor 610 is configured to: obtain N average coordinates, wherein the N average coordinates correspond one-to-one with the coordinates of the detection box of the first image corresponding to the N keyword combinations; before determining the category of the second image based on the similarity between the coordinate information and the first feature vector, the processor 610 is further configured to: determine candidate keyword combinations based on the coordinate information of the first region and the N average coordinates; determine the category of the second image based on the similarity between the coordinate information and the first feature vector, specifically the processor 610 is configured to: determine the category of the second image based on the similarity between the coordinate information corresponding to the candidate keyword combinations and the first feature vector.

[0116] In some embodiments of this application, the processor 610 is specifically used to: vectorize each keyword in the keyword list to obtain a corresponding semantic vector, the keyword list including a first keyword; calculate the similarity between different keywords in the keyword list based on the semantic vector; obtain the first number of times the first keyword and the second keyword can be combined, and the first number of keyword combinations that can be generated, the second keyword being a keyword in the keyword list other than the first keyword; and filter the combined keyword combinations based on the similarity, the first number of times, and the first number to obtain N keyword combinations.

[0117] In some embodiments of this application, the processor 610 is specifically used to: calculate the coordinate information of the first region and the distance value of each average coordinate; and determine the candidate keyword combination from N keyword combinations based on the distance value.

[0118] In some embodiments of this application, the processor 610 is specifically used to classify the second image into the category corresponding to the first keyword when the similarity is greater than or equal to the similarity threshold.

[0119] In some embodiments of this application, the processor 610 is further configured to: divide the second image into regions to obtain M second regions, where M is a positive integer greater than or equal to 2; obtain M first calculated values ​​corresponding one-to-one with the M second regions, wherein, in each second region, the first calculated value is determined based on the maximum pixel value, the minimum pixel value, the maximum value of the color channel, and the minimum value of the color channel in the second region; and take the second region corresponding to the second calculated value as the first region; wherein, the second calculated value is the maximum value among the M first calculated values.

[0120] In this embodiment, after obtaining the coordinate information and the first keyword input by the user, features are extracted from the first region of the second image to be classified to obtain a first feature vector. The first feature vector is then compared with the coordinate information to calculate the similarity result. This similarity result is then used to classify the category of the second image. In this process, since the coordinate information represents the coordinates of the detection box corresponding to the image classified according to the first keyword, it reflects the characteristics of the image classified according to the first keyword. By calculating the similarity between the first feature vector and the coordinate information, the similarity between the image classified according to the first keyword and the second image can be measured, and the second image can be classified based on this similarity. This process overcomes the problem of limited classification scenarios inherent in classification methods using manufacturer-provided classification models. Furthermore, while enabling custom image classification, it improves classification efficiency compared to manually adding labels to images.

[0121] It should be understood that, in this embodiment, the input unit 604 may include a graphics processing unit (GPU) 6041 and a microphone 6042. The GPU 6041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 606 may include a display panel 6061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 607 includes at least one of a touch panel 6071 and other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0122] The memory 609 can be used to store software programs and various data. The memory 609 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 609 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 609 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0123] Processor 610 may include one or more processing units; optionally, processor 610 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 610.

[0124] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image classification method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0125] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0126] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described image classification method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0127] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0128] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the image classification method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0129] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0131] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image classification method, characterized in that, include: Get the first keyword; Obtain coordinate information, which is the coordinate of the detection box corresponding to the first image, and the first image is an image obtained by classification according to the first keyword; Feature extraction is performed on the first region of the second image to obtain the first feature vector; The classification of the second image is determined based on the similarity between the coordinate information and the first feature vector; Before obtaining the coordinate information, the process also includes: Based on the first keyword, N keyword combinations are determined, where N is a positive integer greater than or equal to 1; The acquisition of coordinate information specifically includes: Obtain N average coordinates, and the N average coordinates correspond one-to-one with the coordinates of the detection box of the first image corresponding to the combination of the N keywords; Before determining the category of the second image based on the similarity between the coordinate information and the first feature vector, the method further includes: Based on the coordinate information of the first region and the N average coordinates, determine the candidate keyword combination; Determining the category of the second image based on the similarity between the coordinate information and the first feature vector specifically includes: The classification of the second image is determined based on the coordinate information corresponding to the combination of candidate keywords and the similarity to the first feature vector.

2. The image classification method according to claim 1, characterized in that, The step of determining N keyword combinations based on the first keyword specifically includes: Each keyword in the keyword list is vectorized to obtain a corresponding semantic vector, wherein the keyword list includes the first keyword; Calculate the similarity between different keywords in the keyword list based on the semantic vector; Obtain the first number of times the first keyword and the second keyword can be combined, and the first number of keyword combinations that can be generated, wherein the second keyword is a keyword in the keyword list other than the first keyword; Based on the similarity, the first frequency, and the first quantity, the combined keyword combinations are filtered to obtain N keyword combinations.

3. The image classification method according to claim 1, characterized in that, The step of determining the candidate keyword combination based on the coordinate information of the first region and N average coordinates specifically includes: Calculate the coordinate information of the first region and the distance value of each of the average coordinates; Based on the distance value, the candidate keyword combination is determined from the N keyword combinations.

4. The image classification method according to claim 1, characterized in that, Determining the category of the second image based on the similarity between the coordinate information and the first feature vector specifically includes: If the similarity is greater than or equal to the similarity threshold, the second image is classified into the category corresponding to the first keyword.

5. The image classification method according to any one of claims 1 to 4, characterized in that, Also includes: The second image is divided into regions to obtain M second regions, where M is a positive integer greater than or equal to 2; Obtain M first calculated values ​​corresponding one-to-one with M second regions, wherein, in each second region, the first calculated value is determined based on the maximum pixel value, minimum pixel value, maximum color channel value, and minimum color channel value in the second region; The second region corresponding to the second calculated value is taken as the first region; The second calculated value is the maximum value among the M first calculated values.

6. An image classification device, characterized in that, include: The first acquisition module is used to acquire the first keyword; The second acquisition module is used to acquire coordinate information, which is the coordinates of the detection box corresponding to the first image, and the first image is an image obtained by classification according to the first keyword; The extraction module is used to extract features from the first region of the second image to obtain a first feature vector; The processing module is used to determine the category to which the second image belongs based on the similarity between the coordinate information and the first feature vector; The processing module is also used for: Based on the first keyword, N keyword combinations are determined, where N is a positive integer greater than or equal to 1; The second acquisition module is specifically used for: Obtain N average coordinates, and the N average coordinates correspond one-to-one with the coordinates of the detection box of the first image corresponding to the combination of the N keywords; The processing module is also used for: Based on the coordinate information of the first region and the N average coordinates, determine the candidate keyword combination; The processing module is specifically used for: The classification of the second image is determined based on the coordinate information corresponding to the combination of candidate keywords and the similarity to the first feature vector.

7. The image classification device according to claim 6, characterized in that, The processing module is specifically used for: Each keyword in the keyword list is vectorized to obtain a corresponding semantic vector, wherein the keyword list includes the first keyword; Calculate the similarity between different keywords in the keyword list based on the semantic vector; Obtain the first number of times the first keyword and the second keyword can be combined, and the first number of keyword combinations that can be generated, wherein the second keyword is a keyword in the keyword list other than the first keyword; Based on the similarity, the first frequency, and the first quantity, the combined keyword combinations are filtered to obtain N keyword combinations.

8. The image classification device according to claim 6, characterized in that, The processing module is specifically used for: Calculate the coordinate information of the first region and the distance value of each of the average coordinates; Based on the distance value, the candidate keyword combination is determined from the N keyword combinations.

9. The image classification device according to claim 6, characterized in that, The processing module is specifically used for: If the similarity is greater than or equal to the similarity threshold, the second image is classified into the category corresponding to the first keyword.

10. The image classification apparatus according to any one of claims 6 to 9, characterized in that, The extraction module is also used for: The second image is divided into regions to obtain M second regions, where M is a positive integer greater than or equal to 2; Obtain M first calculated values ​​corresponding one-to-one with M second regions, wherein, in each second region, the first calculated value is determined based on the maximum pixel value, minimum pixel value, maximum color channel value, and minimum color channel value in the second region; The second region corresponding to the second calculated value is taken as the first region; The second calculated value is the maximum value among the M first calculated values.

11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the method as described in any one of claims 1 to 5.

12. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 5.

Images