Image classification method and device based on large model, electronic equipment and storage medium

Abstract

The invention provides an image classification method and device based on a large model, electronic equipment and a storage medium. The method comprises the following steps: acquiring a to-be-classified first image; establishing a multi-stage scene label according to the business scene, the classification requirement and the identification capability of the large model; the first image and the scene label of the minimum level in the multiple levels of scene labels are analyzed through a large model, an initial label of the first image is obtained, the large model carries out image recognition through multiple image recognition technologies, and the initial label of the first image comprises multiple sets of labels; each group of labels is obtained by matching labels identified by an image identification technology with the scene label of the minimum level in the multiple levels of scene labels; and screening the initial label of the first image, and determining a screening result as a target label of the first image. The method can be applied to image intelligent classification in a complex scene, a classification result can be quickly and accurately obtained, the labor cost is effectively saved, and the classification efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of image data processing, and in particular to a large model-based image classification method, device, electronic device and storage medium. Background Art

[0002] In the image acquisition scenario, the image acquisition device will generate a massive amount of image data. The business scenarios and targets involved in these image data are complex and diverse, including large industry scenarios such as environmental protection, forestry, water conservancy, fishery, land, and transportation, as well as business scenarios such as cities, villages, streets, plains, and hills. In order to better apply these image data, it is necessary to classify the image data according to the industry or specific business scenarios. When classifying image data, traditional classification methods mainly rely on manual operations. Traditional methods cannot meet the needs of large-scale and intelligent processing, and have great limitations in processing efficiency and accuracy. Most of the current classification methods are based on simple classification of targets in images or videos, which often cannot meet the classification needs in complex business scenarios. Summary of the invention

[0003] The present application provides a large-model-based image classification method, device, electronic device and storage medium, which are used to solve the problem in the related art that when classifying image data, the classification efficiency is low and cannot meet the classification requirements.

[0004] To solve the above technical problems, this application is implemented as follows: In a first aspect, a large model-based image classification method is provided, comprising: Acquire a first image to be classified; Establish multi-level scenario labels based on business scenarios, classification requirements, and recognition capabilities of large models; The first image and the scene label of the minimum level in the multi-level scene labels are analyzed by the large model to obtain an initial label of the first image, wherein the large model uses multiple image recognition technologies to perform image recognition, and the initial label of the first image includes multiple groups of labels, each group of labels is obtained by matching a label recognized by an image recognition technology with a scene label of the minimum level in the multi-level scene labels; The initial label of the first image is screened, and the screening result is determined as the target label of the first image.

[0005] In a second aspect, a large model-based image classification device is provided, comprising: An acquisition module, acquiring a first image to be classified; Establish modules and build multi-level scene labels based on business scenarios, classification requirements, and recognition capabilities of large models; an analysis module, which analyzes the first image and the scene label of the minimum level in the multi-level scene labels by using the large model to obtain an initial label of the first image, wherein the large model uses multiple image recognition technologies to perform image recognition, and the initial label of the first image includes multiple groups of labels, each group of labels is obtained by matching a label recognized by an image recognition technology with a scene label of the minimum level in the multi-level scene labels; The screening module screens the initial label of the first image and determines the screening result as the target label of the first image.

[0006] According to a third aspect, an electronic device is provided, including: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method as described in the first aspect.

[0007] According to a fourth aspect, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the method described in the first aspect.

[0008] According to a fifth aspect, a computer program product is provided, the computer program product comprising a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute part or all of the steps in the method described in the first aspect.

[0009] In an embodiment of the present application, when classifying the first image, since the first image can be recognized at multiple levels through the large model (i.e., multiple image recognition technologies are used to recognize the first image), and subsequent image classification operations are performed based on the recognized labels, the accuracy of subsequent classification results can be improved; after the label is recognized through the large model, the recognized label can be matched with the pre-established multi-level scene label, and the matched label is used as the initial label for image classification, and the multi-level scene label is constructed based on the business scenario, classification requirements and the recognition ability of the large model, so the classification result can meet the classification requirements in complex business scenarios; after the initial label is obtained, the initial label can be further screened to obtain the target label of the first image, so the accuracy of the classification result can be further improved. The image classification method provided in the embodiment of the present application can be applied to intelligent classification of images in complex scenarios, and can quickly and accurately obtain classification results, effectively save labor costs, and improve classification efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0011] Figure 1 It is a flowchart of an image classification method based on a large model in one embodiment of the present application; Figure 2 is a schematic diagram of a multi-level scene label according to an embodiment of the present application; Figure 3 It is a flowchart of an image classification method based on a large model in one embodiment of the present application; Figure 4 is a schematic diagram of a multi-level universal label according to an embodiment of the present application; Figure 5 is a schematic diagram of a multi-level scene label according to an embodiment of the present application; Figure 6 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application; Figure 7 It is a structural diagram of an image classification device based on a large model in one embodiment of the present application. DETAILED DESCRIPTION

[0012] With the rise of artificial intelligence and big models, data and algorithms have become key elements of new productivity, and data classification is a hot topic in current data research. With the continuous development of advanced technologies such as image segmentation, image recognition, and neural networks, improving the intelligence and accuracy of image data classification is a research hotspot. Current image classification methods usually do not meet the classification needs in complex scenarios, which is mainly manifested in two aspects: first, traditional classification methods mainly rely on manual operations and cannot meet the needs of large-scale, intelligent processing, and have great limitations in processing efficiency and accuracy. Second, current classification methods cannot quickly classify specific business needs and complex business scenarios. Data classification and data use processes are separated, and simple classification is only based on target objects in images or videos.

[0013] The embodiments of the present application provide an image classification method, device, electronic device and storage medium based on a large model. The image classification method includes: obtaining a first image to be classified; establishing a multi-level scene label according to the business scenario, classification requirements and recognition capability of the large model; analyzing the first image and the scene label of the minimum level in the multi-level scene label through the large model to obtain an initial label of the first image, the large model uses a variety of image recognition technologies to perform image recognition, and the initial label of the first image includes multiple groups of labels, each group of labels is obtained by matching a label recognized by an image recognition technology with a scene label of the minimum level in the multi-level scene label; screening the initial label of the first image, and determining the screening result as the target label of the first image.

[0014] In this way, when classifying the first image, since the first image can be recognized at multiple levels through the large model (that is, multiple image recognition technologies are used to recognize the first image), and subsequent image classification operations are performed based on the recognized labels, the accuracy of subsequent classification results can be improved; after the label is recognized through the large model, the recognized label can be matched with the pre-established multi-level scene label, and the matched label is used as the initial label for image classification, and the multi-level scene label is constructed based on the business scenario, classification requirements and the recognition ability of the large model, so the classification result can meet the classification requirements in complex business scenarios; after the initial label is obtained, the initial label can be further screened to obtain the target label of the first image, so the accuracy of the classification result can be further improved. The image classification method provided in the embodiment of the present application can be applied to the intelligent classification of images in complex scenes, and can quickly and accurately obtain classification results, effectively save labor costs, and improve classification efficiency.

[0015] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in this application will be described clearly and completely below in conjunction with the drawings in one or more embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0016] The terms "first", "second", etc. in the present application and claims are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the data used in this way can be interchanged where appropriate so that the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in the present application and claims means at least one of the connected objects, and the character " / " generally means that the objects associated with each other are in an "or" relationship.

[0017] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0018] Figure 1 It is a flowchart of an image classification method based on a large model according to an embodiment of the present application. Figure 1 The image classification method shown is described as follows.

[0019] S12: Acquire a first image to be classified.

[0020] The first image may be an image captured by an image acquisition device in any scene. When acquiring the first image, an image captured by the image acquisition device in real time may be acquired, or a historical image captured by the image acquisition device may be acquired, which is not specifically limited here.

[0021] Optionally, in some embodiments, after the first image is acquired, the first image may be preprocessed to obtain an image that meets the requirements and is convenient for subsequent classification. For example, the first image may be automatically cleaned to remove repeated and ambiguous data.

[0022] S14: Establish multi-level scenario labels based on business scenarios, classification requirements, and recognition capabilities of large models.

[0023] When classifying images based on a large model, in order to meet the classification requirements in complex business scenarios and take into account the recognition capabilities of the large model (for example, how many types of labels the large model can analyze), multi-level scene labels can be established based on business scenarios, classification requirements, and the recognition capabilities of the large model. In other words, the actual scenarios involved in image classification, the recognition capabilities of the large model, and the data requirements for algorithm optimization training can be comprehensively considered to summarize and refine multi-level scene labels.

[0024] Multi-level scene tags are a tag system consisting of two or more levels of scene tags. The number and levels of tags can be expanded and flexibly configured as needed. The next-level tags can be obtained by subdividing the previous-level tags. The next-level tags belong to the previous-level tags. The smallest level of tags (that is, the smallest level of tags) is the dimension that can be collected or analyzed by the large model.

[0025] like Figure 2As shown. The multi-level scene labels include first-level labels, second-level labels, ..., and N-level labels. The first-level labels include label category one, label category two, ..., and label category N. After refining each label category in the first-level labels, the second-level labels can be obtained. The second-level labels include second-level labels 1, 2, ..., and N corresponding to label category one, second-level labels 1, 2, ..., and N corresponding to label category two, and second-level labels 1, 2, ..., and N corresponding to label category N. After refining each second-level label, the third-level label can be obtained. And so on, N-level labels can be obtained. For example, if the first-level label is A, the second-level labels are A1...A n , the third-level label is a 11 ……a 1n , a 21 ……a 2n , ..., a n1 ……a nn , and so on, there are multiple levels of scene business labels A={A1……A n}={(a 11 ……a 1n ), (a 21 ……a 2n ),……,(a n1 ……a nn )}.

[0026] It should be noted that in actual applications, a scene labeling system with more levels and more labels can be set up in combination with actual scenarios, and no specific limitation is made here.

[0027] S16: The first image and the scene label of the minimum level in the multi-level scene label are analyzed by the large model to obtain the initial label of the first image. The large model uses multiple image recognition technologies to perform image recognition. The initial label of the first image includes multiple groups of labels. Each group of labels is obtained by matching the label recognized by an image recognition technology with the scene label of the minimum level in the multi-level scene label.

[0028] After acquiring the first image and constructing the multi-level scene labels, the first image and the multi-level scene labels can be sent to the big model. Among them, the big model can be a currently open source or mature big model, such as the model of identifying all things developed by Tongyi Qianwen cn-clip, etc. There is no specific limitation on the big model used here. When sending the multi-level scene labels to the big model, the scene labels of the smallest level in the multi-level scene labels can be sent to the big model so that the big model can perform a more fine-grained analysis, which is convenient for obtaining refined classification results later. For example, the multi-level scene labels have two levels, namely the first-level scene labels and the second-level scene labels, and the second-level scene labels can be sent to the big model.

[0029] After the first image and the scene label of the minimum level in the multi-level scene label are sent to the large model, the large model can be used to analyze the first image and the scene label of the minimum level in the multi-level scene label to obtain the initial label of the first image. The large model in the embodiment of the present application can use a variety of image recognition technologies to perform image recognition. When the large model is used to analyze the first image and the scene label of the minimum level in the multi-level scene label, specifically, the multiple image recognition technologies can be used to perform image recognition on the first image respectively to obtain recognition results corresponding to the multiple image recognition technologies, and the recognition results can be the recognized labels. Afterwards, for each label recognized by the image recognition technology, it can be matched with the scene label of the minimum level in the multi-level scene label, and the matched labels can be determined as a group of labels corresponding to the image recognition technology, thereby obtaining multiple groups of labels corresponding to the multiple image recognition technologies, and the multiple groups of labels are the initial labels of the first image.

[0030] For example, assuming that the large model uses the image recognition technology 1 to identify the first image, the recognition results are label 1, label 2 and label 3, and that the image recognition technology 2 is used to identify the first image, the recognition results are label 2, label 5 and label 6, and the minimum level scene labels in the multi-level scene labels include label 2, label 3 and label 6. Then label 2 and label 3 can be determined as a group of labels corresponding to image recognition technology 1, and label 2 and label 6 can be determined as a group of labels corresponding to image recognition technology 2. The initial labels of the first image are {label 2, label 3}, {label 2, label 6}.

[0031] In some embodiments, the multiple image recognition technologies used by the large model may include semantic segmentation and feature extraction technology and target detection technology. In this case, the large model analyzes the first image and the scene label of the minimum level in the multi-level scene label to obtain the initial label of the first image, which may include: Performing semantic segmentation and feature extraction on the first image to obtain a multi-scale feature vector; obtaining a first label according to the multi-scale feature vector identification; matching the first label with a scene label of a minimum level in the multi-level scene labels, and determining the matching labels as a first group of labels; Performing object detection on the first image to obtain a second label; matching the second label with a scene label of a minimum level in the multi-level scene labels, and determining the matching labels as a second set of labels; The first set of labels and the second set of labels are determined as initial labels of the first image.

[0032] The above description uses the image recognition technology as semantic segmentation and feature extraction technology and target detection technology as examples. In other implementations, other image recognition technologies may also be used, and the image recognition technology is not specifically limited here.

[0033] In the embodiment of the present application, the multi-level scene label can be extended. When the multi-level scene label is extended, the first image and the scene label of the minimum level in the multi-level scene label are analyzed by the large model to obtain the initial label of the first image, which may include: The first image and the extended scene label are analyzed by the large model to obtain an incremental label of the first image, wherein the incremental label includes multiple groups of labels, and each group of labels is obtained by matching a label identified by an image recognition technology with the extended scene label; The incremental labels are determined as the initial labels of the first image.

[0034] In other words, when analyzing multi-level scene labels through a large model, it is only necessary to analyze the labels once in full. After the labels are expanded, it is only necessary to analyze the expanded scene labels (that is, the newly added scene labels). In this way, the consumption of computing resources by the large model can be effectively reduced, the analysis efficiency can be improved, and the classification efficiency can be improved.

[0035] In some embodiments, after the large model analyzes the first image and the scene labels of the minimum level in the multi-level scene labels, in addition to obtaining the initial labels of the first image, the confidence of each label in the initial labels can also be obtained. Taking the initial labels of the first image as {label 2, label 3} and {label 2, label 6} as an example, the large model can obtain the confidence of each of the four labels while obtaining these four labels.

[0036] S18: Screening the initial label of the first image, and determining the screening result as the target label of the first image.

[0037] The initial label of the first image can be regarded as the initial classification result of the first image, and its accuracy may not meet the requirements. In order to improve the accuracy of the classification result, the initial label of the first image can be screened to obtain the final classification result of the first image, that is, the initial label of the first image is screened, and the screening result is determined as the target label of the first image, which can also be expressed as the target scene label of the first image.

[0038] In some implementations, screening the initial label of the first image and determining the screening result as the target label of the first image may include: Determine an intersection of the multiple sets of labels as a first target label for the first image; For each group of labels, determining a label whose confidence is greater than or equal to a first threshold as a second target label of the first image; The first target label and the second target labels corresponding to each of the plurality of sets of labels are determined as the target labels of the first image.

[0039] Specifically, if the multiple image recognition technologies are specifically two image recognition technologies, the intersection of the two groups of labels corresponding to the two image recognition technologies can be determined as the first target label, that is, the label that can be recognized by both image recognition technologies is determined as the first target label. If the multiple image recognition technologies are specifically N (N is an integer greater than or equal to 3) image recognition technologies, the intersection of the N groups of labels corresponding to the N image recognition technologies can be determined as the first target label, or the intersection of at least two groups of labels corresponding to at least two of the image recognition technologies can be determined as the first target label, that is, the label that can be recognized by these N image recognition technologies is used as the first target label, or the label that can be recognized by at least two of the image recognition technologies is used as the first target label.

[0040] For a set of labels corresponding to each image recognition technology, the labels whose confidence is greater than or equal to the first threshold can be determined as the second target labels. The size of the first threshold can be set according to actual needs and is not specifically limited here. In addition, for different image recognition technologies, the first threshold can be the same or different, which is not specifically limited here.

[0041] After the first target tag and the second target tag are obtained, the first target tag and the second target tag may be determined as the target tags of the first image.

[0042] For example, there are two image recognition technologies. The initial labels of the first image obtained by these two image recognition technologies are {label 2, label 3} and {label 2, label 6}. Then, label 2 can be determined as the first target label. For the remaining labels 3 and 6, assuming that the confidence of label 3 is greater than the threshold and the confidence of label 6 is less than the threshold, label 3 can be determined as the second target label, and the final target labels are label 2 and label 3.

[0043] Since labels that can be identified by multiple image recognition technologies and labels with higher confidence in each group of labels can be determined as target labels for the first image, the accuracy of the target labels can be improved, thereby improving the accuracy of the classification results.

[0044] In some implementations, after obtaining the target labels of the first image, the target labels may be sorted and the sorting results may be output. Specifically, the sorting may include: sorting the target labels of the first image according to the priority and confidence of the labels; Output the ranking result of the target labels of the first image.

[0045] The priority of a tag can be determined based on the number and accuracy of image recognition technologies that can identify the tag (accuracy can characterize the recognition ability of the image recognition technology. The stronger the recognition ability, the higher the accuracy of the image recognition technology). The more image recognition technologies that can identify the tag, the higher the priority of the tag. The higher the accuracy of the image recognition technology that identifies the tag, the higher the priority of the tag. The confidence can be the recognition confidence of the image recognition technology for the tag. When sorting the target tags of the first image based on the priority and confidence of the tags, the tags with higher priority can be sorted first, and the tags with higher confidence can be sorted first.

[0046] For example, the target labels include label 1, label 2, label 3, and label 4. Label 1 is identified by two image recognition technologies, and label 2, label 3, and label 4 are all identified by one image recognition technology. Among them, label 2 and label 3 are identified by image recognition technology 1, and the confidence of label 3 is higher than the confidence of label 2. Label 4 is identified by image recognition technology 2, and the accuracy of image recognition technology 1 is higher than the accuracy of image recognition technology 2. When sorting the target labels, since label 1 is identified by two image recognition technologies, and the remaining labels are identified by one image recognition technology, label 1 has the highest priority and can be sorted first. For label 2, label 3, and label 4, since the accuracy of image recognition technology 1 for identifying label 2 and label 3 is higher than the accuracy of image recognition technology for identifying label 4, label 2 and label 3 are sorted before label 4 and after label 1. For label 2 and label 3, since the confidence of label 3 is higher, label 3 is sorted before label 2. In this way, after sorting, the final sorting result is: label 1, label 3, label 2, label 4.

[0047] In some embodiments, when classifying the first image, in addition to determining the scene label of the first image, a general label of the first image may also be determined. That is, the target label of the first image finally determined may include a target scene label and a target general label, and the target general label may be, for example, time, space, data type, data source, etc.

[0048] When determining the target scene label of the first image, specifically, the following steps may be included: Establish multi-level universal labels based on the data information obtained during image data collection; Determine a universal label of the first image according to the data information collected when acquiring the first image; The universal label of the first image is matched with the universal label of the minimum level in the multi-level universal labels, and the matched label is determined as the target label of the first image.

[0049] Multi-level universal tags are a tag system consisting of two or more levels of universal tags. The number and level of tags can be expanded and flexibly configured as needed. The architecture can be the same as that of multi-level scene tags. For details, see Figure 2 When establishing multi-level universal tags, optionally, the data information that can be obtained during data collection can be structured and stored to form a first-level tag, including but not limited to time, space, data type, data source and other first-level category dimensions, and then expanded downward to establish a second-level tag or more levels of tags. For example, the time dimension can be divided into sample generation time, sample import time, sample season, etc.

[0050] When classifying universal labels, the universal label of the first image can be determined based on the data information collected when the first image is acquired (such as the acquisition time, data source, etc.). The universal label here is the initial universal label. After obtaining the initial universal label, it can be matched with the universal label of the minimum level in the multi-level universal labels, and the matched labels are determined as the final universal labels, and the final universal label is determined as the target label of the first image. The target label of the first image is the target universal label of the first image.

[0051] Optionally, in some implementations, after the initial universal label of the first image is determined, it may not be matched with the multi-level universal labels, but the initial universal label may be directly used as the final target universal label of the first image.

[0052] In some implementations, after obtaining the target label (scene label, or scene label and general label) of the first image, the following steps may also be performed: A tag retrieval function for the first image is provided based on the target tag of the first image.

[0053] The tag search function may include a search function based on a single target tag or a combined search function based on multiple target tags. Thus, when performing image search, the first image can be retrieved according to one or more target tags of the first image, thereby visually finding the required image data with tags by searching as needed.

[0054] The image classification method provided in the embodiment of the present application, when classifying the first image, can improve the accuracy of the subsequent classification results because the first image can be recognized at multiple levels through the large model (that is, multiple image recognition technologies are used to recognize the first image), and subsequent image classification operations are performed based on the recognized labels; after the labels are recognized through the large model, the recognized labels can be matched with the pre-established multi-level scene labels, and the matched labels are used as initial labels for image classification, and the multi-level scene labels are constructed based on business scenarios, classification requirements and the recognition capabilities of the large model, so the classification results can meet the classification requirements in complex business scenarios; after the initial labels are obtained, the initial labels can be further screened to obtain the target labels of the first image, so the accuracy of the classification results can be further improved. The image classification method provided in the embodiment of the present application can be applied to intelligent classification of images in complex scenarios, can quickly and accurately obtain classification results, effectively save labor costs, and improve classification efficiency.

[0055] Figure 3 It is a flowchart of an image classification method based on a large model according to an embodiment of the present application. Figure 3 The illustrated embodiment takes the image recognition technology adopted by the large model as the semantic segmentation and feature extraction technology and the target detection technology as an example for explanation, and specifically includes the following steps.

[0056] Step 1: Get the first image to be classified.

[0057] When acquiring the first image, the camera can be used to collect the original image data of the high-point video surveillance in real time across the country to ensure the diversity of the data and complete the data collection. Among them, the collected image data can include multi-dimensional information such as the collection time, collection area, and data type. Afterwards, the data can be automatically cleaned to remove duplicate and ambiguous data.

[0058] Step 2: Create multi-level scene labels and multi-level general labels.

[0059] Multi-level scene labels can be constructed based on business scenarios, classification requirements, and the recognition capabilities of large models, including but not limited to primary category dimensions such as natural scenes, objects, and animals, and secondary labels such as rivers, mountains, and roads. Multi-level general labels can be constructed based on the data information obtained during image data collection. For example, the data information that can be obtained during data collection can be structured and stored to form labels, including but not limited to primary category dimensions such as time, space, data type, and data source, and then expanded downward to establish secondary labels or more levels of labels. For example, the time dimension can be divided into sample generation time, sample import time, sample season, etc.

[0060] Taking the urban space governance scenario as an example, when establishing a label system, two levels of general labels and two levels of scenario labels can be set separately. Figure 4 As shown, Figure 4 In the above, the general label is used as the first-level label from the three dimensions of time, space and type. Each first-level label can be divided into multiple second-level labels. Figure 5 As shown, Figure 5 In the model, scene labels are divided into six dimensions: motor vehicles, non-motor vehicles, urban issues, industrial impacts, urban facilities, and human activities. Each first-level label can be divided into multiple second-level labels.

[0061] Step 3: Perform semantic segmentation and feature extraction on the first image according to the large model, and perform target detection on the first image according to the large model, match the recognized label with the minimum level scene label in the multi-level scene label, and obtain the initial label and confidence of the first image.

[0062] In step 3, a data classification model can be constructed using image feature extraction, semantic segmentation, and multi-scene target recognition algorithms. The labels to be recognized by the large model are first associated with the data before analysis. According to business needs, the first image to be classified and the established multi-level scene labels are sent to the large model, and the labels only need to be sent to the minimum level. The large model analyzes and stores the results based on the issued labels. Specifically, the large model can use semantic segmentation technology to automatically segment the first image, extract the multi-scale feature vector of the first image, and obtain one or more labels based on the multi-scale feature vector recognition, and then match these labels with the labels issued to the large model, and store the feature vectors and the matching label names and confidences. At the same time, the multi-scene target recognition algorithm can also be used to detect the targets in the first image one by one, and one or more labels are obtained based on the detection, and then these labels are matched with the labels issued to the large model, and the matching label names and confidences are stored. Among them, the label system of multi-level scene labels is expandable. When the label system is expanded, when performing image classification, only incremental labels can be sent to the large model for analysis to achieve incremental addition of labels, which saves more computing power resources than full label analysis.

[0063] Step 4: Filter and sort the initial labels of the first image according to the rule engine to obtain the target scene labels of the first image.

[0064] In step 4, different priorities can be assigned to labels identified by different image recognition technologies, and a rule engine can be designed in combination with confidence, feature information, etc. The rule engine is used to perform comprehensive calculations on the analysis results to automatically generate corresponding target scene labels for the first image. Among them, the rule engine can include the first principle, the second principle, and the third principle. The first principle is that all labels identified by scene-based multi-target recognition and feature vector and semantic segmentation are retained. The second principle is that only labels with a confidence greater than 0.8 for scene-based multi-target recognition are retained. The third principle is that only labels with a confidence greater than 0.3 matched by feature vector semantic segmentation are retained. The corresponding labels that meet the above principles are automatically generated and are still sorted according to this principle in visual display and retrieval sorting. Among them, when sorting labels, the labels retained according to the first principle are sorted at the front (highest accuracy), the labels retained according to the second principle are sorted in the middle, and the labels retained according to the third principle are sorted at the end.

[0065] Step 5: Determine the target universal label of the first image based on the data information collected when acquiring the first image and the multi-level universal labels.

[0066] In step 5, the initial universal label of the first image can be determined based on the data information collected when the first image is acquired, and then the initial universal label of the first image can be matched with the universal label of the minimum level in the multi-level universal labels, and the matched label can be determined as the target universal label of the first image.

[0067] Step 6: Output the target scene label and the target general label of the first image, and provide a retrieval function for the first image based on the target scene label and the target general label.

[0068] In summary, the image classification method provided in the embodiment of the present application has at least the following technical effects: (1) The embodiment of the present application structures the sample data through "label system + big model + rule engine", uses the actual needs of algorithm optimization training and business scenarios to formulate an extensible label system, and uses the rule engine to automatically generate corresponding labels after performing image feature extraction, semantic segmentation and scenario-based multi-target recognition through multi-level big model analysis. This method can be applied to the intelligent classification of image data in complex scenes to effectively save labor costs. This method can be used to convert the original image data into a high-quality data set, and the sample data set can be sold to the outside world to realize data monetization.

[0069] (2) The embodiment of the present application provides a standard for establishing a hierarchical multi-level label system, which supports multi-level label settings such as label categories and label subcategories. The required labels can be set independently according to business and scenario requirements. The set labels are automatically sent to the multi-level large model for analysis. The incremental labels can be sent incrementally to the large model for analysis without sending them in full each time, saving computing resources. The embodiment of the present application is suitable for the classification of image data in various scenarios. The label system can be adjusted at any time according to needs, without manual classification, and can meet the needs of large-scale intelligent data classification, greatly improving processing efficiency and accuracy.

[0070] (3) Advanced technologies such as feature vector extraction and storage, semantic segmentation, and scenario-based multi-target recognition in large models provide intelligent means for data classification, improving the efficiency and accuracy of data classification. This method uses image feature extraction, semantic segmentation, and multi-scenario target recognition algorithms to build a large data classification model, which is the direction of large model application. The model automatically segments the image data using semantic segmentation technology, extracts multi-scale feature vectors of the image, matches them with preset labels, and stores the feature vectors, label names, and confidences; at the same time, it uses multi-scenario target recognition algorithms to detect the targets in the image one by one, and stores the label names and confidences that match the preset labels. When the label system is expanded, only incremental labels can be sent for analysis to achieve incremental addition of image labels, which saves more computing resources than full label analysis.

[0071] (4) This method assigns different priorities to labels identified by different technologies, designs a rule engine based on confidence, feature information, etc., uses the rule engine to perform comprehensive calculations on the analysis results, and automatically generates corresponding labels for the images. The rule engine is the basis for whether labels are generated and searched and sorted.

[0072] (5) This method can provide high-quality data sets for algorithm optimization training through visual display and automatic retrieval. The provided retrieval engine supports multiple label combination queries and can meet the optimization training needs of multiple algorithms. The required labeled sample data can be visually found by searching on demand, thus realizing rapid retrieval of the required data in massive data.

[0073] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0074] Figure 6This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 6 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0075] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0076] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0077] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming an image classification device based on a large model at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations: Acquire a first image to be classified; Establish multi-level scenario labels based on business scenarios, classification requirements, and recognition capabilities of large models; The first image and the scene label of the minimum level in the multi-level scene labels are analyzed by the large model to obtain an initial label of the first image, wherein the large model uses multiple image recognition technologies to perform image recognition, and the initial label of the first image includes multiple groups of labels, each group of labels is obtained by matching a label recognized by an image recognition technology with a scene label of the minimum level in the multi-level scene labels; The initial label of the first image is screened, and the screening result is determined as the target label of the first image.

[0078] The above application Figure 6 The method performed by the large model-based image classification device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in this application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in this application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0079] The electronic device may also perform Figure 1 and Figure 3 The method is used to implement the image classification device based on the large model. Figure 1 and Figure 3 The functions in the illustrated embodiments will not be described in detail in this application.

[0080] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0081] The present application also proposes a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 1 and Figure 3 The method of the embodiment shown is specifically used to perform the following operations: Acquire a first image to be classified; Establish multi-level scenario labels based on business scenarios, classification requirements, and recognition capabilities of large models; The first image and the scene label of the minimum level in the multi-level scene labels are analyzed by the large model to obtain an initial label of the first image, wherein the large model uses multiple image recognition technologies to perform image recognition, and the initial label of the first image includes multiple groups of labels, each group of labels is obtained by matching a label recognized by an image recognition technology with a scene label of the minimum level in the multi-level scene labels; The initial label of the first image is screened, and the screening result is determined as the target label of the first image.

[0082] Figure 7 This is a schematic diagram of the structure of an image classification device 70 based on a large model according to an embodiment of the present application. Figure 7 In a software implementation, the large model-based image classification device 70 may include: an acquisition module 71, a building module 72, an analysis module 73 and a screening module 74, wherein: An acquisition module 71 acquires a first image to be classified; Establishing module 72, establishing multi-level scene labels according to business scenarios, classification requirements and recognition capabilities of large models; An analysis module 73 analyzes the first image and the scene label of the minimum level in the multi-level scene labels by using the large model to obtain an initial label of the first image, wherein the large model uses multiple image recognition technologies to perform image recognition, and the initial label of the first image includes multiple groups of labels, each group of labels is obtained by matching a label recognized by an image recognition technology with a scene label of the minimum level in the multi-level scene labels; The screening module 74 screens the initial labels of the first image and determines the screening results as the target labels of the first image.

[0083] In some embodiments, the plurality of image recognition technologies include semantic segmentation and feature extraction technology and target detection technology; the analysis module 73 analyzes the first image and the scene label of the minimum level in the multi-level scene labels through the large model to obtain an initial label of the first image, including: Performing semantic segmentation and feature extraction on the first image to obtain a multi-scale feature vector; identifying and obtaining a first label based on the multi-scale feature vector; matching the first label with a scene label of a minimum level in the multi-level scene labels, and determining the matching labels as a first group of labels; Performing object detection on the first image to obtain a second label; matching the second label with a scene label of a minimum level in the multi-level scene labels, and determining the matching labels as a second group of labels; The first set of labels and the second set of labels are determined as initial labels of the first image.

[0084] In some implementations, the multi-level scene labels are expandable; when the multi-level scene labels are expanded, the analysis module 73 analyzes the first image and the scene labels of the minimum level in the multi-level scene labels through the large model to obtain an initial label for the first image, including: Analyzing the first image and the extended scene label by the large model to obtain an incremental label of the first image, wherein the incremental label includes multiple groups of labels, and each group of labels is obtained by matching a label identified by an image recognition technology with the extended scene label; The incremental label is determined as an initial label of the first image.

[0085] In some implementations, after analyzing the first image and the scene label of the minimum level in the multi-level scene labels by the large model, the confidence of each label in the initial labels is also obtained; the screening module 74 screens the initial labels of the first image and determines the screening results as the target labels of the first image, including: Determine an intersection of the multiple groups of labels as a first target label of the first image; For each group of labels, determining a label whose confidence is greater than or equal to a first threshold as a second target label of the first image; The first target tag and the second target tags corresponding to each of the plurality of groups of tags are determined as target tags for the first image.

[0086] In some implementations, after obtaining the target label of the first image, the screening module 74 further includes: sorting the target labels of the first image according to the priority and confidence of the labels; Outputting a ranking result of target labels of the first image; The priority of the tag is determined according to the number and accuracy of image recognition technologies that can recognize the tag.

[0087] In some embodiments, the establishment module 72 further establishes multi-level universal labels based on the data information obtained during image data acquisition; The analysis module 73 also determines the general label of the first image based on the data information collected when acquiring the first image; matches the general label of the first image with the general label of the minimum level in the multi-level general labels, and determines the matched label as the target label of the first image.

[0088] In some embodiments, the apparatus 70 further comprises a retrieval module; The retrieval module provides a tag retrieval function for the first image based on a target tag of the first image, wherein the tag retrieval function includes a retrieval function based on a single target tag or a combined retrieval function based on multiple target tags.

[0089] The large model-based image classification device 70 provided in this application can also perform Figure 1 and Figure 3 The method is implemented by the image classification device 70 based on the large model. Figure 1 and Figure 3 The functions of the illustrated embodiments will not be described in detail in this application.

[0090] The present application also proposes a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute part or all of the steps in the above-mentioned large model-based image classification method embodiment.

[0091] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0092] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0093] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0094] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0095] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A large model-based image classification method, characterized in that: include: Acquire a first image to be classified; Establish multi-level scenario labels based on business scenarios, classification requirements, and recognition capabilities of large models; The first image and the scene label of the minimum level in the multi-level scene labels are analyzed by the large model to obtain an initial label of the first image, wherein the large model uses multiple image recognition technologies to perform image recognition, and the initial label of the first image includes multiple groups of labels, each group of labels is obtained by matching a label recognized by an image recognition technology with a scene label of the minimum level in the multi-level scene labels; The initial label of the first image is screened, and the screening result is determined as the target label of the first image.

2. The method according to claim 1, characterized in that The multiple image recognition technologies include semantic segmentation and feature extraction technologies and target detection technologies; the analyzing the first image and the scene label of the minimum level in the multi-level scene labels by the large model to obtain the initial label of the first image includes: Performing semantic segmentation and feature extraction on the first image to obtain a multi-scale feature vector; identifying and obtaining a first label based on the multi-scale feature vector; matching the first label with a scene label of a minimum level in the multi-level scene labels, and determining the matching labels as a first group of labels; Performing object detection on the first image to obtain a second label; matching the second label with a scene label of a minimum level in the multi-level scene labels, and determining the matching labels as a second group of labels; The first set of labels and the second set of labels are determined as initial labels of the first image.

3. The method according to claim 1, characterized in that The multi-level scene label is expandable; when the multi-level scene label is expanded, the first image and the scene label of the minimum level in the multi-level scene label are analyzed by the large model to obtain the initial label of the first image, including: Analyzing the first image and the extended scene label by the large model to obtain an incremental label of the first image, wherein the incremental label includes multiple groups of labels, and each group of labels is obtained by matching a label identified by an image recognition technology with the extended scene label; The incremental label is determined as an initial label of the first image.

4. The method according to claim 1, characterized in that After analyzing the first image and the scene label of the minimum level in the multi-level scene label by the large model, the confidence of each label in the initial label is obtained; the initial label of the first image is screened and the screening result is determined as the target label of the first image, including: Determine an intersection of the multiple groups of labels as a first target label of the first image; For each group of labels, determining a label whose confidence is greater than or equal to a first threshold as a second target label of the first image; The first target tag and the second target tags corresponding to each of the plurality of groups of tags are determined as target tags for the first image.

5. The method according to any one of claims 1 to 4, characterized in that: After obtaining the target label of the first image, the method further includes: sorting the target labels of the first image according to the priority and confidence of the labels; Outputting a ranking result of target labels of the first image; The priority of the tag is determined according to the number and accuracy of image recognition technologies that can recognize the tag.

6. The method according to claim 1, characterized in that The method further comprises: Establish multi-level universal labels based on the data information obtained during image data collection; Determining a universal label of the first image according to data information collected when acquiring the first image; The general label of the first image is matched with the general label of the minimum level in the multi-level general labels, and the matched label is determined as the target label of the first image.

7. The method according to claim 1 or 6, characterized in that The method further comprises: A tag retrieval function for the first image is provided based on the target tag of the first image, wherein the tag retrieval function includes a retrieval function based on a single target tag or a combined retrieval function based on multiple target tags.

8. An image classification device based on a large model, characterized in that: include: An acquisition module, acquiring a first image to be classified; Establish modules and build multi-level scene labels based on business scenarios, classification requirements, and recognition capabilities of large models; an analysis module, which analyzes the first image and the scene label of the minimum level in the multi-level scene labels by using the large model to obtain an initial label of the first image, wherein the large model uses multiple image recognition technologies to perform image recognition, and the initial label of the first image includes multiple groups of labels, each group of labels is obtained by matching a label recognized by an image recognition technology with a scene label of the minimum level in the multi-level scene labels; The screening module screens the initial label of the first image and determines the screening result as the target label of the first image.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 7.

Images