Image processing method and related device

Abstract

The invention discloses an image processing method and a related device. The method comprises the following steps: determining a text information category of each image according to the image difference of a plurality of images and / or the number of classification labels of a current image classification task; extracting text information corresponding to the text information category of each image; determining a text coding feature of each image according to the text information of each image; and determining a classification label of each image according to the text coding feature of each image and the image coding feature of each image. The text information category can be dynamically selected, and the accuracy of the classification task is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an image processing method and related apparatus. Background Technology

[0002] With the continuous development of computer technology, the demand for image processing is increasing. Image classification is a commonly used image processing method. By classifying images, images in an image collection are divided into multiple classes composed of similar images, thereby realizing the organization of image information.

[0003] Currently, image classification is based on uniformly extracted text information. However, since the characteristics of different image classification tasks may vary greatly in real-world scenarios, the extracted text information may not match the classification task, leading to inaccurate classification results. Summary of the Invention

[0004] This application provides an image processing method and related apparatus, which helps to improve the accuracy and versatility of image classification tasks.

[0005] In a first aspect, embodiments of this application provide an image processing method, including:

[0006] The text information category for each image is determined based on the image differences and / or the number of classification labels among multiple images;

[0007] Extract the text information corresponding to the text information category of each image;

[0008] The text encoding features of each image are determined based on the text information of each image; the classification label of each image is determined based on the text encoding features and the image encoding features of each image.

[0009] It can be seen that by dynamically selecting the text information category based on image differences and / or the number of classification labels, the matching degree between the extracted text information and the classification task can be improved, thereby improving the accuracy and versatility of image classification tasks.

[0010] Secondly, embodiments of this application provide an image processing apparatus, including:

[0011] The first determining unit is used to determine the text information category of each image based on the image differences and / or the number of classification labels of multiple images;

[0012] An extraction unit is used to extract text information corresponding to the text information category of each image;

[0013] The second determining unit is used to determine the text encoding features of each image based on the text information of each image;

[0014] The third determining unit is used to determine the classification label of each image based on the text encoding features and the image encoding features of each image.

[0015] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing execution instructions, the memory storing one or more programs; when the processor executes the execution instructions stored in the memory, the processor executes the method described in the first aspect.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing an energy data management program, including execution instructions, which, when executed by a processor of an electronic device, perform the method described in the first aspect.

[0017] Fifthly, embodiments of this application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of embodiments of this application. The computer program product may be a software installation package.

[0018] As can be seen, in this embodiment, the text information category of each image is first determined based on the image differences and / or the number of classification labels of multiple images; then, the text information corresponding to the text information category of each image is extracted; the text encoding features of each image are determined based on the text information of each image; finally, the classification label of each image is determined based on the text encoding features and the image encoding features of each image. This application dynamically selects the text information category based on the image differences and / or the number of classification labels, and extracts text information of different dimensions based on the determined text information category. Compared to the prior art, which only extracts text information of the same type, this application considers the influence of the relationship between the number of image differences and classification labels on the classification labels, making the extracted text information more closely matched to the classification task, thereby improving the accuracy and versatility of image classification tasks. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the architecture of an image processing system provided in an embodiment of this application;

[0021] Figure 2 This is a flowchart of an image processing method provided in an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of the structure of a comprehensive feature extraction model provided in an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of the structure of a text feature extraction model provided in an embodiment of this application;

[0024] Figure 5 This is a flowchart illustrating another image processing method provided in an embodiment of this application;

[0025] Figure 6 This is a functional unit block diagram of an image processing apparatus provided in an embodiment of this application;

[0026] Figure 7 This is a functional unit block diagram of another image processing device provided in the embodiments of this application;

[0027] Figure 8 This is a schematic diagram of the structure of an electronic device proposed in an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0029] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0030] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0031] With the continuous development of computer technology, the demand for image processing is increasing. Image classification is a commonly used image processing method, which divides images in an image collection into multiple classes composed of similar images, thereby organizing image information. Currently, image classification only extracts text information of the same type and classifies images based on this text information. However, due to the significant differences in images and classification labels between different image classification tasks in real-world scenarios, the current image classification mechanism cannot meet the requirements of both the universality of classification tasks and the accuracy of classification results.

[0032] To address the aforementioned problems, this application provides an image processing method and related apparatus. First, the number of image differences and / or classification labels among multiple images determines the text information category of each image. Then, the text information corresponding to the text information category of each image is extracted. Next, the text encoding features of each image are determined based on the text information. Finally, the classification label of each image is determined based on the text encoding features and the image encoding features of each image. This application dynamically selects the text information category based on the number of image differences and / or classification labels, and extracts text information of different dimensions based on the determined text information category. Compared to existing technologies that only extract text information of the same type, this method considers the influence of the relationship between the number of image differences and classification labels on the classification label, making the extracted text information more closely matched to the classification task, thereby improving the accuracy and versatility of image classification tasks.

[0033] The image processing method and related apparatus provided in this application can be applied to, for example... Figure 1 Please refer to the image processing system shown. Figure 1 , Figure 1This is a schematic diagram of the architecture of an image processing system provided in an embodiment of this application. The image processing system 100 includes a terminal 101 and a server 102. The terminal 101 can communicate with the server 102 via a network. The terminal 101 refers to a device used by the user, such as a smartphone or computer. In this solution, the terminal 101 is mainly responsible for acquiring a set of images to be classified and sending them to the server 102 for processing. The user can interact with the system through the terminal and send images to be classified, etc. The server 102 refers to a remote computer used to process large amounts of computing tasks and store data. In this solution, the server 102 classifies the set of images to be classified sent by the terminal 101 and returns the classification results to the terminal 101.

[0034] Accordingly, this application provides an image processing method and related apparatus, which will be described in detail below with reference to the accompanying drawings.

[0035] Please see Figure 2 , Figure 2 This is a flowchart of an image processing method provided in an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0036] S210, determine the text information category of each image based on the image differences and / or the number of classification labels of multiple images.

[0037] The images are those to be classified, which can be photographs of different scenes, such as landscapes, people, and buildings; images of different objects, such as fruits, furniture, and electronic products; or medical images, images of artworks, images of biological samples, images taken by surveillance cameras, etc. The number of classification labels corresponds to the current image classification task.

[0038] Specifically, the text information category includes at least one of the following: sentence-level description of basic image semantics, word-level description of general image semantics, and word-level description of deep image semantics; the basic image semantics is used to characterize the direct description of the scene constructed by some or all elements in the image, the general image semantics is used to characterize the physical characteristics and / or basic use characteristics of the scene, and the deep image semantics is used to characterize the derived use characteristics of the scene; wherein, each element has different descriptive attributes.

[0039] In this context, "elements in an image" refers to the distinguishable objects within the image, such as leaves, chairs, tables, and cups. "Basic image semantics" refers to the intuitive interpretation or description of the image's content. This can include identified elements, the image scene constructed from multiple elements, and the basic attribute descriptions of each element. It involves no additional knowledge, no implicit meaning, emotional expression, or subjective judgment; it simply describes what is in the image and how they are arranged. For example, if an image shows an apple on a table, its direct description might be "a red apple is on a wooden tabletop."

[0040] Image semantic summarization, based on natural language question-and-answer principles applied to visual images, involves understanding the image and then answering specific questions, such as: "What keywords describe this image?" The answers yield multiple inductive keywords that describe the image's main elements, their states, relationships, and the basic scene the image depicts. For example, for an image depicting an apple on a table, keywords might include: the apple has been bitten, and the table.

[0041] Deep semantic analysis of images, based on natural language question answering of visual images, uses specific targets or domain knowledge to extract derivative uses from the image's context. These uses are not the content directly displayed by the image, but rather uncover the image's potential, implicit, or symbolic meaning. For example, an image depicting an empty dining table might be used for an upcoming banquet.

[0042] As can be seen, this embodiment can improve the comprehensiveness of image semantic extraction.

[0043] Specifically, determining the text information category based on the image differences and the number of classification labels of multiple images includes: determining the average image difference of the multiple images; querying a preset set of text information categories using a parameter group as a query identifier, and obtaining the text information category that matches the parameter group, wherein the parameter group includes the average image difference and the number of classification labels, and the set of text information categories includes the correspondence between the parameter group and the text information category.

[0044] Image differences can be between elements within an image, or between scenes constructed from multiple elements. For example, scene differences between images: one image depicts a city street scene, while another depicts a rural field scene. Similarly, differences between elements in images: one image features a cat, while another depicts a dog. Classification labels refer to the different categories into which images are divided. For instance, an image classification task might provide four classification labels, classifying multiple images in a set as a beach, forest, conference room, and kitchen.

[0045] The mean image difference refers to the process of first calculating the difference between every two images to obtain multiple image differences, and then calculating the average of these differences.

[0046] The parameter set is used to evaluate the difficulty of the classification task. The difficulty can be divided into multiple difficulty levels, each corresponding to a text information category group, and each difficulty level also corresponds to a difficulty range. Specifically, after obtaining the mean image difference and the number of classification labels, these values ​​are normalized. Then, based on the weights of the mean image difference and the number of classification labels, the difficulty of the classification task corresponding to the current multiple images is obtained. Next, the corresponding difficulty level is found, and the appropriate text information category group is located within the preset correspondence between difficulty levels and text information category groups.

[0047] One approach to determining text information categories is based on the image differences and the number of classification labels across multiple images. Alternatively, a first round of text information category determination can be performed based on the number of classification labels, followed by a final text information category determined from the first round based on image differences. For example, when the number of classification labels is less than or equal to a preset number, after determining the text information categories including sentence-level descriptions of the image's basic semantics and word-level descriptions of the image's general semantics, the degree of difference between images is further analyzed. If the differences between multiple images to be classified are significant, the sentence-level descriptions of the image's basic semantics can be determined from the text information categories obtained from the number of classification labels as the final text information category. If the differences between multiple images to be classified are small, word-level descriptions of the image's deeper semantics can be added to the text information categories obtained from the number of classification labels as the final text information category.

[0048] As can be seen, this embodiment improves the flexibility of selecting text information types and increases the success rate and accuracy of text information category determination, thereby improving the success rate and accuracy of processing the current image classification task.

[0049] Specifically, determining the text information category based on the image differences of multiple images includes: determining the image differences between any two images among the multiple images to obtain multiple image differences; if the image difference with the largest value among the multiple image differences is less than a preset image difference, then the text information category is determined to include the sentence-level description of the image's basic semantics, the word-level description of the image's general semantics, and the word-level description of the image's deep semantics; if the image difference with the smallest value among the multiple image differences is greater than the preset image difference, then the text information category is determined to include the sentence-level description of the image's basic semantics.

[0050] In cases where only image differences can be obtained—that is, when the user has not provided the number of preset category labels, and the device cannot determine the number of category labels—the difference values ​​between any two images to be classified are calculated. The calculated maximum and minimum difference values ​​are compared with the preset image differences. If the maximum difference value is less than the preset image difference, it indicates that the differences between the images to be classified are small, requiring special natural language question-and-answer methods to obtain differences in the image description text. In this case, multiple dimensions of text information are needed, namely, sentence-level descriptions of basic image semantics, word-level descriptions of general image semantics, and word-level descriptions of deep image semantics. If the minimum difference value is greater than the preset image difference, it indicates that the differences between the images to be classified are large, meaning that the basic semantics of the images are already significantly different. In this case, image general semantics and deep image semantics are not required, and the corresponding text description can be determined based on the sentence-level description of basic image semantics. Furthermore, if the maximum difference value is not less than the preset image difference, or the minimum difference value is not greater than the preset image difference, the text information category can be determined by comparing the average difference of the images of multiple images with the preset image difference.

[0051] As can be seen, this embodiment can improve the success rate and accuracy of text information category determination.

[0052] Specifically, determining the text information category based on the number of classification tags includes: if the number of classification tags is greater than a preset number, then the text information category is determined to include sentence-level descriptions of the image's basic semantics, word-level descriptions of the image's general semantics, and word-level descriptions of the image's deep semantics; if the number of classification tags is less than or equal to a preset number, then the text information category is determined to include sentence-level descriptions of the image's basic semantics and word-level descriptions of the image's general semantics.

[0053] In cases where only the number of image category labels can be obtained—that is, when the user provides a preset number of category labels or the device determines the number of category labels—the problem involves determining the relationship between the number of category labels and the preset number. When the number of category labels is less than or equal to the preset number, the problem is relatively simple and intuitive, requiring less in-depth domain knowledge. For example, classifying a simple set of fruit images as red fruit and green fruit only requires understanding the basic concepts of color and how to classify based on color, without needing complex domain knowledge such as the functions of red and green fruit. Two different dimensions of text description can be determined based on sentence-level descriptions of basic image semantics and word-level descriptions of general image semantics. When the number of category labels exceeds the preset number, the problem becomes more complex and may require in-depth domain knowledge analysis. In this case, three different dimensions of text description can be determined based on sentence-level descriptions of basic image semantics, word-level descriptions of general image semantics, and word-level descriptions of deep image semantics.

[0054] As can be seen, this embodiment can improve the success rate and accuracy of text information category determination.

[0055] S220, extract the text information corresponding to the text information category of each image.

[0056] Specifically, the step of extracting text information corresponding to the text information category of each image includes: for each image's text information category, if the text information category includes a sentence-level description of the image's basic semantics, then identifying elements in the image; determining the scene of the image based on the identified elements; creating a direct description statement of the scene based on the scene vocabulary and the identified elements, and using the direct description statement of the scene as the sentence-level description text information of the image's basic semantics; if the text information category includes a sentence-level description of the image's basic semantics and a word-level description of the image's general semantics, then identifying elements in the image; determining the scene of the image based on the identified elements; creating a direct description statement of the scene based on the scene vocabulary and the identified elements, and using the direct description statement of the scene as the sentence-level description text information of the image's basic semantics; and creating... based on the scene's physical characteristics and / or basic usage characteristics... The basic usage descriptors of the scene are used as word-level text information for the general semantics of the image. If the text information category includes sentence-level descriptions of the basic semantics of the image, word-level descriptions of the general semantics of the image, and word-level descriptions of the deep semantics of the image, elements in the image are identified. The scene of the image is determined based on the elements. A direct description statement of the scene is created based on the scene vocabulary and the identified elements, and the direct description statement of the scene is used as sentence-level text information for the basic semantics of the image. Furthermore, basic usage descriptors of the scene are created based on the physical characteristics and / or basic usage characteristics of the scene, and the basic usage descriptors of the scene are used as word-level text information for the general semantics of the image. Finally, derived usage descriptors of the scene are created based on the determined derived usage characteristics of the scene, and the derived usage descriptors of the scene are used as word-level text information for the deep semantics of the image.

[0057] This can involve using a trained comprehensive feature extraction model to generate multi-layered text and extract multi-modal features from an image. Please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is a schematic diagram of the structure of a comprehensive feature extraction model provided in an embodiment of this application, such as... Figure 3As shown, the comprehensive feature extraction model can be an image-to-text model, including an image encoder and a text encoder. By inputting an image to be classified, image-encoded features and text-encoded features are obtained through the text encoder and image encoder, respectively. These obtained image-encoded features and text-encoded features are then fused to obtain a comprehensive representation, which is then output. The image encoder can be implemented using a Convolutional Neural Network (CNN), specifically the 16-layer VisualGeometry Group (VGG16) from Oxford University, or a ResNet residual network. The image encoder extracts image-related feature vectors. Specifically, the image can be encoded into a latent space, where image information is compressed and encoded into a set of numerical vectors or feature vectors, thus preserving key information and features from the original image. Furthermore, the feature vectors can include global image features and local image features. Global image features can be color histograms describing the color distribution in the image; texture features such as gray-level co-occurrence matrices characterizing texture roughness and contrast; or shape features describing the image contours. Local image features can be local binary patterns used to describe the texture features of local regions of an image; scale-invariant feature transforms; histograms of oriented gradients used to describe the shape and contour of objects in an image; or speed-up robust features, etc.

[0058] The text encoder in the image-to-text model can be implemented using a text feature extraction model that integrates natural language understanding and natural language generation tasks, and the text encoder in the contrastive language-image pre-training model (CLIP). By inputting an image to be classified, at least one text description for the image to be classified can be obtained according to the trained text feature extraction model. Then, the text encoder part in the CLIP model converts the text information output by each layer into a vector representation, that is, a feature vector that captures the important information in the text. The obtained text feature vector is then normalized to obtain the text encoded features.

[0059] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of a text feature extraction model provided in an embodiment of this application, such as... Figure 4As shown, the text feature extraction model can be a Bootstrapping Language-Image Pre-training (Blip2) model, or other similar models. The text feature extraction model structure mainly includes the following parts: an image encoder, a querying transformer (Q-Former), and a large language model. The image encoder, as a visual feature extractor, is kept frozen during pre-training to reduce computational cost and avoid catastrophic forgetting. The Q-Former is a trainable module in the BLIP2 model, a lightweight transformer that includes an image transformation submodule and a text transformation submodule, which share an attention layer. The image transformation submodule interacts with the image encoder to extract visual features, while the text transformation submodule acts as both a text encoder and decoder to process text input. The Q-Former uses a set of learnable query vectors to extract visual features from the image encoder. The large language model is used for text generation and is also kept frozen during pre-training. The large-scale language model includes multiple text generation layers, such as sentence-level description of basic image semantics, word-level description of generalized image semantics, and word-level description of deep image semantics. This enables the generation of more diverse and in-depth text information, focusing on language features in different dimensions.

[0060] The BLIP2 model is trained in two stages. The first stage is visual-language representation learning, where the Q-former is connected to an image encoder and pre-trained using image-text pairs. This allows the final Q-former to extract the most relevant visual representations to the text, resulting in high-quality image-text aligned vector representations. The second stage is visual-language generative learning, where the Q-former is connected to a large language model. A fully connected layer projects the Q-former's output query embedding onto the same dimension as the embeddings of multiple text description layers in the large language model, thus generating text. During pre-training, the Q-former facilitates interaction and alignment between vision and language through specific pre-training tasks, such as image-text contrast learning, image-based text generation, and image-text matching, enabling image-text retrieval, image captioning generation, and visual question answering.

[0061] Among them, such as Figure 4As shown, a large language model includes multiple text description layers, which can include three layers: caption-level, keyword-level, and prompt-level. The caption-level corresponds to a sentence-level description of the image's basic semantics, the keyword-level corresponds to a word-level description of the image's general semantics, and the prompt-level corresponds to a word-level description of the image's deep semantics. For an example, please refer to [link to example]. Figure 5 , Figure 5 This is a flowchart illustrating another image processing method provided in an embodiment of this application, as shown below. Figure 5 As shown, an image of a large conference room can be input into the comprehensive feature extraction model. After passing through the text feature extraction model Blip2, assuming a three-layer text description is obtained, for example, at the title level, the output dialog box content "The conference room has many long tables and black chairs" is the text information corresponding to the sentence-level description of the image's basic semantics; at the keyword level, core keywords are obtained based on the question and answer, specifically, "Q: Which keywords describe this picture? A: Conference room, large space, multiple rows," the answer to the question is the text information corresponding to the word-level description of the image's general semantics; at the prompt level, the functional description of the image is obtained based on the question and answer, specifically, "Q: What is the room in the picture usually used for? A: Presentations, training," the answer to this question is the text information corresponding to the word-level description of the image's deep semantics.

[0062] As can be seen, in this embodiment, by extracting basic or deep text information from the image, the extracted text information is better matched with the classification task, thereby improving the accuracy of the classification task.

[0063] S230, determine the text encoding features of each image based on the text information of each image.

[0064] Among them, such as Figure 5 As shown, the text information of the multiple layers is encoded by the contrastive language-image pre-trained model (CLIP) to obtain their respective text representations, namely F1, F2 and F3.

[0065] If only one text information category is determined, then one dimension of text information is output, and one text encoding feature is extracted accordingly. If two or more text information categories are determined, then multiple dimensions of text information are output, and multiple text encoding features are extracted accordingly. In this case, the multiple text encoding features need to be fused to obtain the final text encoding feature. For example, as shown... Figure 5As shown, assuming there are 3 categories of text information, we get 3 text encoding features, namely F1, F2 and F3. We concatenate each text encoding feature to get the final text encoding feature F. The specific formula is: F = concat(F1, F2, F3).

[0066] S240, determine the classification label of each image based on the text encoding features and the image encoding features of each image.

[0067] Specifically, determining the classification label of each image based on the text encoding features and the image encoding features of each image includes: fusing the text encoding features and the image encoding features of each image to obtain the fused features of each image; and performing classification processing on the fused features of each image to obtain the classification label of each image.

[0068] The image encoding features are feature vectors related to the image content, output by the image encoder in the image-to-text model. The image content includes at least one of the image's color, texture, and shape. The image encoder in the image-to-text model can be the image encoder part of the CLIP model. By inputting the image to be classified, the image encoder part of the CLIP model converts the input image into a vector representation, that is, a feature vector that captures the important information in the image, and normalizes the obtained image feature vector to obtain the image encoding features.

[0069] In this process, after obtaining the text-encoded features output by the text encoder and the image-encoded features output by the image encoder, the text-encoded features and image-encoded features are fused to obtain a comprehensive feature corresponding to the image. For example, such as... Figure 5 As shown, the text encoding feature F and the image encoding feature L are concatenated to obtain the comprehensive feature V, and the specific formula is: V = concat(L, F).

[0070] The comprehensive feature V is classified using a classification algorithm to obtain the classification result. This algorithm includes, but is not limited to, K-Means, density-based classification, and hierarchical classification algorithms. The classification result includes a classification label identifying the category to which each image belongs, a classification center representing typical features or representative images of each category, and a distance or similarity metric from each image to its respective classification center.

[0071] Furthermore, other feature fusion methods can be used, such as weighted fusion, cross-modal attention, and conditional batch normalization. Specifically, different weights can be assigned to text-encoded features and image-encoded features, and then weighted summation can be performed; or an attention mechanism can be used to match image features and text features to achieve finer-grained feature fusion; or textual cues in natural language descriptions can be used to adjust conditional batch normalization to enhance the visual semantic embedding of the generated network feature maps.

[0072] The encoded features are not limited to images and text; they can be extended to include more modal information, such as audio and video. The modal features can be expanded according to the actual usage.

[0073] As can be seen, in this example, adding multi-layered textual description features to image classification increases the richness of the textual description, captures image features more accurately, and improves the accuracy of image classification results.

[0074] As can be seen, in this example, this application dynamically selects the text information category based on the image differences and / or the number of classification labels, and extracts text information of different dimensions based on the determined text information category. Compared with the prior art, which only extracts text information of the same type, this application considers the influence of the relationship between the number of image differences and classification labels on the classification labels in the classification task, making the extracted text information more compatible with the classification task, thereby improving the accuracy and versatility of image classification tasks.

[0075] In one possible implementation, after obtaining the classification result for each image, all generated text information can be summarized, or all generated text information can be input into a large model. Based on understanding the input text, a concise and accurate summary description reflecting its textual description can be generated. Then, text similarity is calculated for the classification result and the summary description respectively, thereby obtaining the accuracy of the classification result and text information. Alternatively, a powerful model for sentence vector generation can be used to calculate and accurately evaluate the classification result. Furthermore, the comprehensive feature extraction model can be adjusted based on the evaluation result. The large model can be a Generative Pre-trained Transformer 3 (GPT-3), or other models of the same type.

[0076] As can be seen, in this example, accurate classification performance evaluation can be achieved through text similarity measurement, and the accuracy of classification tasks in multiple scenarios can be improved.

[0077] For examples consistent with the above embodiments, please refer to... Figure 6 , Figure 6This is a functional unit block diagram of an image processing device provided in an embodiment of this application, such as... Figure 6 As shown, the image processing device 60 includes: a first determining unit 61, configured to determine the text information category of each image based on the image differences and / or the number of classification labels of multiple images; an extraction unit 62, configured to extract the text information corresponding to the text information category of each image; a second determining unit 63, configured to determine the text encoding features of each image based on the text information of each image; and a third determining unit 64, configured to determine the classification label of each image based on the text encoding features and the image encoding features of each image.

[0078] In one possible embodiment, regarding the text information category, the first determining unit 61 is specifically configured to: the text information category includes at least one of the following: sentence-level description of image basic semantics, word-level description of image general semantics, and word-level description of image deep semantics; the image basic semantics is used to characterize a direct description of a scene constructed by some or all elements in the image, the image general semantics is used to characterize the physical characteristics and / or basic usage characteristics of the scene, and the image deep semantics is used to characterize the derived usage characteristics of the scene; wherein, each element has different descriptive attributes.

[0079] In one possible embodiment, in determining the text information category, the first determining unit 61 is further configured to: determine the text information category based on the image differences of multiple images and the number of classification labels, including: determining the average image difference of the multiple images; querying a preset set of text information categories using a parameter group as a query identifier, and obtaining the text information category matching the parameter group, wherein the parameter group includes the average image difference and the number of classification labels, and the set of text information categories includes the correspondence between the parameter group and the text information category.

[0080] In one possible embodiment, in determining the text information category, the first determining unit 61 is further configured to: determine the image difference between any two images among the plurality of images to obtain a plurality of image differences; if the image difference with the largest value among the plurality of image differences is detected to be less than a preset image difference, then the text information category is determined to include the sentence-level description of the image's basic semantics, the word-level description of the image's general semantics, and the word-level description of the image's deep semantics; if the image difference with the smallest value among the plurality of image differences is detected to be greater than the preset image difference, then the text information category is determined to include the sentence-level description of the image's basic semantics.

[0081] In one possible embodiment, in determining the text information category, the first determining unit 61 is further configured to: if the number of the classification tags is detected to be greater than a preset number, determine that the text information category includes the sentence-level description of the image basic semantics, the word-level description of the image general semantics, and the word-level description of the image deep semantics; if the number of the classification tags is detected to be less than or equal to the preset number, determine that the text information category includes the sentence-level description of the image basic semantics and the word-level description of the image general semantics.

[0082] In one possible embodiment, in extracting text information corresponding to the text information category of each image, the extraction unit 62 is specifically configured to: for each image's text information category, if the text information category includes a sentence-level description of the image's basic semantics, then identify elements in the image; determine the scene of the image based on the identified elements; create a direct description statement of the scene based on the scene vocabulary and the identified elements, and use the direct description statement of the scene as text information of the sentence-level description of the image's basic semantics; if the text information category includes a sentence-level description of the image's basic semantics and a word-level description of the image's general semantics, then identify elements in the image; determine the scene of the image based on the identified elements; create a direct description statement of the scene based on the scene vocabulary and the identified elements, and use the direct description statement of the scene as text information of the sentence-level description of the image's basic semantics; and, based on the physical characteristics of the scene and / or The process involves: creating basic usage descriptors for the scene based on its basic usage characteristics; using these basic usage descriptors as word-level descriptions of the image's general semantics; identifying elements in the image if the text information category includes sentence-level descriptions of the image's basic semantics, word-level descriptions of the image's general semantics, and word-level descriptions of the image's deep semantics; determining the scene of the image based on these elements; creating direct descriptions of the scene based on the scene vocabulary and the identified elements; using these direct descriptions as sentence-level descriptions of the image's basic semantics; creating basic usage descriptors for the scene based on its physical characteristics and / or basic usage characteristics; using these basic usage descriptors as word-level descriptions of the image's general semantics; and creating derived usage descriptors for the scene based on the determined derived usage characteristics; using these derived usage descriptors as word-level descriptions of the image's deep semantics.

[0083] In one possible embodiment, in determining the classification label of each image based on the text encoding features and the image encoding features of each image, the third determining unit 64 is specifically configured to: fuse the text encoding features and the image encoding features of each image to obtain the fused features of each image; and perform classification processing on the fused features of each image to obtain the classification label of each image.

[0084] It is understood that since the method embodiments and the device embodiments are different presentations of the same technical concept, the content of the method embodiment section in this application should be adapted to the device embodiment section in a synchronous manner, and will not be repeated here.

[0085] In the case of using integrated units, please refer to Figure 7 , Figure 7 This is a functional unit block diagram of another image processing device provided in the embodiments of this application, such as... Figure 7 As shown, the image processing apparatus 60 includes a processing module 602 and a communication module 601. The processing module 602 controls and manages the operation of the image processing apparatus 60, for example, executing the steps of the first determining unit 61, the extraction unit 62, the second determining unit 63, and the third determining unit 64, and / or performing other processes of the technology described herein. The communication module 601 is used for interaction between the image processing apparatus 60 and other devices. Figure 7 As shown, the image processing apparatus 60 may further include a storage module 603, which is used to store the program code and data of the image processing apparatus 60.

[0086] The processing module 602 can be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The communication module 601 can be a transceiver, RF circuitry, or a communication interface, etc. The storage module 603 can be a memory.

[0087] All relevant content in each scenario involved in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here. The above image processing device 60 can perform the above... Figure 2 The image processing method shown.

[0088] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device proposed in an embodiment of this application, as shown below. Figure 8 As shown, the electronic device 80 includes a processor 810, a memory 820, a communication interface 830, and one or more programs 821. The one or more programs 821 are stored in the memory and configured to be executed by the processor. When the program is executed, it includes some or all of the steps of any of the image processing methods described in the above method embodiments. The processor, memory, and communication interface are interconnected and complete communication between them.

[0089] The memory can be volatile memory such as Dynamic Random Access Memory (DRAM) or non-volatile memory such as a hard disk drive. The memory stores a set of executable program code, and the processor calls the executable program code stored in the memory to execute some or all of the steps of any energy data management method described in the above image processing method embodiments.

[0090] As can be seen, the electronic device 80 described in this application first determines the text information category of each image based on the image differences and / or the number of classification labels of multiple images; then, it extracts the text information corresponding to the text information category of each image; it determines the text encoding features of each image based on the text information of each image; and finally, it determines the classification label of each image based on the text encoding features and the image encoding features of each image. This application dynamically selects the text information category based on the image differences and / or the number of classification labels, and extracts text information of different dimensions based on the determined text information category. Compared to the prior art, which only extracts text information of the same type, this application considers the influence of the relationship between the number of image differences and classification labels on the classification labels, making the extracted text information more closely matched to the classification task, thereby improving the accuracy and versatility of image classification tasks.

[0091] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the methods described in the above method embodiments, wherein the computer includes an electronic device.

[0092] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments. The computer program product may be a software installation package, and the computer may include an electronic device.

[0093] It should be noted that, for the sake of simplicity, the aforementioned methods are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are optional, and the actions and modules involved are not necessarily essential to this application.

[0094] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0095] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0096] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0097] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software program module.

[0098] If the integrated unit is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0099] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: a flash drive, a read-only memory, a random access memory, a magnetic disk, or an optical disk, etc.

[0100] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The above description of the embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized in that, include: The text information category for each image is determined based on the image differences and / or the number of classification labels among multiple images; Extract the text information corresponding to the text information category of each image; The text encoding features of each image are determined based on the text information of each image; The classification label of each image is determined based on the text encoding features and the image encoding features of each image.

2. The method according to claim 1, characterized in that, The text information categories include at least one of the following: sentence-level description of basic image semantics, word-level description of generalized image semantics, and word-level description of deep image semantics; The image basic semantics are used to characterize the direct description of the scene constructed by some or all of the elements in the image; the image general semantics are used to characterize the physical characteristics and / or basic use characteristics of the scene; and the image deep semantics are used to characterize the derived use characteristics of the scene; wherein, each element has different descriptive attributes.

3. The method according to claim 2, characterized in that, The process of determining the text information category based on image differences and the number of classification labels from multiple images includes: Determine the mean image difference of the plurality of images; Using a parameter group as the query identifier, a preset set of text information categories is queried to obtain the text information categories that match the parameter group. The parameter group includes the mean of the image difference and the number of the classification labels. The set of text information categories includes the correspondence between the parameter group and the text information categories.

4. The method according to claim 2, characterized in that, The method of determining the text information category based on image differences among multiple images includes: Determine the image difference between any two images among the plurality of images to obtain multiple image differences; If the image difference with the largest value among the multiple image differences is less than the preset image difference, then the text information category is determined to include the sentence-level description of the basic semantics of the image, the word-level description of the general semantics of the image, and the word-level description of the deep semantics of the image. If the smallest image difference among the plurality of image differences is detected to be greater than the preset image difference, then the text information category is determined to include the sentence-level description of the image's basic semantics.

5. The method according to claim 2, characterized in that, The process of determining the text information category based on the number of category tags includes: If the number of the classification labels is greater than the preset number, then the text information category is determined to include the sentence-level description of the basic semantics of the image, the word-level description of the general semantics of the image, and the word-level description of the deep semantics of the image; If the number of the classification labels is less than or equal to a preset number, then the text information category is determined to include sentence-level descriptions of the image's basic semantics and word-level descriptions of the image's general semantics.

6. The method according to any one of claims 2-5, characterized in that, The extraction of text information corresponding to the text information category of each image includes: For each image's text information category, if the text information category includes a sentence-level description of the image's basic semantics, then elements in the image are identified; the scene of the image is determined based on the identified elements; a direct description statement of the scene is created based on the scene vocabulary and the identified elements, and the direct description statement of the scene is used as the text information of the sentence-level description of the image's basic semantics; If the text information category includes a sentence-level description of the image's basic semantics and a word-level description of the image's general semantics, then elements in the image are identified; the scene of the image is determined based on the identified elements; a direct description statement of the scene is created based on the scene vocabulary and the identified elements, and the direct description statement of the scene is used as the text information of the sentence-level description of the image's basic semantics; and, basic usage descriptive words of the scene are created based on the scene's physical characteristics and / or basic usage characteristics, and the basic usage descriptive words of the scene are used as the text information of the word-level description of the image's general semantics. If the text information category includes a sentence-level description of the image's basic semantics, a word-level description of the image's general semantics, and a word-level description of the image's deep semantics, then elements in the image are identified; the scene of the image is determined based on the elements; a direct description statement of the scene is created based on the scene vocabulary and the identified elements, and the direct description statement of the scene is used as the text information of the sentence-level description of the image's basic semantics; and, basic usage descriptive words of the scene are created based on the scene's physical characteristics and / or basic usage characteristics, and the basic usage descriptive words of the scene are used as the text information of the word-level description of the image's general semantics; and, derived usage descriptive words of the scene are created based on the determined derived usage characteristics of the scene, and the derived usage descriptive words of the scene are used as the text information of the word-level description of the image's deep semantics.

7. The method according to any one of claims 2-5, characterized in that, The step of determining the classification label for each image based on the text encoding features and the image encoding features of each image includes: The text encoding features and image encoding features of each image are fused together to obtain the fused features of each image; The fusion features of each image are classified to obtain a classification label for each image.

8. An image processing apparatus, characterized in that, include: The first determining unit is used to determine the text information category of each image based on the image differences and / or the number of classification labels of multiple images; An extraction unit is used to extract text information corresponding to the text information category of each image; The second determining unit is used to determine the text encoding features of each image based on the text information of each image; The third determining unit is used to determine the classification label of each image based on the text encoding features and the image encoding features of each image.

9. An electronic device, characterized in that, The method includes a processor and a memory storing execution instructions, the memory storing one or more programs; when the processor executes the execution instructions stored in the memory, the processor performs the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The device stores an energy data management program, including execution instructions, which, when executed by the processor of the electronic device, perform the method according to any one of claims 1-7.

Images