Image semantic information generation method and electronic device

Abstract

The application discloses an image semantic information generation method and an electronic device, and relates to the technical field of artificial intelligence. The method comprises the following steps: generating multi-dimensional image semantic features according to a target subject and context information of an image to be processed; inputting the multi-dimensional image semantic features into a semantic label graph and a nonlinear mapping network respectively to obtain semantic representation information and a semantic space alignment prediction result with the same dimension as a label number; and fusing the semantic space alignment prediction result and the semantic representation information to generate image semantic information of the image to be processed. The semantic label graph is a heterogeneous graph neural network with semantic labels as graph nodes, label semantic description features as graph node features, and node connection edges determined based on semantic similarity and co-occurrence relationships between the semantic labels. The application can solve the problem that related technologies cannot meet the demand of users for the quality of semantic information generation, and can generate high-precision image semantic information.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for generating image semantic information and an electronic device. Background Technology

[0002] Image semantic information generation is a task in which computers automatically extract and output semantic descriptive information from input image data, describing the overall content, scene, main objects, object attributes, and their interrelationships of the image. However, related technologies, when processing multi-semantic images, suffer from limitations in their ability to fully understand multiple semantic information points, leading to biases and limitations in the generated semantic information and failing to meet users' requirements for the quality of semantic information generation.

[0003] Therefore, generating high-quality image semantic information is a technical problem that needs to be solved by those skilled in the art.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] This application provides an image semantic information generation method and electronic device, which can generate high-precision image semantic information.

[0006] To solve the above-mentioned technical problems, this application provides the following technical solution:

[0007] This application provides a method for generating image semantic information, comprising: generating multi-dimensional image semantic features based on the target subject and context information of the image to be processed; inputting the multi-dimensional image semantic features into a semantic label graph to obtain semantic representation information representing the degree of similarity or association between the image to be processed and each semantic label; wherein the semantic label graph is a heterogeneous graph neural network that uses semantic labels as graph nodes, uses label semantic description features as graph node features, and determines node connection edges based on semantic similarity and co-occurrence relationships between semantic labels; inputting the multi-dimensional image semantic features into a nonlinear mapping network to obtain a semantic space alignment prediction result with the same dimension as the number of labels; and fusing the semantic space alignment prediction result with the semantic representation information to generate image semantic information corresponding to the image to be processed.

[0008] This application also provides an electronic device, including a memory and a processor, wherein the processor is used to implement the steps of the above-described image semantic information generation method when executing a computer program stored in the memory.

[0009] The advantages of the technical solution provided in this application are as follows: First, by extracting the target subject and its contextual information to construct multi-dimensional image semantic features, it can comprehensively cover the local object and global scene semantics of the image, enhancing the richness of information and the integrity of the structure of the image representation, and effectively enhancing the semantic expression capability of the image. Second, by inputting the multi-dimensional features into a semantic label heterogeneous graph neural network constructed based on semantic similarity and co-occurrence relationship, the structural dependency relationship between labels can be explicitly modeled, improving the recognition capability of semantic association relationship. Third, by performing direct prediction based on nonlinear mapping network and association reasoning based on graph structure in parallel on the same image feature, the complementarity and synergy of multi-view semantic understanding are realized. Finally, by fusing the above two prediction results, the discriminative ability of end-to-end classification and the structured inference advantage of label relationship modeling are effectively combined, thereby significantly improving the accuracy, robustness and generalization ability of semantic understanding of images in multi-label scenarios, effectively improving the recognition accuracy of multiple semantic concepts in images, and thus helping to improve the execution accuracy and efficiency of image processing tasks that use image semantic information, such as image intelligent understanding tasks and visual semantic analysis tasks. Furthermore, this application also provides corresponding implementation devices, electronic devices, computer-readable storage media, and computer program products for the image semantic information generation method, further making the method more practical, and the devices, electronic devices, computer-readable storage media, and computer program products also have corresponding advantages.

[0010] The technical features mentioned above, those to be mentioned below, and those shown individually in the accompanying drawings can be arbitrarily combined, as long as the combined technical features are not contradictory. All feasible combinations of features are the technical content explicitly described in this application. Any one of the multiple sub-features contained in the same statement can be applied independently, without necessarily being applied together with other sub-features.

[0011] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating an image semantic information generation method provided in this application.

[0014] Figure 2A flowchart illustrating another method for generating image semantic information provided in this application.

[0015] Figure 3 This is a structural framework diagram of an exemplary embodiment of the image semantic information generation apparatus provided in this application.

[0016] Figure 4 This is a structural diagram of an exemplary embodiment of the electronic device provided in this application. Detailed Implementation

[0017] To enable those skilled in the art to better understand the technical solutions of this application, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. The terms "first," "second," "third," "fourth," etc., used in the specification and the aforementioned drawings are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. The term "exemplary" means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior to or better than other embodiments.

[0018] As computer vision technology is applied to more and more industries, image understanding tasks have evolved from simple object recognition to the generation of semantic information for in-depth understanding and description of image content. Image semantic information generation enables machines to automatically output semantic information covering the main objects in an image, the global scene, object attributes, and their interrelationships, providing the necessary image analysis data for various applications such as image recognition, image search, image tag recommendation, intelligent interaction, and image content analysis.

[0019] In practical applications, an image often contains multiple semantic concepts simultaneously. When understanding multi-semantic images, related technologies extract image features using convolutional neural networks and employ independent classifiers to identify each semantic label, resulting in multiple independent semantic information outputs. Because these technologies ignore the potential semantic relationships and co-occurrence structures between different semantic labels, they struggle to capture the dependencies between complex semantics. Furthermore, images contain a wealth of contextual information, such as the spatial relationships between local objects and the global scene / target. Related technologies fail to effectively utilize this information, leading to limitations and biases in image semantic understanding. To address the problems of using convolutional neural networks in image semantic understanding, related technologies introduce graph neural networks to uncover the relationships between different semantics and combine them with attention mechanisms to enhance local features. However, this method determines relationships by analyzing the similarity between different semantic information, without considering co-occurrence structures. Moreover, insufficient extraction of image contextual features, especially the inability to deeply understand the spatial dependencies between local objects and the subject, fails to take into account the advantages of multiple semantic label classification methods, resulting in inaccurate multi-view fusion and an inability to ensure semantic and structural consistency in the label graph learning process.

[0020] To address the challenges of existing technologies in image semantic understanding, such as the inability to closely integrate visual feature extraction with semantics, resulting in limited depth of generated semantic information due to insufficient utilization of multi-level image contextual structures, and the inability to ensure the internal structural consistency of generated semantic information, this application utilizes a heterogeneous graph model to construct complex structural dependencies between multiple labels; integrates multi-source contextual features to enhance image semantic expression capabilities; and integrates multi-view prediction structures to effectively improve the recognition accuracy and stability of different semantic categories. This approach fully explores the semantic structure of images, understands the deep dependencies between different semantics, and effectively integrates multiple features, thereby significantly improving the accuracy and robustness of multi-label semantic understanding.

[0021] The specific application environment architecture or hardware architecture upon which the image semantic information generation method depends is described herein. Examples of possible application scenarios related to the technical solutions of this application are provided below, including the following:

[0022] The intelligent image management platform needs to automatically annotate a large number of images uploaded by users, generating accurate and comprehensive semantic tags to facilitate users' quick image retrieval and management. The platform covers various image types, such as landscape photos, portraits, everyday scenes, and work scenes. Each image may contain multiple semantic concepts (such as seaside, sunset, people, beach, etc.), requiring the platform to accurately identify and annotate this semantic information.

[0023] The image semantic information generation model is deployed to the backend server of the intelligent image management platform, configuring the necessary hardware resources (such as graphics processors and memory) and software environment (such as deep learning frameworks and dependency libraries) for model operation. Users upload images to be labeled through the platform frontend, and the image data is transmitted to the backend server, which then transmits the images to the image semantic information generation model. The image semantic feature extraction layer of the image semantic information generation model processes the input image, identifies the target subject (such as a person) and local objects (such as a beach, waves, and the sun) in the image, and extracts the visual features of the target subject, the semantic category, visual features, and spatial distance of the local objects from the subject to form visual feature information; at the same time, it extracts global scene semantic information (such as a seaside scene) and depth features (such as a person being in front of the beach and the sun being in the sky) to form global feature information; the two types of features are then concatenated to obtain multi-dimensional image semantic features. Semantic label graph processing: Multi-dimensional image semantic features are input into a semantic label graph, which uses platform-preset semantic labels (such as seaside, sunset, people, beach, waves, etc.) as nodes. The initial features of the nodes are generated by encoding with a large language model, and the edges are determined by the semantic similarity between labels (e.g., high semantic similarity between seaside and beach) and co-occurrence relationship (e.g., high co-occurrence frequency between sunset and seaside). Through multiple rounds of feature propagation and transformation, label embedding representation vectors are generated, thereby obtaining the association information between the image and each label (e.g., association degree of 0.95 for people, 0.92 for seaside, and 0.88 for sunset). Nonlinear mapping prediction refers to inputting multi-dimensional image semantic features into a nonlinear mapping network. The network uses a multilayer perceptron to map the features to prediction results with the same number of dimensions as the number of labels (e.g., probability of people 0.93, probability of seaside 0.90, probability of sunset 0.86, etc.). The output layer weightedly fuses the association information of the semantic label graph with the prediction results of the nonlinear mapping network to generate the final image semantic labels (such as seaside, sunset, people, beach), and returns the labels to the platform front end for display to users; simultaneously, the labels are associated with the images and stored for subsequent retrieval. Furthermore, the intelligent image management platform regularly collects user feedback on the automatic annotation results (such as correcting erroneous labels and supplementing missing labels), using this feedback data as samples with real labels for iterative training of the image semantic information generation model, updating model parameters, and improving the accuracy of the image labeling task.

[0024] In summary, this invention systematically integrates local and global feature information of images, models various dependencies between labels, and fuses different classification perspectives, effectively improving the performance of multi-label image semantic understanding tasks. It can quickly and accurately generate comprehensive semantic labels for user-uploaded images, and the automatic labeling function replaces manual labeling, reducing platform operating costs, improving image management efficiency, and enabling users to quickly retrieve target images through semantic labels, thus enhancing the platform's user experience.

[0025] It should be noted that the above application scenarios are only shown to facilitate understanding of the ideas and principles of this application, and the implementation methods of this application are not limited in any way. On the contrary, the implementation methods of this application can be applied to any applicable scenario. After introducing the technical solution of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. First, please refer to... Figure 1 According to the image semantic information generation method provided in this application, it can be implemented as a computer program product. It can be installed and run on a cloud server cluster corresponding to an image and video content understanding platform, or on edge computing devices such as smart cameras, network video recorders, and video analysis gateways. In this application scenario, the edge computing devices are used for real-time video stream analysis, and the cloud server cluster is used to perform batch annotation, content retrieval, and classification archiving of large-scale image / video libraries. It can also be installed and run on mobile smart terminals or embedded interactive devices corresponding to an intelligent question-and-answer interaction system for understanding images. Mobile smart terminals include smartphones, tablets, and AR (Augmented Reality) / VR (Virtual Reality) devices. Embedded interactive devices include smart speakers, service robots, and intelligent car cockpit systems. It can also be installed and run on professional visual analysis systems, such as medical image analysis workstations, industrial visual inspection terminal equipment, and remote sensing image interpretation workstations. It can also be installed and run on graphics stations running professional image processing software or cloud design platforms capable of intelligent image retouching and material management, and AI (artificial intelligence) assisted image and video design services, to implement the understanding of image content and generate corresponding semantic information. In some embodiments of this method, the method includes the following steps:

[0026] S101: Generate multi-dimensional image semantic features based on the target subject and its context information of the image to be processed.

[0027] The image to be processed refers to the raw image data from which semantic information needs to be extracted. It may contain various visual elements such as the main subject, local objects, and the global scene. The target subject is the most prominent object in the image or the object of most interest to the user; it is the object of semantic understanding, such as the main person in a photo or the core object in a scene. Contextual information is the relevant visual information surrounding the target subject, including local objects (objects other than the subject), the global scene (such as background, environment, lighting, etc.), and spatial relationships (such as distance and positional distribution between objects). Multi-dimensional image semantic features are comprehensive feature vectors that integrate the visual features of the target subject, the relevant features of local objects, and the semantic and depth features of the global scene. They can comprehensively represent the semantic information of the image, representing the content of the image from different perspectives (such as appearance, category, spatial layout, and scene atmosphere). This step can be implemented using any method or neural network model capable of extracting image features; this will not affect the implementation of this step.

[0028] This step is used to construct the semantic features of the image to be processed by modeling the target subject and its context in the image. For example, an object detection network can be used to extract the semantic category, visual features and spatial distance of local objects from the subject; a depth estimation network and a global feature extraction network can be used to obtain the semantic information and depth features of the global scene; and the above information can be fused to construct a multi-faceted semantic representation vector of the image.

[0029] S102: Input the multi-dimensional image semantic features into the semantic label map to obtain semantic representation information that represents the degree of similarity or association between the image to be processed and each semantic label.

[0030] In multi-label tasks, complex semantic relationships often exist between labels, such as synonymy, near-synonymy, opposition, and co-occurrence. To model these dependencies, a heterogeneous graph structure, or semantic label graph, is constructed with labels as nodes. The semantic label graph is a graph structure built based on heterogeneous graph neural networks. It uses possible semantic labels as nodes (graph nodes), and the semantic description features of the labels are the node features. That is, the initial features of the nodes originate from the semantic encoding of the label text description (e.g., the natural scene of a blue sky), giving them linguistic meaning. The nodes' connecting edges are determined by the semantic similarity and co-occurrence relationships between labels, used to model the complex dependencies between them. The connecting edges reflect the relationships between labels, and their construction is based on two types of information: first, the similarity of the labels in linguistic meaning (i.e., semantic similarity); and second, the frequency with which they co-occur in a large number of images (i.e., co-occurrence relationships). Semantic labeling graphs can process and fuse such multi-dimensional relationships. By inputting image features into the semantic labeling graph, the matching degree or association strength between the image content of the image to be processed in S101 and each label node in the graph structure of the semantic labeling graph can be calculated. The output is a set of "semantic representation information", such as the association probability score of each label. That is, the semantic representation information is the output data after processing by the semantic labeling graph, which is the label embedding vector, reflecting the information of the similarity or association between the image to be processed and each semantic label.

[0031] In this step, multiple semantic labels can be predefined. The semantic label graph construction process is as follows: each semantic label is treated as a graph node, and the node features are initialized with the semantic representation of the label definition generated by a large language model. Heterogeneous graph edges are constructed based on the semantic similarity and co-occurrence relationship between labels. The heterogeneous graph neural network is used to perform representation learning on the semantic label graph to generate label embedding vectors with structural dependencies. Since the node connection edges of the semantic label graph are jointly determined by semantic similarity and co-occurrence relationship, the loss function needs to consider both semantic similarity and co-occurrence probability during the training process of the semantic label graph, and the label embedding graph structure is trained under supervision.

[0032] S103: Input multi-dimensional image semantic features into a nonlinear mapping network to obtain semantic space alignment prediction results with the same number of label dimensions.

[0033] The nonlinear mapping network is a feedforward neural network (such as a multilayer perceptron, MLP) that directly maps the image features of the S101 image to be processed into a probability space that corresponds one-to-one with all semantic labels through learned complex nonlinear transformations. The semantic space aligned prediction result refers to the output of the nonlinear mapping network, whose dimension is the same as the total number of labels, directly reflecting the predicted probability of each label.

[0034] S104: Fuse the semantic space alignment prediction results with semantic representation information to generate image semantic information corresponding to the image to be processed.

[0035] In this process, image semantic information is the final generated semantic description information that describes the overall content, scene, main object, object attributes, and their interrelationships of the image. Its dimension is consistent with the total number of labels. The fusion operation can be a simple weighted average or a more complex weighted calculation after concatenation, neither of which affects the implementation of this step. The purpose of this fusion step is to combine the advantages of the two paths and make full use of the semantic relationship between image information and labels: Path 1 (i.e., S102) projects the image semantic representation onto the label embedding space and uses the structured knowledge between labels for reasoning. The output of this path is to calculate the similarity score with each label representation (i.e., calculate the label probability based on the similarity between the image semantic vector corresponding to the semantic representation information and the label embedding), thereby judging the degree of matching between the image and the label. Path 2 (i.e., S103) directly maps the multi-dimensional image semantic features of the image to be processed in S101 to the same output dimension as the number of labels, realizing end-to-end prediction (i.e., outputting the label probability through a nonlinear mapping network based on the image semantic vector corresponding to the semantic space alignment prediction result), and using an end-to-end data-driven mode for recognition. The fused image semantic information is a set of structured semantic labels with confidence, which is more robust and accurate than single-path prediction.

[0036] In the technical solution provided in this application embodiment, multi-dimensional image semantic features are constructed by extracting the target subject and its context information. This comprehensively covers the local object and global scene semantics of the image, enhancing the richness and structural integrity of the image representation information and effectively enhancing the image semantic expression capability. Secondly, the multi-dimensional features are input into a semantic label heterogeneous graph neural network constructed based on semantic similarity and co-occurrence relationship, enabling explicit modeling of the structural dependency relationship between labels and improving the recognition capability of semantic association relationship. Furthermore, by performing direct prediction based on nonlinear mapping network and association reasoning based on graph structure in parallel on the same image feature, the complementarity and synergy of multi-view semantic understanding are realized. Finally, by fusing the above two prediction results, the discriminative ability of end-to-end classification and the structured inference advantage of label relationship modeling are effectively combined, thereby significantly improving the accuracy, robustness and generalization ability of semantic understanding of images in multi-label scenarios, effectively improving the recognition accuracy of multiple semantic concepts in images, and helping to improve the execution accuracy and efficiency of tasks using image semantic information such as image intelligent understanding and visual semantic analysis.

[0037] Considering that insufficient extraction of image context features, such as focusing only on features of local objects or a single dimension of the global scene without effectively fusing local and global features, can lead to incomplete image semantic representation and affect the accuracy of subsequent semantic information generation, based on the above embodiments, this embodiment also provides an exemplary multi-dimensional image semantic feature generation process, which may include the following:

[0038] The visual features of the target subject in the image to be processed, the semantic category, visual features, and spatial distance information between each local object and the target subject are obtained as visual feature information; the semantic information and depth features of the global scene of the image to be processed are obtained as global feature information; the fusion features of visual feature information and global feature information are used as multi-dimensional image semantic features.

[0039] Visual feature information comprises the visual features of the target subject, the semantic categories of each local object within the target, visual features, and spatial distances between the target subject and the target. It characterizes the visual attributes and positional relationships of local objects in an image. Global feature information consists of semantic and depth features of the global scene. Semantic information reflects the overall category of the scene (e.g., park, office), while depth features reflect the relative spatial structure of each pixel within the scene. Fusion features are feature vectors obtained by combining visual and global feature information through methods such as concatenation and weighted summation. These features comprehensively cover both local and global semantic information of the image.

[0040] In this step, for example, an object detection network (such as Faster-RCNN (Fast Region Convolutional Neural Network)) can be used to identify the target subject and local objects in the image to be processed, determine the region of the target subject, and extract its visual features (encoded through a convolutional neural network such as ResNet (Deep Residual Network)). The semantic category of each local object is obtained based on the output of the object detection network. The visual features of each local object are obtained using methods such as convolutional neural networks. The spatial distance between each local object and the target subject is calculated using bounding box coordinates based on the object detection output. The visual features of the target subject, the semantic category of the local objects, the visual features, and the spatial distance information are integrated to form visual feature information. A network with semantic encoding capabilities (such as ResNet) is used to encode the entire image to obtain the semantic information of the global scene. A monocular depth estimation model is used to estimate the depth of the image, obtaining depth features reflecting the spatial structure of the scene. The global scene semantic information and depth features are integrated to form global feature information. Vector concatenation is used to concatenate the feature vectors corresponding to the visual feature information with the feature vectors corresponding to the global feature information to obtain multi-dimensional image semantic features.

[0041] As can be seen from the above, this embodiment extracts visual feature information and global feature information respectively, comprehensively capturing the local object attributes, positional relationships, and global scene semantics and spatial structure of the image; after fusing the two types of features, the generated multi-dimensional image semantic features have richer semantic information, providing comprehensive feature support for subsequent semantic prediction and improving the comprehensiveness and accuracy of semantic information generation.

[0042] For example, simple existence detection cannot distinguish the different importance of nearby key objects and distant background objects to the described subject. Based on the above embodiments, this embodiment introduces spatial distance weighting to enable the generated local features to more accurately reflect the semantic context, which may include the following:

[0043] In the image to be processed after size normalization and pixel value standardization, the target subject is identified and the corresponding target patch is extracted; the target patch is encoded to obtain the visual features of the target subject; based on the spatial distance information between each target local object and the target subject, the weight value of each target local object is determined in a way that the weight decreases as the distance increases; based on the semantic category, visual features, position information and weight value of each target local object, the local object features are obtained.

[0044] In this embodiment, size normalization refers to adjusting the image to be processed to a preset uniform size (e.g., 224×224 pixels) to ensure the consistency of image input and facilitate subsequent model processing. Pixel value normalization is the normalization of image pixel values ​​(e.g., mapping pixel values ​​to the [0, 1] interval or standardizing them to a distribution with a mean of 0 and a variance of 1), eliminating the impact of pixel value range differences on model training and inference. Target patches are image patches of the target subject area extracted from the preprocessed image. They are data for extracting the visual features of the target subject. For example, a convolutional neural network can be used to encode the target patches to extract deep visual features of the target subject. Weight values ​​are weight coefficients determined based on the spatial distance between the target local object and the target subject, used to quantify the contribution of different local objects to the semantics of the subject. For example, the spatial distance between each target local object and the target subject can be calculated based on the Euclidean distance of the bounding box center coordinates. The weight value of each target local object is determined according to the rule that the weight decreases as the distance increases (e.g., using the reciprocal of the distance as the weight base and then normalizing).

[0045] As can be seen from the above, this embodiment ensures the consistency of input data through size normalization and pixel value standardization; determining the weight of local objects based on spatial distance can highlight the influence of nearby local objects on the semantics of the main subject and weaken the interference of distant objects; integrating the multi-dimensional information and weight values ​​of local objects makes the generated local object features more targeted and effective, thus improving the quality of visual feature information.

[0046] It is understandable that the modeling effect of inter-tag dependency relationships directly affects the accuracy of semantic prediction. In order to obtain high-quality semantic information, based on the above embodiments, this embodiment also provides how to make the semantic tag graph not only include its own definition, but also absorb and reflect the implementation process of complex dependency relationships in the entire tag system, which may include the following:

[0047] The learning process of the semantic label graph includes: using the initial semantic representation features of each semantic label as the initial features of the corresponding graph nodes of the semantic label graph, and updating the features of each graph node through multiple rounds of feature propagation and transformation; wherein, in each round of update, the features of each graph node are obtained by aggregating the feature information of its connected neighbor nodes, and performing linear transformation and nonlinear activation processing on the aggregated features; after multiple rounds of update, a label embedding representation vector that can reflect the semantic and structural information of the semantic label in the semantic label graph is obtained.

[0048] The initial semantic representation features are the initial feature vectors of the semantic labels, generated by encoding the label definition text using natural language models (such as BERT (Bidirectional Encoder Representation Network Model based on Transformer Model) and LLM (Large Language Model)). These features possess rich linguistic semantic information. Feature propagation and transformation are the node feature transformation processes in graph neural networks, including neighbor node feature aggregation, linear transformation, and nonlinear activation processing. Aggregation refers to a node collecting the current feature information of all its directly connected neighbor nodes via edges. For example, for a keyboard node, features from neighbor nodes such as computer, mouse, and desktop are aggregated. Aggregation methods can include weighted summation of neighbor features based on edge weights (such as similarity or co-occurrence strength). Transformation involves the aggregated information representing the node's current local network context. Subsequently, this aggregated information undergoes a linear transformation (i.e., multiplication with a learnable weight matrix) to combine and refine the information. Then, a nonlinear activation process (such as ReLU) introduces nonlinear fitting capabilities into the model. The label embedding representation vector is a label feature vector obtained after multiple rounds of feature updates, which can simultaneously reflect the semantic and structural information of the semantic label in the semantic label graph.

[0049] In this embodiment, a large language model can be used to encode the definition text of each semantic label to generate an initial feature vector with semantic expressive capabilities. This vector is then used as the initial feature of the corresponding node in the semantic label graph. Regarding edge construction, two types of structures are considered: one is semantic edges generated between labels based on the cosine similarity of the embedding space; the other is co-occurrence edges calculated based on the common frequency of labels in the training data. These two types of edges jointly reflect the semantic and statistical correlation between labels. The network parameters of the semantic label graph are initialized, including the adjacency matrix and the linear transformation parameter matrix. In each round of updates, each node aggregates the feature information of its connected neighbor nodes to obtain aggregated features; the aggregated features are then subjected to linear transformation (through a learnable parameter matrix) and nonlinear activation processing (such as ReLU) to generate the updated features of the current node. This feature update process is repeated for a preset number of rounds (e.g., 3-5 rounds), after which the update stops. At this point, the feature vector of each node is the label embedding representation vector, which comprehensively reflects the semantic information of the semantic label and its structural association in the graph. After multiple rounds of such updates, the features of each node are iteratively fused with information from its multi-hop neighbors. The resulting label embedding vector is no longer merely the original textual semantics of the label, but a comprehensive representation encoding its position and surrounding relationships within the label relationship network. This reflects the semantic and structural information of the semantic label in the semantic label graph. This structurally rich embedding vector allows for more reasonable matching by implicitly utilizing the association knowledge between labels when calculating the similarity between images and labels.

[0050] For example, the semantic label graph is learned through a feature update relation, which can be represented as: in, It is the normalized adjacency matrix. For activation function, It is the learnable parameter matrix for each layer. Indicates the first After layer propagation and transformation, the matrix consists of the embedding vectors of all semantic tags. Indicates the first The matrix formed by the embedding vectors of all semantic tags after layer propagation and transformation.

[0051] The normalized adjacency matrix is ​​obtained by normalizing the original adjacency matrix of the semantic label graph. It eliminates the influence of node degree differences on feature propagation, ensuring fairness in feature updates across nodes. The activation function (such as ReLU or Sigmoid) introduces non-linear transformations, enhancing the network's ability to fit complex features and preventing the model from falling into linear inseparability problems. The learnable parameter matrix is ​​the weight matrix of each layer of the semantic label graph, continuously updated through backpropagation during model training. It is used to perform linear transformations on the aggregated features. The embedding vector matrix represents the matrix composed of all semantic label embedding vectors after feature propagation and transformation at each network layer. Each row of the matrix corresponds to a feature vector of a semantic label.

[0052] In this embodiment, a graph neural network is used to learn the constructed heterogeneous graph. Based on this feature update formula, information from adjacent labels is continuously aggregated through multi-layer graph convolution operations, thereby improving the semantic expressiveness of each label representation. Through multiple rounds of feature propagation and transformation, label nodes can aggregate the semantic and structural information of their neighboring nodes. The generated label embedding vectors possess superior semantic expressiveness and structural correlation, providing a high-quality feature foundation for subsequent semantic matching and prediction, and improving the effectiveness of label relationship modeling. This enables the generated label embedding vectors to more accurately capture the semantic correlations and structural dependencies between labels, enhancing the representational power of the semantic label graph.

[0053] Based on the above embodiments, this application also provides a method for modularizing or systematizing the entire image semantic information generation method to form a complete and deployable technical solution, which may include the following:

[0054] An image semantic information generation model is pre-constructed, comprising an input layer, an image semantic feature extraction layer, a semantic label map, a nonlinear mapping network, and an output layer. The input layer, connected to the input of the image semantic feature extraction layer, serves as the entry point for the model, receiving the image to be processed and transmitting it to the image semantic feature extraction layer, thus realizing the input and transmission of image data. The output of the image semantic feature extraction layer is connected to the input of the semantic label map and the nonlinear mapping network, respectively, inputting multi-dimensional image semantic features into these networks. The outputs of the semantic label map and the nonlinear mapping network are connected to the output layer. The image semantic information generation model is trained using publicly available training samples or training samples constructed according to actual needs. Once the image semantic information generation model is trained, the image to be processed is input into the model. The image semantic feature extraction layer generates multi-dimensional image semantic features based on the target subject and its context information of the image to be processed. The semantic label map processes the multi-dimensional image semantic features and outputs the generated semantic representation information to the output layer. The nonlinear mapping network performs nonlinear mapping processing on the multi-dimensional image semantic features and outputs the generated semantic space alignment prediction result to the output layer. The output layer fuses the semantic space alignment prediction result and the semantic representation information and outputs the image semantic information of the image to be processed.

[0055] This embodiment achieves end-to-end processing of image feature extraction, label relationship modeling, bidirectional prediction, and result fusion by constructing a model architecture. The modules work together to reduce the loss of intermediate data transmission and improve the efficiency of semantic information generation. At the same time, the model integrates the advantages of multi-dimensional feature extraction and multi-view prediction, further improving the accuracy and stability of semantic information generation.

[0056] To further improve the performance of the image semantic information generation model, this embodiment also provides an exemplary training process for the image semantic information generation model, which may include the following:

[0057] Obtain an image sample dataset carrying real semantic labels and input the image sample dataset into the image semantic information generation model according to a preset batch size; obtain the multi-label prediction results output by the image semantic information generation model and the label embedding sample vectors generated from the semantic label graph; determine the first loss term based on the difference between the multi-label prediction results and the corresponding real semantic labels; determine the second loss term based on the difference between the label embedding sample vectors and the predefined label semantic associations and statistical co-occurrence relationships; combine the first loss term and the second loss term with weights to determine the total loss function; based on the total loss function, update the network parameters of the image semantic information generation model simultaneously through the backpropagation algorithm until the model iteration training stopping condition is reached.

[0058] The image sample dataset is a collection of image samples carrying real semantic labels, used for model training and parameter optimization. The samples must cover various scenes and semantic types to ensure the model's generalization ability. The preset batch size refers to the number of image samples input to the model each time during training (e.g., 32, 64, etc.), used to balance training efficiency and model convergence. The multi-label prediction result refers to the semantic prediction result output by the image semantic information generation model for the input image samples, consistent with the dimension of the real semantic labels. The label embedding sample vector is the label embedding vector generated after processing the image sample features of the semantic label map, used to reflect the semantic and structural representation of the labels during training. The first loss term is the loss value calculated based on the difference between the multi-label prediction result and the real semantic labels, used to supervise the accuracy of the model's prediction results. The second loss term is the loss value calculated based on the difference between the label embedding sample vector and the preset label association relationship, used to supervise the rationality of the label embedding representation. The total loss function is a weighted combination of the first and second loss terms, used to guide the update of the overall model parameters. Backpropagation is a parameter update algorithm used in model training. It optimizes the model by calculating the gradient of the total loss function with respect to each network parameter and updating the parameters along the gradient descent direction. Iterative training termination conditions are preset conditions for terminating model training, such as reaching a preset number of training epochs, the total loss function value falling below a set threshold, or the validation set accuracy no longer improving.

[0059] In this embodiment, the training process of the image semantic information generation model is as follows: Figure 2 As shown, the first step is to acquire an image sample dataset. The image samples in this dataset undergo data input and preprocessing to provide a unified and standardized input image for subsequent feature extraction. The original image samples are first normalized in size and standardized in pixel values ​​to ensure consistency in the input distribution during model training or inference. Image preprocessing: Image samples are normalized in size to a preset fixed size; then, pixel values ​​are standardized to ensure the pixel value distribution meets the model's processing requirements. Next, an object detection model is used to identify the main figures or key object regions in the image samples and extract their corresponding image patches. This region serves as the basis for subsequent visual semantic modeling. In the preprocessed image, the bounding box location of the target subject is determined using an object detection model, and the corresponding target patch is extracted based on the bounding box. A convolutional neural network (such as ResNet) is used to encode the target patch, outputting the deep visual features of the target subject. After extracting this region, a convolutional neural network (such as ResNet) is used to encode it to obtain the deep visual features of the target subject. Simultaneously, the model detects other local objects in the image, acquiring their category, location, and visual features. To reasonably reflect the semantic contribution of different objects to the main region, a spatial distance-based attention weighting mechanism is introduced. The importance of each object is dynamically weighted based on distance, thereby constructing local contextual fusion features. Besides local regions, images also contain a wealth of global scene information, such as background, environment, lighting, and atmosphere. This global information also significantly impacts image semantics. Therefore, utilizing panoramic images to extract global semantic features and depth information is crucial for semantic feature extraction. Deep features can be obtained through networks with semantic encoding capabilities (such as ResNet), while... The depth is estimated by a monocular depth estimation model, reflecting the relative spatial structure of each pixel in the scene. The aforementioned main features, local object features, and global features are then concatenated to obtain a unified multi-faceted semantic representation of the image sample. concat means concatenation, and can be represented as: This serves as the foundation for the model to understand multi-label information in images. Correspondingly, the image semantic feature extraction layer can include object detection models, convolutional neural networks, and monocular depth estimation models. In multi-label tasks, complex semantic relationships often exist between labels, such as synonymy, near-synonymy, opposition, and co-occurrence. To model these dependencies between labels, a heterogeneous graph structure with labels as nodes is constructed: nodes represent the label definition text embedding vectors obtained through a large language model, possessing rich linguistic semantics. The edges between nodes include two types of structures: one is semantic edges generated based on the cosine similarity of the embedding space between labels; the other is co-occurrence edges calculated based on the frequency of co-occurrence of labels in the training data. These two types of edges jointly reflect the correlation between labels at both the semantic and statistical levels. To fully utilize the semantic relationships between image information and labels, a dual-path prediction mechanism is designed for label judgment. The first path is to extract the semantic representation of the image... Projected into the label embedding space, with each label Calculate a similarity score to determine the degree of matching between the image and the label. The second approach is to use a multilayer perceptron (MLP) to... The model is directly mapped to the same output dimension as the number of labels, enabling end-to-end prediction. In constructing the loss function, a first loss term is used to ensure the alignment between the model output and the actual labels. To further constrain the semantic representation of the label embeddings to be consistent with the co-occurrence structure, semantic consistency loss and co-occurrence consistency loss are used to optimize the rationality of edge weight generation and embedding structure during graph learning; these two together constitute the second loss term.

[0060] Based on the above embodiments, to ensure the alignment between the model output and the actual labels, this embodiment can use binary cross-entropy loss to independently supervise each label. This loss independently measures the difference between the model's prediction for each label and the actual situation, independently supervising the prediction accuracy of each label and driving the model's final output to be as close as possible to the true label. Accordingly, the first loss term can be expressed as: In the formula, Indicates the first loss. Indicates the first N image samples are used to identify the sequence number of different image samples in the training batch, with values ​​ranging from 1 to N. The number of image samples in the training batch. Indicates the first The multi-label prediction result for each image sample is a label prediction vector for that image sample, where each element represents the predicted probability of the corresponding label. Indicates the first The true semantic label of an image sample is the actual semantic label vector of the image sample. For example, each element in the vector can be 0 or 1, which means that the label does not exist or exists, respectively.

[0061] This embodiment uses a binary cross-entropy loss function to calculate the first loss term, which can independently measure the difference between the predicted result and the actual situation of each label, and accurately capture the prediction error of each label. By averaging the loss of batch samples, the stability and reliability of the loss value are ensured, providing an accurate supervision signal for optimizing the model's prediction accuracy and improving the model's prediction accuracy for each label.

[0062] For example, the second loss term in this embodiment includes semantic consistency loss and co-occurrence consistency loss. Semantic consistency loss measures the difference between the cosine similarity between the label embeddings learned based on the graph neural network and the initial semantic similarity of the labels. Co-occurrence consistency loss measures the consistency between the predicted embedding structure and the statistical co-occurrence of the labels. The second loss term can be expressed as: in, In the formula, This indicates the second loss. This represents the semantic consistency loss. This indicates the co-occurrence consistency loss. The number of image samples in the training batch. The cosine similarity between predicted label embeddings is calculated from the label embedding sample vectors generated by the model, reflecting the degree of semantic association between labels learned by the model. The semantic similarity between tag definitions is based on the cosine similarity pre-calculated from the tag definition text, reflecting the inherent semantic connections between tags. This represents the cosine similarity between predicted label embedding structures. It is calculated based on the structure matrix constructed from the label embedding sample vectors, reflecting the structural associations between labels learned by the model. The superscript indicates the label co-occurrence value, which is a numerical value calculated based on the frequency of co-occurrence of labels in the training data. It reflects the actual co-occurrence relationship between labels. Represents a label or subscript. Let be the index of the image sample. The final loss function can be a weighted sum of the two, and can be expressed as: ,in This is the balance coefficient.

[0063] This embodiment, through the design of a dual loss term, not only supervises the alignment between the prediction results and the true labels, but also constrains the consistency between the label embedding representation and the semantic association and co-occurrence relationship between labels. This ensures that the label embedding vector conforms to both the inherent semantic association of the labels and the actual statistical co-occurrence relationship, achieving comprehensive optimization of the model parameters. The parameter update based on the backpropagation algorithm ensures that the model can quickly converge to the optimal state, improving the model's generalization ability and the stability and accuracy of semantic information generation.

[0064] It should be noted that there is no strict order of execution for the steps in this application. As long as they conform to a logical order, these steps can be executed simultaneously or in a certain preset order. Figures 1-2 This is just an illustrative example and does not mean that this is the only possible execution order.

[0065] This application also provides a corresponding apparatus for the image semantic information generation method, further enhancing the practicality of the method. The apparatus can be described from both a functional module perspective and a hardware perspective. The image semantic information generation apparatus provided in this application is described below. This apparatus is used to implement the image semantic information generation method provided in this application. In this embodiment, the image semantic information generation apparatus may include or be divided into one or more program modules. These one or more program modules are stored in a storage medium and executed by one or more processors to complete the image semantic information generation method disclosed in Embodiment 1. The program module referred to in this embodiment is a series of computer program instruction segments capable of performing specific functions, which is more suitable than the program itself for describing the execution process of the image semantic information generation apparatus in the storage medium. The following description will specifically introduce the functions of each program module in this embodiment. The image semantic information generation apparatus described below can be referred to in correspondence with the image semantic information generation method described above.

[0066] From the perspective of functional modules, see Figure 3 , Figure 3This is a structural diagram of the image semantic information generation device provided in this embodiment under a specific implementation. The device may include:

[0067] The image feature representation module 301 is used to generate multi-dimensional image semantic features based on the target subject of the image to be processed and its contextual information.

[0068] The semantic representation information generation module 302 is used to input multi-dimensional image semantic features into the semantic label graph to obtain semantic representation information that represents the degree of similarity or association between the image to be processed and each semantic label. The semantic label graph is a heterogeneous graph neural network that uses semantic labels as graph nodes, uses label semantic description features as graph node features, and determines the node connection edges based on the semantic similarity and co-occurrence relationship between semantic labels.

[0069] The semantic prediction result generation module 303 is used to input multi-dimensional image semantic features into a nonlinear mapping network to obtain semantic space aligned prediction results with the same number of labels.

[0070] The fusion generation module 304 is used to fuse the semantic space alignment prediction results and semantic representation information to generate image semantic information corresponding to the image to be processed.

[0071] For example, in some embodiments of this example, the semantic representation information generation module 302 can also be used for: the learning process of the semantic tag graph includes: using the initial semantic representation features of each semantic tag as the initial features of the corresponding graph nodes of the semantic tag graph, and updating the features of each graph node through multiple rounds of feature propagation and transformation; wherein, in each round of update, the features of each graph node are obtained by aggregating the feature information of its connected neighbor nodes, and performing linear transformation and nonlinear activation processing on the aggregated features; after multiple rounds of update, a tag embedding representation vector that can reflect the semantic and structural information of the semantic tag in the semantic tag graph is obtained.

[0072] As an exemplary implementation of the above embodiments, the semantic representation information generation module 302 can also be used to: learn the semantic label graph through a feature update relation, wherein the feature update relation is: in, It is the normalized adjacency matrix. For activation function, It is the learnable parameter matrix for each layer. Indicates the first After layer propagation and transformation, the matrix consists of the embedding vectors of all semantic tags. Indicates the first The matrix consisting of the embedding vectors of all semantic tags after layer propagation and transformation.

[0073] For example, in some other embodiments of this embodiment, the image feature representation module 301 can also be used to: acquire the visual features of the target subject of the image to be processed, the semantic category, visual features and spatial distance information between each target local object and the target subject, as visual feature information; acquire the semantic information and depth features of the global scene of the image to be processed, as global feature information; and fuse the visual feature information and the global feature information as multi-dimensional image semantic features.

[0074] As an exemplary implementation of the above embodiments, the image feature representation module 301 can also be used to: determine the target subject in the image to be processed after size normalization and pixel value standardization, and extract the corresponding target patch; encode the target patch to obtain the visual features of the target subject; determine the weight value of each target local object according to the spatial distance information between each target local object and the target subject, in a manner that the weight decreases as the distance increases; and obtain the local object features according to the semantic category, visual features, position information and weight value of each target local object.

[0075] For example, in some other embodiments of this example, the above-mentioned apparatus may further include an end-to-end generation module for inputting the image to be processed into an image semantic information generation model; the image semantic information generation model includes an input layer, an image semantic feature extraction layer, a semantic label map, a nonlinear mapping network, and an output layer; the input layer is connected to the input of the image semantic feature extraction layer and is used to input the image to be processed into the image semantic feature extraction layer; the output of the image semantic feature extraction layer is connected to the input of the semantic label map and the nonlinear mapping network respectively, and inputs multi-dimensional image semantic features into the semantic label map and the nonlinear mapping network respectively; the output of the semantic label map and the output of the nonlinear mapping network are connected to the output layer; the image semantic feature extraction layer generates multi-dimensional image semantic features based on the target subject and its context information of the image to be processed; the semantic label map processes the multi-dimensional image semantic features and outputs the generated semantic representation information to the output layer; the nonlinear mapping network performs nonlinear mapping processing on the multi-dimensional image semantic features and outputs the generated semantic space alignment prediction result to the output layer; the output layer fuses the semantic space alignment prediction result and the semantic representation information and outputs the image semantic information of the image to be processed.

[0076] As an exemplary implementation of the above embodiments, the end-to-end generation module can also be used for: the training process of the image semantic information generation model includes: acquiring an image sample dataset carrying real semantic labels, and inputting the image sample dataset into the image semantic information generation model according to a preset batch size; acquiring the multi-label prediction results output by the image semantic information generation model and the label embedding sample vector generated by the semantic label graph; determining a first loss term based on the difference between the multi-label prediction results and the corresponding real semantic labels; determining a second loss term based on the difference between the label embedding sample vector and the predefined label semantic association and statistical co-occurrence relationship; weighting and combining the first loss term and the second loss term to determine the total loss function; and updating the network parameters of the image semantic information generation model simultaneously through the backpropagation algorithm based on the total loss function until the model iteration training stopping condition is reached.

[0077] As an exemplary implementation of the above embodiments, the first loss term is: In the formula, Indicates the first loss. Indicates the first Image samples, The number of image samples in the training batch. Indicates the first Multi-label prediction results for each image sample Indicates the first The true semantic labels of the image samples.

[0078] As another exemplary implementation of the above embodiments, the second loss term is: in, In the formula, Indicates the second loss. This represents the semantic consistency loss. This indicates the co-occurrence consistency loss. The number of image samples in the training batch. This represents the cosine similarity between predicted label embeddings. Define semantic similarity between tags. This represents the cosine similarity between predicted label embedding structures. Indicates the co-occurrence value of the label statistics, superscript Represents a label or subscript. This is the index of the image sample.

[0079] The image semantic information generation device mentioned above is described from the perspective of functional modules. Furthermore, this application also provides an electronic device, which is described from the perspective of hardware. Figure 4This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. The electronic device includes a memory 40 for storing a computer program; and a processor 41 for executing the computer program to implement the steps of the image semantic information generation method mentioned in any of the above embodiments.

[0080] The processor 41 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 41 may also be a controller, microcontroller, microprocessor, or other data processing chip. The processor 41 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 41 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 41 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, the processor 41 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0081] The memory 40 may include one or more computer non-volatile storage media, which may be non-transitory. The memory 40 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the memory 40 may be an internal storage unit of an electronic device, such as a server hard drive. In other embodiments, the memory 40 may be an external storage device of an electronic device, such as a plug-in hard drive on a server, a Smart Media Card (SMC), a Secure Digital (SD) card, or a Flash Card. Furthermore, the memory 40 may include both internal and external storage units of the electronic device. The memory 40 can be used not only to store application software and various types of data installed on the electronic device, such as code in the process of executing the image semantic information generation method, but also to temporarily store data that has been output or will be output. In this embodiment, the memory 40 is used to store at least the following computer program 401, which, after being loaded and executed by the processor 41, is capable of implementing the relevant steps of the image semantic information generation method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 40 may also include an operating system 402 and data 403, and the storage method may be temporary storage or permanent storage. The operating system 402 may include Windows, Unix, Linux, etc. The data 403 may include, but is not limited to, data corresponding to the image semantic information generation results.

[0082] In some embodiments, the aforementioned electronic device may further include a display screen 42, an input / output interface 43, a communication interface 44 (or network interface), a power supply 45, and a communication bus 46. The display screen 42 and input / output interface 43, such as a keyboard, are user interfaces. Exemplary user interfaces may also include standard wired interfaces, wireless interfaces, etc. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a display screen or display unit, used to display information processed in the electronic device and to display a visual user interface. The communication interface 44 may exemplary include wired and / or wireless interfaces, such as a Wi-Fi interface, a Bluetooth interface, etc., typically used to establish communication connections between the electronic device and other electronic devices. The communication bus 46 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0083] Those skilled in the art will understand that Figure 4 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, such as sensors 47 that perform various functions.

[0084] It is understood that if the image semantic information generation method in the above embodiments is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes, but is not limited to, various media capable of storing program code, such as: USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, removable disk, CD-ROM, magnetic disk, or optical disk. Based on this, this application also provides a non-volatile storage medium storing a computer program, which, when executed by a processor, performs the steps of the image semantic information generation method as described in any of the above embodiments.

[0085] It is understood that if the image semantic information generation method in the above embodiments is implemented as a software functional unit and sold or used as an independent product, the computer software product may not need to be stored in a physical storage medium. For example, it can be directly transmitted to a computer or other device with information processing capabilities via a wired or wireless network to execute all or part of the steps of the methods in the various embodiments of this application. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, or all or part of the technical solution, can be embodied in the form of a software product. Based on this, this application also provides a computer program product, which stores a computer program, and when the computer program is executed by a processor, it performs the steps of the image semantic information generation method as described in any of the above embodiments.

[0086] The foregoing has provided a detailed description of an image semantic information generation method and electronic device provided in this application. The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Whether the units and algorithm steps of the various examples described in the disclosed embodiments are executed by electronic hardware or computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, and such implementations should not be considered beyond the scope of this application. Several improvements and modifications can be made to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for generating semantic information from an image, characterized in that, include: Generate multi-dimensional image semantic features based on the target subject and its contextual information of the image to be processed; The multi-dimensional image semantic features are input into the semantic label graph to obtain semantic representation information that represents the degree of similarity or association between the image to be processed and each semantic label; the semantic label graph is a heterogeneous graph neural network that uses semantic labels as graph nodes, uses label semantic description features as graph node features, and determines the node connection edges based on the semantic similarity and co-occurrence relationship between semantic labels. The node connection edges are determined by the semantic similarity between semantic labels and the co-occurrence relationship obtained statistically from the training data; the semantic label graph is learned through a feature update relation, which is: in, This is the normalized adjacency matrix, obtained by normalizing the original adjacency matrix of the semantic tag graph. For activation function, It is the learnable parameter matrix for each layer. Indicates the first After layer propagation and transformation, the matrix consists of the embedding vectors of all semantic tags. Indicates the first The matrix consisting of the embedding vectors of all semantic tags after layer propagation and transformation; The multi-dimensional image semantic features are input into a nonlinear mapping network to obtain a semantic space alignment prediction result with the same dimension as the number of labels. The semantic space alignment prediction result and the semantic representation information are fused to generate image semantic information corresponding to the image to be processed; the semantic representation information calculates the label probability through the semantic similarity between the image semantic features and the label embedding; and the semantic space alignment prediction result generates the label probability through an end-to-end nonlinear mapping network.

2. The image semantic information generation method according to claim 1, characterized in that, The learning process of the semantic label graph includes: The initial semantic representation features of each semantic label are used as the initial features of the corresponding graph nodes in the semantic label graph. The features of each graph node are updated through multiple rounds of feature propagation and transformation. In each round of update, the features of each graph node are obtained by aggregating the feature information of its connected neighbor nodes and performing linear transformation and nonlinear activation processing on the aggregated features. After multiple rounds of updates, a tag embedding representation vector reflecting the semantic and structural information of the semantic tag in the semantic tag graph is obtained.

3. The image semantic information generation method according to claim 1, characterized in that, Based on the target subject and its contextual information in the image to be processed, multi-dimensional image semantic features are generated, including: The visual features of the target subject in the image to be processed, the semantic category, visual features, and spatial distance information between each local object and the target subject are obtained as visual feature information. Obtain the semantic information and depth features of the global scene of the image to be processed, as global feature information; The fusion of the visual feature information and the global feature information is used as a multi-dimensional image semantic feature.

4. The image semantic information generation method according to claim 1, characterized in that, The visual features of the target subject in the image to be processed, the semantic category, visual features, and spatial distance information between each local object of the target and the target subject are obtained, including: In the image to be processed after size normalization and pixel value standardization, the target subject is identified and the corresponding target patch is extracted; The target image patch is encoded to obtain the visual features of the target subject; Based on the spatial distance information between each target local object and the target main body, the weight value of each target local object is determined in a manner that the weight decreases as the distance increases; Based on the semantic category, visual features, location information, and weight values ​​of each target local object, local object features are obtained.

5. The image semantic information generation method according to any one of claims 1 to 4, characterized in that, include: The image to be processed is input into the image semantic information generation model; The image semantic information generation model includes an input layer, an image semantic feature extraction layer, a semantic label map, a nonlinear mapping network, and an output layer. The input layer is connected to the input of the image semantic feature extraction layer and is used to input the image to be processed into the image semantic feature extraction layer. The output of the image semantic feature extraction layer is connected to the input of the semantic label map and the nonlinear mapping network, respectively, and inputs the multi-dimensional image semantic features into the semantic label map and the nonlinear mapping network. The output of the semantic label map and the output of the nonlinear mapping network are connected to the output layer. The image semantic feature extraction layer generates multi-dimensional image semantic features based on the target subject and its context information of the image to be processed. The semantic label map processes the multi-dimensional image semantic features and outputs the generated semantic representation information to the output layer. The nonlinear mapping network performs nonlinear mapping processing on the multi-dimensional image semantic features and outputs the generated semantic space alignment prediction result to the output layer. The output layer fuses the semantic space alignment prediction result with the semantic representation information and outputs the image semantic information of the image to be processed.

6. The image semantic information generation method according to claim 5, characterized in that, The training process of the image semantic information generation model includes: Obtain an image sample dataset carrying real semantic labels, and input the image sample dataset into the image semantic information generation model according to a preset batch size; Obtain the multi-label prediction results output by the image semantic information generation model and the label embedding sample vector generated from the semantic label graph; The first loss term is determined based on the difference between the multi-label prediction result and the corresponding real semantic label; The second loss term is determined based on the difference between the embedded sample vector and the predefined semantic association and statistical co-occurrence relationship of the labels; The first loss term and the second loss term are weighted and combined to determine the total loss function; Based on the total loss function, the network parameters of the image semantic information generation model are updated simultaneously through the backpropagation algorithm until the model iteration training stops.

7. The image semantic information generation method according to claim 6, characterized in that, The first loss term is: In the formula, Indicates the first loss. Indicates the first Image samples, The number of image samples in the training batch. Indicates the first Multi-label prediction results for each image sample Indicates the first The true semantic labels of the image samples.

8. The image semantic information generation method according to claim 6, characterized in that, The second loss item is: in, In the formula, Indicates the second loss. This represents the semantic consistency loss. This indicates the co-occurrence consistency loss. The number of image samples in the training batch. This represents the cosine similarity between predicted label embeddings. Define semantic similarity between tags. This represents the cosine similarity between predicted label embedding structures. Indicates the co-occurrence value of the label statistics, superscript Represents a label or subscript. This is the index of the image sample.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the image semantic information generation method as described in any one of claims 1 to 8.

Images