Image semantic extraction method and device, computer equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence, and relates to an image semantic extraction method, which comprises the following steps: acquiring paired sample images and description texts, and preprocessing the sample images and the description texts to obtain input images and input texts; respectively carrying out feature extraction on the input image and the input text to obtain image features and text features; performing feature fusion on the text features and the image features to obtain fusion features; inputting the fusion features into an initial diffusion model for model training until the initial diffusion model meets a preset condition, and obtaining an optimal diffusion model; and obtaining image features of a to-be-processed image, and inputting the image features of the to-be-processed image into the diffusion model for semantic text generation to obtain a semantic text of the to-be-processed image. The invention further provides an image semantic extraction device, computer equipment and a storage medium. According to the method, the correlation between the image and the description text is fully mined, the generated semantic text is higher in accuracy, and the generation speed is higher.

Description

Technical Field

[0001] The present application relates to the technical fields of artificial intelligence and image semantic extraction, and in particular to an image semantic extraction method, apparatus, computer equipment and storage medium. Background Art

[0002] Image-to-text system, also often called image description generation or image caption generation, is a technology that uses computer vision and natural language processing technology to automatically generate descriptive text from images. This process involves extracting semantic information from images, building a text generation model guided by image semantics, and converting the output of the model into a natural language description that humans can understand. Among them, in medical imaging, the image-to-text system can automatically recognize and annotate text information in medical images, such as lesion marks on X-rays, CT scan results, etc., to assist doctors in making diagnosis and treatment decisions. This not only improves the accuracy of diagnosis, but also greatly reduces the workload of doctors.

[0003] Most of the current mainstream image-to-text systems use an encoder-decoder architecture, the core idea of ​​which is to use an image encoder to extract the semantic information of an image, and then use a text decoder to decode the image semantic information to generate the corresponding description text of the image. This type of method usually generates image description text based on an autoregressive model, but the autoregressive generation model has the problem of error accumulation and propagation, which makes the encoder-decoder architecture image-to-text system more likely to generate erroneous description text. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose an image semantic extraction method, apparatus, computer equipment and storage medium to solve the technical problem of a high error rate of the description text generated by the image-to-text system in the prior art.

[0005] In order to solve the above technical problems, the present application embodiment provides an image semantic extraction method, which adopts the following technical solution:

[0006] Acquire a pair of sample images and description texts, and preprocess the sample images and description texts to obtain input images and input texts;

[0007] Extracting features from the input image and the input text respectively to obtain image features and text features;

[0008] Performing feature fusion on the text features and the image features to obtain fused features;

[0009] The fused features are input into an initial diffusion model, and in the forward diffusion process of the initial diffusion model, the text features in the fused features are subjected to noise processing to obtain noisy text features; in the reverse diffusion process of the initial diffusion model, the noisy text features are subjected to noise prediction to obtain predicted noise text;

[0010] According to a preset semantic loss function, the semantic loss of the noisy text feature and the predicted noisy text is calculated, and the model parameters of the initial diffusion model are iteratively optimized according to the semantic loss until the initial diffusion model meets the preset conditions, thereby obtaining an optimal diffusion model;

[0011] The image features of the image to be processed are obtained, and the image features of the image to be processed are input into the diffusion model to generate semantic text, so as to obtain the semantic text of the image to be processed.

[0012] Furthermore, the preprocessing of the sample image and the description text to obtain the input image and the input text includes:

[0013] Performing normalization processing on the sample image to obtain the input image;

[0014] The description text is filled to obtain the input text.

[0015] Furthermore, the extracting features from the input image and the input text respectively includes:

[0016] Inputting the input image and the input text into a pre-trained CLIP model respectively, wherein the CLIP model includes an image encoder and a text encoder;

[0017] Extracting image representation from the input image by using the image encoder to obtain the image features;

[0018] The text encoder extracts text representation from the input image to obtain the text feature.

[0019] Furthermore, the text features and image features are fused to obtain fused features, including:

[0020] Calling a preset multimodal feature fusion network, wherein the multimodal feature fusion network includes an attention mechanism module and a fully connected module;

[0021] Inputting the text features and the image features into the multimodal feature fusion network, and extracting attention features of different dimensions from the text features and the image features based on the attention mechanism module, to obtain text attention features and image attention features of multiple dimensions;

[0022] The text attention features and image attention features of multiple dimensions are input into the fully connected module to obtain the fusion features.

[0023] Furthermore, the text attention features and image attention features of multiple dimensions are input into the fully connected module to obtain the fusion features, including:

[0024] The text attention features and image attention features of multiple dimensions are concatenated at a shallow layer to obtain shallow fusion features;

[0025] The text attention features and image attention features of multiple dimensions interact at a deep level to obtain deep fusion features;

[0026] The shallow fusion features and the deep fusion features are concatenated to obtain the fusion features.

[0027] Furthermore, the fully connected module includes an adaptive feature alignment unit and a fully connected unit, and the shallow fusion feature and the deep fusion feature are spliced ​​to obtain the fusion feature, including:

[0028] The shallow fusion features and the deep fusion features are respectively input into the adaptive feature alignment unit for feature alignment, and the weights of different modal features are dynamically adjusted to obtain shallow alignment features and deep alignment features;

[0029] The shallow alignment features and the deep alignment features are input into the fully connected unit to obtain the fusion features.

[0030] Furthermore, the fully connected module further includes a feature selection unit, which inputs the shallow fusion features and the deep fusion features into the adaptive feature alignment unit for feature alignment, dynamically adjusts the weights of different modal features, and obtains the text alignment features and the image alignment features, including:

[0031] The shallow fusion features and the deep fusion features are input into the feature selection unit, and the most relevant feature subset is screened based on the gating mechanism preset in the feature selection unit to obtain the optimal shallow fusion features and the deep fusion features.

[0032] In order to solve the above technical problems, the embodiment of the present application further provides an image semantic extraction device, which adopts the following technical solution:

[0033] A preprocessing module, used to obtain a pair of sample images and description texts, and preprocess the sample images and description texts to obtain input images and input texts;

[0034] A feature extraction module, used to extract features from the input image and input text respectively to obtain image features and text features;

[0035] A feature fusion module, used for fusing the text features and the image features to obtain fused features;

[0036] A prediction module, used for inputting the fused features into an initial diffusion model, performing noise processing on the text features in the fused features in the forward diffusion process of the initial diffusion model to obtain a noisy text feature; and performing noise prediction on the noisy text feature in the reverse diffusion process of the initial diffusion model to obtain a predicted noise text;

[0037] A parameter optimization module, used for calculating the semantic loss of the noisy text features and the predicted noisy text according to a preset semantic loss function, and iteratively optimizing the model parameters of the initial diffusion model according to the semantic loss until the initial diffusion model meets the preset conditions, thereby obtaining an optimal diffusion model;

[0038] The text generation module is used to obtain image features of the image to be processed, and input the image features of the image to be processed into the diffusion model for semantic generation to obtain the semantic text of the image to be processed.

[0039] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0040] The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of any one of the above image semantic extraction methods when executing the computer-readable instructions.

[0041] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0042] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the image semantic extraction method as described in any one of the above items are implemented.

[0043] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: by extracting image features and text features of sample images and corresponding description texts and fusing them, the diffusion model is trained based on the fused features, so that the diffusion model can learn more image semantic information, and at the same time, the text features and image features are fused, so that the two features are unified into the same feature space, which is conducive to achieving high-quality image-to-text effects, fully exploring the correlation between images and description texts, and obtaining more accurate semantic descriptions; and the use of the diffusion model to generate semantic text for images has a higher accuracy and a faster generation speed than the use of the autoregressive model to generate semantic text in the prior art. At the same time, the semantic text with higher accuracy can effectively help doctors make quick diagnosis and treatment decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0046] Figure 2 A flowchart of an embodiment of the image semantic extraction method according to the present application;

[0047] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S202;

[0048] Figure 4 yes Figure 2 A flowchart of a specific implementation of step S203;

[0049] Figure 5 yes Figure 4 A flowchart of a specific implementation of step S403;

[0050] Figure 6 yes Figure 5 A flowchart of a specific implementation of step S503;

[0051] Figure 7 is a structural schematic diagram of an embodiment of an image semantic extraction device according to the present application;

[0052] Figure 8 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0054] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0055] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0056] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0057] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0058] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0059] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0060] It should be noted that the image semantic extraction method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the image semantic extraction device is generally arranged in the server / terminal device.

[0061] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0062] Continue to refer Figure 2 , shows a flow chart of an embodiment of the image semantic extraction method according to the present application. The image semantic extraction method comprises the following steps:

[0063] Step S201 , obtaining a pair of sample images and description texts, and preprocessing the sample images and description texts to obtain input images and input texts.

[0064] In this embodiment, the electronic device (eg, Figure 1 The server / terminal device shown in the figure) can receive a request for obtaining a pair of sample images and description texts through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, Wi-Fi connection, Bluetooth connection, Wi MAX connection, Zigbee connection, UWB (Ultra Wi DeBand) connection, and other wireless connection methods currently known or to be developed in the future. Among them, the paired sample images and description texts can be called or downloaded from the local medical record database of the hospital or the Internet. The description text can be the diagnosis conclusion of the medical image, etc., and the image data can be related examination images.

[0065] Step S202: extract features from the input image and input text respectively to obtain image features and text features.

[0066] In this embodiment, in order to fully explore the complementarity between the input image and the input text, feature extraction is performed on the input image and the input text respectively to obtain corresponding image features and text features, which facilitates subsequent mining of correlations from the image features and text features.

[0067] Step S203, performing feature fusion on the text features and the image features to obtain fused features.

[0068] In this embodiment, the image features of the medical image provide visual detail information such as scenes, objects, and colors, and the text features contain high-level information such as semantics, emotions, and causality. Feature fusion of text features and image features can fully mine the semantic association information between the image features in the medical image and the text features of the corresponding descriptive text.

[0069] Step S204, inputting the fused features into the initial diffusion model, performing noise processing on the text features in the fused features in the forward diffusion process of the initial diffusion model to obtain noisy text features; performing noise prediction on the noisy text features in the reverse diffusion process of the initial diffusion model to obtain predicted noise text;

[0070] Step S205, according to a preset semantic loss function, calculate the semantic loss of the noisy text features and the predicted noisy text, and iteratively optimize the model parameters of the initial diffusion model according to the semantic loss until the initial diffusion model meets the preset conditions to obtain the optimal diffusion model.

[0071] In this embodiment, the diffusion model is a multimodal pre-trained neural network with efficient generation capabilities in multiple fields such as image generation, natural language processing, and sound synthesis. Therefore, the diffusion model is used in this application to realize the semantic text generation of medical images. An initial diffusion model is constructed and trained. The training process includes: inputting the fusion features obtained in the above steps into the initial diffusion model, performing noise processing on the text features in the fusion features during the forward diffusion process of the initial diffusion model, and not performing noise processing on the image features in the fusion features to obtain the noisy text features; predicting the noisy text features during the reverse diffusion process of the initial diffusion model, and keeping the image features unchanged. In the reverse diffusion process, the noisy text features are diffused in multiple steps to obtain the predicted noisy text. Among them, the image features remain unchanged during model training, which facilitates the model to learn the noisy text features and the corresponding relationship between the predicted noisy text and the image features.

[0072] Based on the noisy text features and the predicted noisy text, the semantic loss of the noisy text features and the predicted noisy text is calculated according to the preset semantic loss function to obtain the semantic loss, and the model parameters of the initial diffusion model are iteratively optimized according to the semantic loss; and by analogy, the model parameters of the initial diffusion model are continuously updated until the initial diffusion model meets the preset conditions, that is, the optimal diffusion model is obtained. Among them, when the similarity between the description text generated by the diffusion model and the real description text reaches the set threshold, the preset conditions for diffusion model training are met.

[0073] Step S206, acquiring image features of the image to be processed, and inputting the image features of the image to be processed into the diffusion model to generate semantic text, so as to obtain the semantic text of the image to be processed.

[0074] In this embodiment, an image to be processed is obtained, and feature extraction is performed on the image to be processed to obtain image features of the image to be processed; the target image features are input into a trained diffusion model to generate semantic text, and the semantic text of the image to be processed is obtained. When the diffusion model is trained and put into use, the semantic text of the image to be processed can be generated by simply inputting the image features of the image to be processed into the diffusion model, without obtaining the description text corresponding to the image to be processed, and without performing text feature extraction of the description text and fusion of the image features and text features.

[0075] The present application extracts image features and text features of sample images and corresponding description texts, fuses them, trains a diffusion model based on the fused features, enables the diffusion model to learn more image semantic information, and fuses text features and image features, unifying the two features into the same feature space, which is conducive to achieving high-quality image-to-text effects, fully mining the correlation between images and description texts, and obtaining more accurate semantic descriptions; and the diffusion model is used to generate semantic texts for images. Compared with the prior art of using an autoregressive model to generate semantic texts, the generated semantic texts have higher accuracy and faster generation speed. At the same time, the semantic texts with higher accuracy can effectively help doctors make quick diagnosis and treatment decisions.

[0076] In some optional implementations of this embodiment, step 201 includes the following steps:

[0077] Step S2011, normalizing the sample image to obtain the input image;

[0078] Step S2012, filling the description text to obtain the input text.

[0079] In this embodiment, the preprocessing includes normalizing the sample images and padding the description texts. All sample images are normalized to ensure that all sample images have the same size; all acquired description texts are padded to make the lengths of the description texts in all samples consistent. By preprocessing the sample images and description texts, it is easier for the model to learn the sample images and description texts.

[0080] like Figure 3 As shown, in some optional implementations of this embodiment, step 202 includes the following steps:

[0081] Step S301, inputting the input image and the input text into a pre-trained CLIP model respectively, wherein the CLIP model includes an image encoder and a text encoder;

[0082] Step S3021, extracting image representation from the input image by using the image encoder to obtain the image features;

[0083] Step S3022: extract text representation from the input image by using the text encoder to obtain the text features.

[0084] In this embodiment, in order to better extract the features of the input image and the input text, a contrastive language-image pre-training (CLI P) model is used to extract features of the input image and the input text respectively. The CLI P model is a multimodal model that uses text information to train a zero-sample learning. The network structure of the CLI P model includes an image encoder and a text encoder.

[0085] This embodiment adopts the CLIP model to obtain text features and image features of semantic information exchange between input text and input image in the same feature space, effectively solving the problem that the diffusion model in the traditional method uses TransformerDecoder to achieve cross-modal control effect, avoiding the shortcoming of the slow generation speed of the diffusion model from being further amplified, and there is no need to rely on the cross-modal control capability of the cross-attention mechanism.

[0086] like Figure 4 As shown, in some optional implementations of this embodiment, step 203 includes the following steps:

[0087] Step S401, calling a preset multimodal feature fusion network, wherein the multimodal feature fusion network includes an attention mechanism module and a fully connected module;

[0088] Step S402, inputting the text features and the image features into the multimodal feature fusion network, and extracting attention features of different dimensions from the text features and the image features based on the attention mechanism module, to obtain text attention features and image attention features of multiple dimensions;

[0089] Step S403: input the text attention features and image attention features of multiple dimensions into the fully connected module to obtain the fusion features.

[0090] In this embodiment, the attention mechanism is an important technology in deep learning. Through the attention mechanism, the user can allow the model to selectively learn the key parts of image features and text features, improve the model's attention to important features, and then mine features. For image features and text features, a multimodal feature fusion network is constructed, and the semantic association between the two features is learned through the attention mechanism, and the complementary information between the features is mined to generate a fused multimodal feature representation. Among them, for text features, by setting different convolution kernel sizes such as 3, 4, 5, etc., text features of different dimensions are extracted to obtain a multi-scale text feature vector with a dimension of 512. For image features, by setting different pooling layer sizes, image features of different dimensions such as 64x64, 128x128, 256x256 are extracted. The extracted text features are input into the attention mechanism, and the text attention features of different dimensions are obtained by calculating the similarity between features of different dimensions. Similarly, the image features are input into the attention mechanism to obtain image attention features of different dimensions.

[0091] This embodiment uses a multimodal feature fusion network to learn the semantic association and complementarity between image features and text features, enhances the representation of the image and the corresponding description text possessed by the fused features, and helps to improve the accuracy of the description text generated by the diffusion model.

[0092] like Figure 5 As shown, in some optional implementations of this embodiment, step 403 includes the following steps:

[0093] Step S501, concatenating the text attention features and image attention features of multiple dimensions at a shallow layer to obtain shallow fusion features;

[0094] Step S502, interacting the text attention features and image attention features of multiple dimensions in a deep layer to obtain a deep fusion feature;

[0095] Step S503, concatenating the shallow fusion features and the deep fusion features to obtain the fusion features.

[0096] In this embodiment, in the shallow layer, the text attention features and image attention features of multiple dimensions are spliced, and the feature dimension after splicing is 1024, and then a 256-dimensional fully connected layer is passed to obtain the fused shallow features. In the deep layer, the text attention features and image attention features of multiple dimensions are input into the collaborative attention mechanism, and the interactive fusion of the two modal features is realized by calculating the attention weights between the text features and the image features. Finally, a 128-dimensional fully connected layer is passed to obtain the deep fusion features. The shallow fusion features and the deep fusion features are then fused in a cascade manner, and finally the fusion features of the text image fusion are obtained. In this way, the information of the two modalities of text and image is fully utilized to ensure the comprehensiveness of the fusion features. The staged fusion method of the shallow and deep layers takes into account both efficiency and fusion degree, forming an effective multimodal fusion method.

[0097] like Figure 6 As shown, in some optional implementations of this embodiment, step 503 includes the following steps:

[0098] Step 601, inputting the shallow fusion features and the deep fusion features into the adaptive feature alignment unit for feature alignment, dynamically adjusting the weights of different modal features, and obtaining shallow alignment features and deep alignment features;

[0099] Step 602: input the shallow alignment features and the deep alignment features into the fully connected unit to obtain the fusion features.

[0100] In this embodiment, in order to alleviate the problem of feature distribution differences, an adaptive feature alignment unit is set in the fully connected module, and the shallow fusion features and the deep fusion features are respectively input into the adaptive feature alignment unit for feature alignment, and the weights of the shallow fusion features and the deep fusion features are dynamically adjusted by the adaptive feature alignment unit to obtain shallow alignment features and deep alignment features; the obtained shallow alignment features and deep alignment features are input into the fully connected unit to obtain fusion features. Among them, the input end of the feature alignment unit is connected to the output end of the attention mechanism module, and the output end of the feature alignment module is connected to the input end of the fully connected unit.

[0101] In this embodiment, the shallow alignment features and the deep alignment features are aligned by an adaptive feature alignment unit, so as to effectively alleviate the feature distribution difference between the shallow alignment features and the deep alignment features, so that the fused features are smoother, which is beneficial to the representation learning of the fused features by the subsequent diffusion model.

[0102] In some optional implementations of this embodiment, before step 501, the following steps are included:

[0103] Step S500, inputting the shallow fusion features and the deep fusion features into the feature selection unit, screening the most relevant feature subset based on the gating mechanism preset in the feature selection unit, and obtaining the optimal shallow fusion features and the deep fusion features.

[0104] In this embodiment, considering that after the fusion of text alignment features and image alignment features, the fusion feature dimension obtained is high, which is easy to increase the model parameters and computational overhead, a feature selection unit based on the gating mechanism GRU is preset in the fully connected module, and the shallow fusion features and the deep fusion features are input into the feature selection unit. Based on the gating mechanism GRU, the most relevant feature subsets are adaptively selected from the shallow fusion features and the deep fusion features to obtain the optimal shallow fusion features and deep fusion features. Among them, the input end of the feature selection unit is connected to the output end of the attention mechanism module, the output end of the feature selection unit is connected to the input end of the feature alignment unit, and the output end of the feature alignment unit is connected to the input end of the fully connected unit. The gating mechanism GRU can effectively capture the long-term dependency between features by setting the update gate and the reset gate. The update gate controls the degree to which the state information of the previous moment is brought into the current state, and the reset gate controls the degree to which the state information of the previous moment is ignored. Through the gating mechanism, GRU can adaptively adjust the weight of the feature to realize the dynamic selection of the feature. For each input feature, GRU calculates its relevance score and retains the top 30% of the features with the highest score as the most relevant feature subset.

[0105] In this embodiment, a feature selection unit is used to screen out the most relevant feature subsets from the text attention features and image attention features based on a gating mechanism as the optimal text attention features and image attention features, effectively reducing the feature dimension after fusion and greatly reducing the model parameters and computational overhead.

[0106] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0107] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0108] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0109] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0110] Further references Figure 7 , as a response to the above Figure 2 The present application provides an embodiment of an image semantic extraction device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0111] like Figure 7 As shown, the image semantic extraction device 700 described in this embodiment includes: a preprocessing module 701, a feature extraction module 702, a feature fusion module 703, a prediction module 704, a parameter optimization module 705 and a text generation module 706. Among them:

[0112] A preprocessing module 701 is used to obtain a pair of sample images and description texts, and preprocess the sample images and description texts to obtain input images and input texts;

[0113] A feature extraction module 702 is used to extract features from the input image and input text to obtain image features and text features;

[0114] A feature fusion module 703 is used to fuse the text features and the image features to obtain fused features;

[0115] The prediction module 704 is used to input the fused features into the initial diffusion model, perform noise processing on the text features in the fused features in the forward diffusion process of the initial diffusion model to obtain noisy text features; and perform noise prediction on the noisy text features in the reverse diffusion process of the initial diffusion model to obtain predicted noise text;

[0116] A parameter optimization module 705 is used to calculate the semantic loss of the noisy text features and the predicted noisy text according to a preset semantic loss function, and iteratively optimize the model parameters of the initial diffusion model according to the semantic loss until the initial diffusion model meets the preset conditions, thereby obtaining an optimal diffusion model;

[0117] The text generation module 706 is used to obtain image features of the image to be processed, and input the image features of the image to be processed into the diffusion model to generate semantic text, so as to obtain the semantic text of the image to be processed.

[0118] In this embodiment, image features and text features of a sample image and a corresponding description text are extracted and fused, and a diffusion model is trained based on the fused features, so that the diffusion model can learn more image semantic information. At the same time, the text features and image features are fused, and the two features are unified into the same feature space, which is conducive to achieving high-quality image-to-text effects, fully exploring the correlation between the image and the description text, and obtaining a more accurate semantic description; and the use of the diffusion model to generate semantic text for the image is faster than the use of the autoregressive model to generate semantic text in the prior art.

[0119] In some optional implementations of this embodiment, the preprocessing module 701 includes:

[0120] A normalization unit, used for performing normalization processing on the sample image to obtain the input image;

[0121] A filling unit is used to fill the description text to obtain the input text.

[0122] In some optional implementations of this embodiment, the feature extraction module 702 includes:

[0123] An input unit, used to input the input image and the input text into a pre-trained CLIP model, wherein the CLIP model includes an image encoder and a text encoder;

[0124] An image feature extraction unit, configured to extract image representation from the input image through the image encoder to obtain the image feature;

[0125] A text feature extraction unit is used to extract text representation from the input image through the text encoder to obtain the text feature.

[0126] In some optional implementations of this embodiment, the feature fusion module 703 includes:

[0127] A calling unit, used to call a preset multimodal feature fusion network, wherein the multimodal feature fusion network includes an attention mechanism module and a fully connected module;

[0128] An attention feature extraction unit, used for inputting the text features and the image features into the multimodal feature fusion network, and extracting attention features of different dimensions from the text features and the image features based on the attention mechanism module, so as to obtain text attention features and image attention features of multiple dimensions;

[0129] A feature fusion unit is used to input the text attention features and image attention features of multiple dimensions into the fully connected module to obtain the fused features.

[0130] In some optional implementations of this embodiment, the feature fusion unit includes:

[0131] A shallow fusion unit, used for splicing the text attention features and image attention features of multiple dimensions at a shallow level to obtain shallow fusion features;

[0132] A deep fusion unit, used for interacting the text attention features and image attention features of multiple dimensions in a deep layer to obtain deep fusion features;

[0133] The multi-level feature fusion unit is used to combine the shallow fusion features and the deep fusion features to obtain the fusion features.

[0134] In some optional implementations of this embodiment, the multi-level feature fusion unit includes:

[0135] A feature alignment subunit, used for inputting the shallow fusion features and the deep fusion features into the adaptive feature alignment unit for feature alignment, dynamically adjusting the weights of features of different modalities, and obtaining shallow alignment features and deep alignment features;

[0136] The fully connected subunit is used to input the shallow alignment feature and the deep alignment feature into the fully connected unit to obtain the fusion feature.

[0137] In some optional implementations of this embodiment, the multi-level feature fusion unit further includes:

[0138] The feature selection subunit is used to input the shallow fusion features and the deep fusion features into the feature selection unit, screen the most relevant feature subset based on the gating mechanism preset in the feature selection unit, and obtain the optimal shallow fusion features and the deep fusion features.

[0139] To solve the above technical problems, the present application also provides a computer device. Figure 8 , Figure 8 This is a basic structural block diagram of the computer device in this embodiment.

[0140] The computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 8 having a memory 81, a processor 82, and a network interface 83, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0141] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0142] The memory 81 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 81 can be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 81 can also be an external storage device of the computer device 8, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 8. Of course, the memory 81 can also include both the internal storage unit of the computer device 8 and its external storage device. In this embodiment, the memory 81 is generally used to store the operating system and various application software installed on the computer device 8, such as computer-readable instructions of the image semantic extraction method, etc. In addition, the memory 81 can also be used to temporarily store various types of data that have been output or are to be output.

[0143] The processor 82 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 82 is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 82 is used to run computer-readable instructions stored in the memory 81 or process data, such as computer-readable instructions for running the image semantic extraction method.

[0144] The network interface 83 may include a wireless network interface or a wired network interface. The network interface 83 is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0145] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the image semantic extraction method as described above.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0147] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

[0148] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. A method for extracting image semantics, characterized in that: The steps include: Acquire a pair of sample images and description texts, and preprocess the sample images and description texts to obtain input images and input texts; Extracting features from the input image and the input text respectively to obtain image features and text features; Performing feature fusion on the text features and the image features to obtain fused features; The fused features are input into an initial diffusion model, and in a forward diffusion process of the initial diffusion model, text features in the fused features are subjected to noise processing to obtain noisy text features; In the reverse diffusion process of the initial diffusion model, noise prediction is performed on the noisy text features to obtain a predicted noise text; According to a preset semantic loss function, the semantic loss of the noisy text feature and the predicted noisy text is calculated, and the model parameters of the initial diffusion model are iteratively optimized according to the semantic loss until the initial diffusion model meets the preset conditions, thereby obtaining an optimal diffusion model; The image features of the image to be processed are obtained, and the image features of the image to be processed are input into the diffusion model to generate semantic text, so as to obtain the semantic text of the image to be processed.

2. The image semantic extraction method according to claim 1, characterized in that: The preprocessing of the sample image and the description text to obtain the input image and the input text includes: Normalizing the sample image to obtain the input image; The description text is filled to obtain the input text.

3. The image semantic extraction method according to claim 1, characterized in that: The extracting features from the input image and the input text respectively comprises: Inputting the input image and the input text into a pre-trained CLIP model respectively, wherein the CLIP model includes an image encoder and a text encoder; Extracting image representation from the input image by using the image encoder to obtain the image features; The text encoder extracts text representation from the input image to obtain the text feature.

4. The image semantic extraction method according to claim 1, characterized in that: The step of fusing the text features and the image features to obtain fused features includes: Calling a preset multimodal feature fusion network, wherein the multimodal feature fusion network includes an attention mechanism module and a fully connected module; Inputting the text features and the image features into the multimodal feature fusion network, and extracting attention features of different dimensions from the text features and the image features based on the attention mechanism module, to obtain text attention features and image attention features of multiple dimensions; The text attention features and image attention features of multiple dimensions are input into the fully connected module to obtain the fusion features.

5. The image semantic extraction method according to claim 4, characterized in that: The step of inputting the text attention features and the image attention features of multiple dimensions into the fully connected module to obtain the fusion features includes: The text attention features and image attention features of multiple dimensions are concatenated at a shallow layer to obtain shallow fusion features; The text attention features and image attention features of multiple dimensions interact at a deep level to obtain deep fusion features; The shallow fusion features and the deep fusion features are concatenated to obtain the fusion features.

6. The image semantic extraction method according to claim 5, characterized in that: The fully connected module includes an adaptive feature alignment unit and a fully connected unit, and the shallow fusion feature and the deep fusion feature are spliced ​​to obtain the fusion feature, including: The shallow fusion features and the deep fusion features are respectively input into the adaptive feature alignment unit for feature alignment, and the weights of different modal features are dynamically adjusted to obtain shallow alignment features and deep alignment features; The shallow alignment features and the deep alignment features are input into the fully connected unit to obtain the fusion features.

7. The image semantic extraction method according to claim 6, characterized in that: The fully connected module further includes a feature selection unit, which inputs the shallow fusion features and the deep fusion features into the adaptive feature alignment unit for feature alignment, dynamically adjusts the weights of different modal features, and obtains the shallow alignment features and the deep alignment features, including: The shallow fusion features and the deep fusion features are input into the feature selection unit, and the most relevant feature subset is screened based on the gating mechanism preset in the feature selection unit to obtain the optimal shallow fusion features and the deep fusion features.

8. An image semantic extraction device, characterized in that: include: A preprocessing module, used to obtain a pair of sample images and description texts, and preprocess the sample images and description texts to obtain input images and input texts; A feature extraction module, used to extract features from the input image and input text respectively to obtain image features and text features; A feature fusion module, used for fusing the text features and the image features to obtain fused features; A prediction module, used for inputting the fused features into an initial diffusion model, and performing noise processing on the text features in the fused features during the forward diffusion process of the initial diffusion model to obtain a noisy text feature; In the reverse diffusion process of the initial diffusion model, noise prediction is performed on the noisy text features to obtain a predicted noise text; A parameter optimization module, used for calculating the semantic loss of the noisy text features and the predicted noisy text according to a preset semantic loss function, and iteratively optimizing the model parameters of the initial diffusion model according to the semantic loss until the initial diffusion model meets the preset conditions, thereby obtaining an optimal diffusion model; The text generation module is used to obtain image features of the image to be processed, and input the image features of the image to be processed into the diffusion model for semantic generation to obtain the semantic text of the image to be processed.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the image semantic extraction method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the image semantic extraction method according to any one of claims 1 to 7 are implemented.

Images