Visual question answering method, device, equipment and storage medium
By constructing a standard reference dataset and a multimodal feature fusion network, the problem of failing to effectively utilize similar sample information in visual question answering systems was solved, thereby improving the accuracy and precision of prediction results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2022-09-05
- Publication Date
- 2026-06-02
Smart Images
Figure CN115346095B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a visual question-answering method, apparatus, electronic device, and readable storage medium. Background Technology
[0002] Visual Question Answering (VQA) is a multimodal learning task involving computer vision and natural language processing. A VQA system takes an image and a question as input, extracts features from the image and the question text, as well as the relationships between them, and outputs a reasonable predicted answer.
[0003] Intelligent visual question answering systems need to be able to extract features from both image and question text modalities, as well as their relationships, in order to output relatively accurate answers. Current technologies primarily train the model on a collected visual question answering dataset and then use the model with the highest accuracy on the validation set for prediction. However, this prediction process does not reuse reference information from similar samples in the dataset, resulting in low prediction accuracy. Summary of the Invention
[0004] This invention provides a visual question answering method, apparatus, electronic device, and readable storage medium, the main purpose of which is to improve the accuracy of prediction results in visual question answering.
[0005] To achieve the above objectives, the present invention provides a visual question-answering method, comprising:
[0006] Obtain an original reference data set containing reference images and reference questions and answers, extract reference image features from the reference images, and construct a standard reference data set based on the reference image features and the reference questions and answers;
[0007] Obtain visual data to be predicted, which includes the image to be predicted and the question to be predicted; extract sample image features of the image to be predicted; and match the target reference question and answer in the standard reference data set based on the sample image features.
[0008] A pre-constructed multimodal feature fusion network is used to fuse the features of the question to be predicted, the features of the sample image, and the target reference question answer to obtain fused features;
[0009] Feature prediction is performed on the fused features to obtain the prediction results.
[0010] Optionally, the step of extracting reference image features from the reference image and constructing a standard reference data set based on the reference image features and the reference question-and-answer includes:
[0011] The reference image features of the reference images in the original reference data set are extracted using a preset image encoder, and the reference image features are associated with the reference questions and answers corresponding to the reference images;
[0012] The standard reference data set is constructed by summarizing all associated reference image features and the reference questions and answers.
[0013] Optionally, the step of extracting sample image features from the image to be predicted and matching the target reference question and answer in the standard reference dataset based on the sample image features includes:
[0014] The image encoder is used to extract sample image features of the image to be predicted;
[0015] Calculate the similarity between the features of the sample image and the features of the reference images in the standard reference data set one by one;
[0016] The reference image features with a similarity greater than a preset similarity threshold are identified as the matched target image features, and the reference questions and answers corresponding to the target image features are taken as the target reference questions and answers.
[0017] Optionally, the similarity between the sample image features and the reference image features in the standard reference dataset can be calculated using the following formula:
[0018]
[0019] Where cos(A,B) represents the similarity between feature A of the sample image and feature B of the reference image, n represents the feature dimension, and a i Let b represent the i-th dimension feature vector in feature A of the sample image. i Let represent the i-th dimensional feature vector in feature B of the sample image.
[0020] Optionally, before fusing features of the question to be predicted, the sample image features, and the target reference question answer using a pre-constructed multimodal feature fusion network, the method further includes:
[0021] By concatenating the preset first self-attention module, the preset first cross-attention module, and the preset first forward propagation module, an image processing sub-network is obtained.
[0022] By concatenating the preset second self-attention module, the preset second cross-attention module, and the preset second forward propagation module, a text processing sub-network is obtained.
[0023] The first self-attention module and the second cross-attention module are connected in series, and the second self-attention module and the first cross-attention module are connected in series.
[0024] The image processing subnetwork and text processing subnetwork after concatenation are used as modal fusion subnetworks. The modal fusion subnetworks are stacked a preset number of times to obtain the multimodal feature fusion network.
[0025] Optionally, the step of using a pre-constructed multimodal feature fusion network to fuse the features of the question to be predicted, the sample image features, and the target reference question answer to obtain fused features includes:
[0026] The question to be predicted and the target reference question and answer are respectively processed by word segmentation and vectorization to obtain vectorized information;
[0027] The text features in the vectorized information are extracted using a preset text encoder;
[0028] The multimodal feature fusion network is used to perform feature interaction between the text features and the sample image features to obtain standard text features and standard image features;
[0029] The standard text features and the standard image features are fused according to the preset channel dimensions to obtain the fused features.
[0030] Optionally, the step of performing feature prediction on the fused features to obtain a prediction result includes:
[0031] The fused features are used to predict the predicted labels by using a pre-defined number of fully connected layers.
[0032] The predicted answer corresponding to the predicted label is taken as the prediction result.
[0033] To address the above problems, the present invention also provides a visual question-answering device, the device comprising:
[0034] The reference data construction module is used to acquire an original reference data set containing reference images and reference questions and answers, extract reference image features from the reference images, and construct a standard reference data set based on the reference image features and the reference questions and answers.
[0035] The feature matching module is used to acquire visual data to be predicted, which includes the image to be predicted and the question to be predicted, extract sample image features of the image to be predicted, and match the target reference question and answer in the standard reference data set based on the sample image features.
[0036] The feature fusion module is used to fuse the features of the question to be predicted, the sample image features, and the target reference question answer using a pre-built multimodal feature fusion network to obtain fused features;
[0037] The feature prediction module is used to perform feature prediction on the fused features to obtain prediction results.
[0038] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:
[0039] Memory, storing at least one computer program; and
[0040] The processor executes the computer program stored in the memory to implement the visual question-answering method described above.
[0041] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the visual question-answering method described above.
[0042] This invention extracts reference image features from a reference image, constructs a standard reference dataset based on these features and reference question-and-answer formats, and thus fully utilizes information from similar samples within the standard reference dataset during the prediction process. Furthermore, a pre-constructed multimodal feature fusion network integrates the target reference question-and-answer features into the prediction process, improving prediction accuracy and precision. Moreover, by introducing a multimodal feature fusion network, the interaction between textual and image features is strengthened, thereby fully utilizing both modalities for prediction and further enhancing accuracy and precision. Therefore, the visual question-answering method, apparatus, electronic device, and computer-readable storage medium proposed in this invention can improve the accuracy of prediction results in visual question-answering. Attached Figure Description
[0043] Figure 1 This is a flowchart illustrating a visual question-answering method provided in an embodiment of the present invention;
[0044] Figure 2 This is a functional block diagram of a visual question-answering device provided in an embodiment of the present invention;
[0045] Figure 3 This is a schematic diagram of the structure of an electronic device that implements the visual question answering method according to an embodiment of the present invention.
[0046] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0047] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0048] This invention provides a visual question-answering method. The executing entity of the visual question-answering method includes, but is not limited to, at least one of the following: a server, a terminal, or other electronic devices configured to execute the method provided in this invention. In other words, the visual question-answering method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0049] Reference Figure 1 The diagram shown is a flowchart of a visual question-answering method provided in an embodiment of the present invention.
[0050] In this embodiment, the visual question-answering method includes the following steps S1-S4:
[0051] S1. Obtain the original reference data set containing reference images and reference questions and answers, extract the reference image features of the reference images, and construct a standard reference data set based on the reference image features and the reference questions and answers.
[0052] In this embodiment of the invention, the original reference data set is reference data used for assisting in visual question answering. For example, in the financial field, the original reference data set includes product images of different funds, insurance products, and the corresponding questions and answers, such as the product image corresponding to product XX1 and the questions and answers (product introduction, uses, etc.) corresponding to product XX1.
[0053] Specifically, the step of extracting reference image features from the reference image and constructing a standard reference data set based on the reference image features and the reference question-and-answer includes:
[0054] The reference image features of the reference images in the original reference data set are extracted using a preset image encoder, and the reference image features are associated with the reference questions and answers corresponding to the reference images;
[0055] The standard reference data set is constructed by summarizing all associated reference image features and the reference questions and answers.
[0056] In an optional embodiment of the present invention, the image encoder can be a deep learning network, such as R-CNN, FastRCNN, FasterRCNN, YOLO, SSD, etc. Simultaneously, a key-value pair method is used to associate reference image features with corresponding reference questions and answers; for example, the key value is the image feature, and the value value is the reference question and answer.
[0057] S2. Obtain visual data to be predicted, which includes the image to be predicted and the question to be predicted. Extract sample image features from the image to be predicted. Match the target reference question and answer in the standard reference data set based on the sample image features.
[0058] In this embodiment of the invention, the visual data to be predicted refers to the data to be queried input by the user under visual question and answer. For example, in the financial field, a user inputs a product image of a certain product on an intelligent customer service page, and inputs a question about the product image.
[0059] Specifically, the step of extracting sample image features of the image to be predicted and matching the target reference question and answer in the standard reference dataset based on the sample image features includes:
[0060] The image encoder is used to extract sample image features of the image to be predicted;
[0061] Calculate the similarity between the features of the sample image and the features of the reference images in the standard reference data set one by one;
[0062] The reference image features with a similarity greater than a preset similarity threshold are identified as the matched target image features, and the reference questions and answers corresponding to the target image features are taken as the target reference questions and answers.
[0063] In an optional embodiment of the present invention, feature extraction is first performed on the image to be predicted during the visual question answering prediction process. The extracted features are then used to search in a standard reference data set, that is, the features of the sample image are compared with the features of other reference images in the dataset, the similarity is calculated, and the question and answer text corresponding to the reference image with a similarity greater than a preset similarity threshold (0.7) is found as the target reference question and answer.
[0064] In an optional embodiment of the present invention, the similarity between the features of the sample image and the features of the reference image in the standard reference dataset is calculated using the following formula:
[0065]
[0066] Where cos(A,B) represents the similarity between feature A of the sample image and feature B of the reference image, n represents the feature dimension, and a iLet b represent the i-th dimension feature vector in feature A of the sample image. i Let represent the i-th dimensional feature vector in feature B of the sample image.
[0067] In this embodiment of the invention, by matching the target reference question and answer in the standard reference data set with the features of the sample image, the similar data in the standard reference data set can be fully utilized, thereby improving the accuracy of visual question answering.
[0068] S3. Use a pre-constructed multimodal feature fusion network to fuse the features of the question to be predicted, the sample image features, and the target reference question answer to obtain fused features.
[0069] In this embodiment of the invention, the pre-constructed multimodal feature fusion network is used to fuse image features and text features, thereby better uncovering the hidden relationship between image features and text features.
[0070] Specifically, before performing feature fusion on the question to be predicted, the sample image features, and the target reference question answer using a pre-constructed multimodal feature fusion network, the method further includes:
[0071] By concatenating the preset first self-attention module, the preset first cross-attention module, and the preset first forward propagation module, an image processing sub-network is obtained.
[0072] By concatenating the preset second self-attention module, the preset second cross-attention module, and the preset second forward propagation module, a text processing sub-network is obtained.
[0073] The first self-attention module and the second cross-attention module are connected in series, and the second self-attention module and the first cross-attention module are connected in series.
[0074] The image processing subnetwork and text processing subnetwork after concatenation are used as modal fusion subnetworks. The modal fusion subnetworks are stacked a preset number of times to obtain the multimodal feature fusion network.
[0075] In an optional embodiment of the present invention, image features and text features first pass through a self-attention module, then a cross-attention mechanism is used to realize the interaction between the two different modal features of image and text, and finally a forward propagation module is used for output. The above modal fusion sub-networks are stacked M times (M=6) to form the final multimodal feature fusion network, thereby better performing feature interaction and fusion.
[0076] In detail, the method of using a pre-constructed multimodal feature fusion network to fuse the features of the question to be predicted, the sample image features, and the target reference question answer to obtain fused features includes:
[0077] The question to be predicted and the target reference question and answer are respectively processed by word segmentation and vectorization to obtain vectorized information;
[0078] The text features in the vectorized information are extracted using a preset text encoder;
[0079] The multimodal feature fusion network is used to perform feature interaction between the text features and the sample image features to obtain standard text features and standard image features;
[0080] The standard text features and the standard image features are fused according to the preset channel dimensions to obtain the fused features.
[0081] In an optional embodiment of the present invention, the preset text encoder can be a BERT model. The target reference question and the question to be predicted are segmented into words, and then word embeddings are performed on the words in the text, mapping them into a word vector form that is easily processed by computers. That is, each word is represented by a 512-dimensional word vector to obtain the word embeddings of the question text. Then, the text encoder is used to extract features to obtain text features. A multimodal feature fusion network is then used to perform feature interaction between the text features and the sample image features. The two features output by the multimodal feature fusion network are concatenated along the channel dimension to obtain the fused features.
[0082] In this embodiment of the invention, by introducing a multimodal feature fusion network, the interaction between text features and image features, two different modal information, is enhanced, thereby making full use of the two modal information for prediction and improving prediction accuracy and precision.
[0083] S4. Perform feature prediction on the fused features to obtain the prediction result.
[0084] In detail, the step of performing feature prediction on the fused features to obtain the prediction result includes:
[0085] The fused features are used to predict the predicted labels by using a pre-defined number of fully connected layers.
[0086] The predicted answer corresponding to the predicted label is taken as the prediction result.
[0087] In an optional embodiment of the present invention, the fused features can be predicted and output through two fully connected layers to obtain the final predicted label. The predicted answer text corresponding to the predicted label is taken as the final prediction result. For example, if the predicted label is "1", then the answer text corresponding to "1" is taken as the prediction result.
[0088] This invention extracts reference image features from a reference image, constructs a standard reference dataset based on these features and reference question-and-answer formats, and thus fully utilizes information from similar samples within the standard reference dataset during the prediction process. Furthermore, a pre-constructed multimodal feature fusion network integrates the target reference question-and-answer features into the prediction process, improving prediction accuracy and precision. Moreover, by introducing a multimodal feature fusion network, the interaction between textual and image features is strengthened, allowing for full utilization of both modalities and further enhancing prediction accuracy and precision. Therefore, the visual question-and-answer method proposed in this invention can improve the accuracy of prediction results in visual question-and-answer.
[0089] like Figure 2 The diagram shown is a functional block diagram of a visual question-answering device provided in an embodiment of the present invention.
[0090] The visual question-answering device 100 of this invention can be installed in an electronic device. Depending on the functions implemented, the visual question-answering device 100 may include a reference data construction module 101, a feature matching module 102, a feature fusion module 103, and a feature prediction module 104. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.
[0091] In this embodiment, the functions of each module / unit are as follows:
[0092] The reference data construction module 101 is used to obtain an original reference data set containing reference images and reference questions and answers, extract reference image features of the reference images, and construct a standard reference data set based on the reference image features and the reference questions and answers;
[0093] The feature matching module 102 is used to acquire visual data to be predicted, which includes the image to be predicted and the question to be predicted, extract sample image features of the image to be predicted, and match the target reference question and answer in the standard reference data set based on the sample image features.
[0094] The feature fusion module 103 is used to perform feature fusion on the question to be predicted, the sample image features, and the target reference question and answer using a pre-constructed multimodal feature fusion network to obtain fused features;
[0095] The feature prediction module 104 is used to perform feature prediction on the fused features to obtain prediction results.
[0096] The specific implementation methods of each module of the visual question-answering device 100 are as follows:
[0097] Step 1: Obtain the original reference data set containing reference images and reference questions and answers, extract the reference image features of the reference images, and construct a standard reference data set based on the reference image features and the reference questions and answers.
[0098] In this embodiment of the invention, the original reference data set is reference data used for assisting in visual question answering. For example, in the financial field, the original reference data set includes product images of different funds, insurance products, and the corresponding questions and answers, such as the product image corresponding to product XX1 and the questions and answers (product introduction, uses, etc.) corresponding to product XX1.
[0099] Specifically, the step of extracting reference image features from the reference image and constructing a standard reference data set based on the reference image features and the reference question-and-answer includes:
[0100] The reference image features of the reference images in the original reference data set are extracted using a preset image encoder, and the reference image features are associated with the reference questions and answers corresponding to the reference images;
[0101] The standard reference data set is constructed by summarizing all associated reference image features and the reference questions and answers.
[0102] In an optional embodiment of the present invention, the image encoder can be a deep learning network, such as R-CNN, FastRCNN, FasterRCNN, YOLO, SSD, etc. Simultaneously, a key-value pair method is used to associate reference image features with corresponding reference questions and answers; for example, the key value is the image feature, and the value value is the reference question and answer.
[0103] Step 2: Obtain visual data to be predicted, which includes the image to be predicted and the question to be predicted; extract sample image features of the image to be predicted; and match the target reference question and answer in the standard reference data set based on the sample image features.
[0104] In this embodiment of the invention, the visual data to be predicted refers to the data to be queried input by the user under visual question and answer. For example, in the financial field, a user inputs a product image of a certain product on an intelligent customer service page, and inputs a question about the product image.
[0105] Specifically, the step of extracting sample image features of the image to be predicted and matching the target reference question and answer in the standard reference dataset based on the sample image features includes:
[0106] The image encoder is used to extract sample image features of the image to be predicted;
[0107] Calculate the similarity between the features of the sample image and the features of the reference images in the standard reference data set one by one;
[0108] The reference image features with a similarity greater than a preset similarity threshold are identified as the matched target image features, and the reference questions and answers corresponding to the target image features are taken as the target reference questions and answers.
[0109] In an optional embodiment of the present invention, feature extraction is first performed on the image to be predicted during the visual question answering prediction process. The extracted features are then used to search in a standard reference data set, that is, the features of the sample image are compared with the features of other reference images in the dataset, the similarity is calculated, and the question and answer text corresponding to the reference image with a similarity greater than a preset similarity threshold (0.7) is found as the target reference question and answer.
[0110] In an optional embodiment of the present invention, the similarity between the features of the sample image and the features of the reference image in the standard reference dataset is calculated using the following formula:
[0111]
[0112] Where cos(a,B) represents the similarity between feature A of the sample image and feature B of the reference image, n represents the feature dimension, and a i Let b represent the i-th dimension feature vector in feature A of the sample image. i Let represent the i-th dimensional feature vector in feature B of the sample image.
[0113] In this embodiment of the invention, by matching the target reference question and answer in the standard reference data set with the features of the sample image, the similar data in the standard reference data set can be fully utilized, thereby improving the accuracy of visual question answering.
[0114] Step 3: Use a pre-constructed multimodal feature fusion network to fuse the features of the question to be predicted, the sample image features, and the target reference question answer to obtain fused features.
[0115] In this embodiment of the invention, the pre-constructed multimodal feature fusion network is used to fuse image features and text features, thereby better uncovering the hidden relationship between image features and text features.
[0116] Specifically, before performing feature fusion on the question to be predicted, the sample image features, and the target reference question answer using a pre-constructed multimodal feature fusion network, the method further includes:
[0117] By concatenating the preset first self-attention module, the preset first cross-attention module, and the preset first forward propagation module, an image processing sub-network is obtained.
[0118] By concatenating the preset second self-attention module, the preset second cross-attention module, and the preset second forward propagation module, a text processing sub-network is obtained.
[0119] The first self-attention module and the second cross-attention module are connected in series, and the second self-attention module and the first cross-attention module are connected in series.
[0120] The image processing subnetwork and text processing subnetwork after concatenation are used as modality fusion subnetworks. The modality fusion subnetworks are stacked a preset number of times to obtain the multimodal feature fusion network.
[0121] In an optional embodiment of the present invention, image features and text features first pass through a self-attention module, then a cross-attention mechanism is used to realize the interaction between the two different modal features of image and text, and finally a forward propagation module is used for output. The above modal fusion sub-networks are stacked M times (M=6) to form the final multimodal feature fusion network, thereby better performing feature interaction and fusion.
[0122] In detail, the method of using a pre-constructed multimodal feature fusion network to fuse the features of the question to be predicted, the sample image features, and the target reference question answer to obtain fused features includes:
[0123] The question to be predicted and the target reference question and answer are respectively processed by word segmentation and vectorization to obtain vectorized information;
[0124] The text features in the vectorized information are extracted using a preset text encoder;
[0125] The multimodal feature fusion network is used to perform feature interaction between the text features and the sample image features to obtain standard text features and standard image features;
[0126] The standard text features and the standard image features are fused according to the preset channel dimensions to obtain the fused features.
[0127] In an optional embodiment of the present invention, the preset text encoder can be a BERT model. The target reference question and the question to be predicted are segmented into words, and then word embeddings are performed on the words in the text, mapping them into a word vector form that is easily processed by computers. That is, each word is represented by a 512-dimensional word vector to obtain the word embeddings of the question text. Then, the text encoder is used to extract features to obtain text features. A multimodal feature fusion network is then used to perform feature interaction between the text features and the sample image features. The two features output by the multimodal feature fusion network are concatenated along the channel dimension to obtain the fused features.
[0128] In this embodiment of the invention, by introducing a multimodal feature fusion network, the interaction between text features and image features, two different modal information, is enhanced, thereby making full use of the two modal information for prediction and improving prediction accuracy and precision.
[0129] Step 4: Perform feature prediction on the fused features to obtain the prediction results.
[0130] In detail, the step of performing feature prediction on the fused features to obtain the prediction result includes:
[0131] The fused features are used to predict the predicted labels by using a pre-defined number of fully connected layers.
[0132] The predicted answer corresponding to the predicted label is taken as the prediction result.
[0133] In an optional embodiment of the present invention, the fused features can be predicted and output through two fully connected layers to obtain the final predicted label. The predicted answer text corresponding to the predicted label is taken as the final prediction result. For example, if the predicted label is "1", then the answer text corresponding to "1" is taken as the prediction result.
[0134] This invention extracts reference image features from a reference image, constructs a standard reference dataset based on these features and reference question-and-answer formats, and thus fully utilizes information from similar samples within the standard reference dataset during the prediction process. Furthermore, a pre-constructed multimodal feature fusion network integrates the target reference question-and-answer features into the prediction process, improving prediction accuracy and precision. Moreover, by introducing the multimodal feature fusion network, the interaction between textual and image features is strengthened, allowing for full utilization of both modalities and further enhancing prediction accuracy and precision. Therefore, the visual question-and-answer device proposed in this invention can improve the accuracy of prediction results in visual question-and-answer.
[0135] like Figure 3 The diagram shown is a structural schematic of an electronic device that implements the visual question-answering method according to an embodiment of the present invention.
[0136] The electronic device may include a processor 10, a memory 11, a communication interface 12 and a bus 13, and may also include a computer program, such as a visual question-answering program, stored in the memory 11 and capable of running on the processor 10.
[0137] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of a visual question-and-answer program, but also to temporarily store data that has been output or will be output.
[0138] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules (e.g., visual question-and-answer programs) stored in the memory 11 and calls data stored in the memory 11 to perform various functions of the electronic device and process data.
[0139] The communication interface 12 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.
[0140] The bus 13 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 13 can be divided into an address bus, a data bus, a control bus, etc. The bus 13 is configured to enable communication between the memory 11 and at least one processor 10, etc.
[0141] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3 The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.
[0142] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.
[0143] Furthermore, the electronic device may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device and other electronic devices.
[0144] Optionally, the electronic device may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.
[0145] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.
[0146] The visual question-answering program stored in the memory 11 of the electronic device is a combination of multiple instructions, which, when run in the processor 10, can achieve the following:
[0147] Obtain an original reference data set containing reference images and reference questions and answers, extract reference image features from the reference images, and construct a standard reference data set based on the reference image features and the reference questions and answers;
[0148] Obtain visual data to be predicted, which includes the image to be predicted and the question to be predicted; extract sample image features of the image to be predicted; and match the target reference question and answer in the standard reference data set based on the sample image features.
[0149] A pre-constructed multimodal feature fusion network is used to fuse the features of the question to be predicted, the features of the sample image, and the target reference question answer to obtain fused features;
[0150] Feature prediction is performed on the fused features to obtain the prediction results.
[0151] Specifically, the specific implementation method of the processor 10 for the above instructions can be referred to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, and will not be repeated here.
[0152] Furthermore, if the modules / units integrated into the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).
[0153] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:
[0154] Obtain an original reference data set containing reference images and reference questions and answers, extract reference image features from the reference images, and construct a standard reference data set based on the reference image features and the reference questions and answers;
[0155] Obtain visual data to be predicted, which includes the image to be predicted and the question to be predicted; extract sample image features of the image to be predicted; and match the target reference question and answer in the standard reference data set based on the sample image features.
[0156] A pre-constructed multimodal feature fusion network is used to fuse the features of the question to be predicted, the features of the sample image, and the target reference question answer to obtain fused features;
[0157] Feature prediction is performed on the fused features to obtain the prediction results.
[0158] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.
[0159] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0160] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.
[0161] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.
[0162] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.
[0163] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.
[0164] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.
[0165] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.
[0166] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.
[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims
1. A visual question-answering method, characterized in that, The method includes: Obtain an original reference data set containing reference images and reference questions and answers, extract reference image features from the reference images, and construct a standard reference data set based on the reference image features and the reference questions and answers, wherein the reference questions and answers are the questions and answers corresponding to the reference images; Obtain visual data to be predicted, which includes the image to be predicted and the question to be predicted. Use an image encoder to extract sample image features of the image to be predicted. Calculate the similarity between the sample image features and the reference image features in the standard reference data set one by one. Determine all reference image features with a similarity greater than a preset similarity threshold as target image features. Use the reference question and answer corresponding to the target image features as target reference question and answer. The text features of the question to be predicted, the sample image features, and the text features of the target reference question answer are fused using a pre-constructed multimodal feature fusion network to obtain fused features; Feature prediction is performed on the fused features to obtain the prediction results.
2. The visual question-answering method as described in claim 1, characterized in that, The step of extracting reference image features from the reference image and constructing a standard reference data set based on the reference image features and the reference question-and-answer includes: The reference image features of the reference images in the original reference data set are extracted using a preset image encoder, and the reference image features are associated with the reference questions and answers corresponding to the reference images; The standard reference data set is constructed by summarizing all associated reference image features and the reference questions and answers.
3. The visual question-answering method as described in claim 1, characterized in that, The similarity between the features of the sample image and the features of the reference image in the standard reference dataset is calculated using the following formula: in, Representing sample image features Features of the reference image similarity, Representing feature dimension, Representing sample image features The first in 3D feature vectors Representing sample image features The first in 3D eigenvectors.
4. The visual question-answering method as described in claim 1, characterized in that, Before fusing the text features of the question to be predicted, the sample image features, and the text features of the target reference question answer using a pre-constructed multimodal feature fusion network, the method further includes: By concatenating the preset first self-attention module, the preset first cross-attention module, and the preset first forward propagation module, an image processing sub-network is obtained. By concatenating the preset second self-attention module, the preset second cross-attention module, and the preset second forward propagation module, a text processing sub-network is obtained. The first self-attention module and the second cross-attention module are connected in series, and the second self-attention module and the first cross-attention module are connected in series. The image processing subnetwork and text processing subnetwork after concatenation are used as modality fusion subnetworks. The modality fusion subnetworks are stacked a preset number of times to obtain the multimodal feature fusion network.
5. The visual question-answering method as described in claim 1, characterized in that, The method utilizes a pre-constructed multimodal feature fusion network to fuse the text features of the question to be predicted, the sample image features, and the text features of the target reference question answer to obtain fused features, including: The question to be predicted and the target reference question and answer are respectively processed by word segmentation and vectorization to obtain vectorized information; The text features in the vectorized information are extracted using a preset text encoder; The multimodal feature fusion network is used to perform feature interaction between the text features and the sample image features to obtain standard text features and standard image features; The standard text features and the standard image features are fused according to the preset channel dimensions to obtain the fused features.
6. The visual question-answering method as described in claim 1, characterized in that, The step of performing feature prediction on the fused features to obtain prediction results includes: The fused features are used to predict the predicted labels by using a pre-defined number of fully connected layers. The predicted answer corresponding to the predicted label is taken as the prediction result.
7. A visual question-answering device, characterized in that, The device includes: The reference data construction module is used to obtain an original reference data set containing reference images and reference questions and answers, extract reference image features from the reference images, and construct a standard reference data set based on the reference image features and the reference questions and answers, wherein the reference questions and answers are the questions and answers corresponding to the reference images; The feature matching module is used to acquire visual data to be predicted, which includes the image to be predicted and the question to be predicted. It uses an image encoder to extract sample image features of the image to be predicted, calculates the similarity between the sample image features and the reference image features in the standard reference data set one by one, determines all reference image features with a similarity greater than a preset similarity threshold as target image features, and uses the reference question and answer corresponding to the target image features as target reference question and answer. The feature fusion module is used to fuse the text features of the question to be predicted, the sample image features, and the text features of the target reference question answer using a pre-built multimodal feature fusion network to obtain fused features; The feature prediction module is used to perform feature prediction on the fused features to obtain prediction results.
8. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the visual question answering method as described in any one of claims 1 to 6.
9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the visual question answering method as described in any one of claims 1 to 6.