Visual question and answer processing method, device, medium and system

Abstract

The invention provides a visual question and answer processing method and device, a medium and a system. The method comprises the following steps: performing distillation processing on configuration information and parameters of a first CLIP model by adopting a neural network model to obtain distillation parameters; training a second CLIP model by adopting the distillation parameters to obtain a trained second CLIP model; and optimizing the output of the neural network model by using the first CLIP model and the trained second CLIP model to obtain an optimal distillation parameter, and optimizing the trained second CLIP model by using the optimal distillation parameter to obtain an optimal second CLIP model. According to the technical scheme, the deference delay is reduced, the processing speed of the visual question and answer task is improved, and the problem that the processing speed is low due to the fact that a network structure for processing the visual question and answer task in an existing scheme is complex is solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision and artificial intelligence technology, and more specifically, to a method, device, medium and system for processing visual question answering. Background Art

[0002] Visual question answering tasks refer to the situation where a picture and a question about the picture are given and the answer to the question is given based on the picture. There are many types of questions, which can be multiple-choice questions or essay questions.

[0003] Most existing technologies use two network structures to extract image and text information respectively, and then interact the information of the two in some way to get the answer. Although existing technologies have achieved certain results, they require a large network structure and a lot of labeled data to train the network structure in order to achieve good results. In other words, the performance on the visual question answering task is positively correlated with the size of the model network structure and the amount of data used. The larger the network structure and the more data, the better the performance, but correspondingly, the more computing resources are used. At the same time, a larger network structure will lead to a larger inference delay when the actual application is deployed, which is not conducive to improving user experience and promoting the product.

[0004] That is, the network structure of the existing solution for processing visual question answering tasks is relatively complex, resulting in a slow processing speed. Summary of the invention

[0005] The main purpose of the present application is to provide a method, device, medium and system for processing visual question answering, so as to at least solve the problem that the network structure for processing visual question answering tasks in existing solutions is relatively complex, resulting in slow processing speed.

[0006] In order to achieve the above objective, according to one aspect of the present application, a method for processing visual question answering is provided, the method comprising:

[0007] Using a neural network model, distilling configuration information and parameters of a first CLIP model to obtain distillation parameters, wherein the configuration information of the first CLIP model includes the number of layers and the vector dimension of each layer, and the first CLIP model is used to score the degree of matching between the image and the text to obtain a matching score;

[0008] Training a second CLIP model using the distillation parameters to obtain a trained second CLIP model, wherein the number of parameters of the second CLIP model is less than the number of parameters of the first CLIP model;

[0009] The first CLIP model and the trained second CLIP model are used to optimize the output of the neural network model to obtain optimal distillation parameters, and the trained second CLIP model is optimized using the optimal distillation parameters to obtain an optimal second CLIP model.

[0010] Optionally, the output of the neural network model is optimized using the first CLIP model and the trained second CLIP model to obtain optimal distillation parameters, including: obtaining a test data set, wherein the test data set includes multiple test images and multiple test texts; using the first CLIP model to process the test data set to obtain a first PPL; using the trained second CLIP model to process the test data set to obtain a second PPL; based on the first PPL and the second PPL, optimizing the output of the neural network model to obtain the optimal distillation parameters.

[0011] Optionally, based on the first PPL and the second PPL, the output of the neural network model is optimized to obtain the optimal distillation parameters, including: determining a loss value as a difference between the second PPL and the first PPL; when the loss value is greater than or equal to a loss threshold, continuously adopting the configuration information and parameters of the first CLIP model to optimize the output of the neural network model to obtain optimized distillation parameters, until the loss value is less than the loss threshold, determining that the currently optimized distillation parameters are the optimal distillation parameters.

[0012] Optionally, after determining that the loss value is the absolute value of the difference between the first PPL and the second PPL, the method further includes: when the loss value is less than the loss threshold, determining that the optimized distillation parameter is the optimal distillation parameter.

[0013] Optionally, in the process of using a neural network model to distill the configuration information and parameters of the first CLIP model to obtain the distillation parameters, the method also includes: using a word segmenter to segment the configuration information of the first CLIP model to obtain a word segmentation result; and using an embedding layer of the neural network model to vectorize the word segmentation result to obtain a configuration information vector.

[0014] Optionally, after optimizing the trained second CLIP model by using the optimal distillation parameters to obtain an optimal second CLIP model, the method further includes: training the optimal second CLIP model by using the training set of the first CLIP model to obtain a final second CLIP model.

[0015] Optionally, after optimizing the trained second CLIP model by using the optimal distillation parameters to obtain the optimal second CLIP model, the method further includes: converting weights and activation values ​​of the optimal second CLIP model into INT8 format.

[0016] According to another aspect of the present application, a visual question answering processing device is provided, the device comprising:

[0017] a first processing unit, configured to use a neural network model to perform distillation processing on configuration information and parameters of a first CLIP model to obtain distillation parameters, wherein the configuration information of the first CLIP model includes the number of layers and the vector dimension of each layer, and the first CLIP model is used to score the degree of matching between the image and the text to obtain a matching score;

[0018] A second processing unit is used to train a second CLIP model using the distillation parameters to obtain a trained second CLIP model, wherein the number of parameters of the second CLIP model is less than the number of parameters of the first CLIP model;

[0019] The third processing unit is used to optimize the output of the neural network model by using the first CLIP model and the trained second CLIP model to obtain an optimal distillation parameter, and optimize the trained second CLIP model by using the optimal distillation parameter to obtain an optimal second CLIP model.

[0020] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0021] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the described methods.

[0022] By applying the technical solution of the present application, the configuration information and parameters of the first CLIP model are distilled by using a neural network model to obtain distillation parameters; the second CLIP model is trained by using the distillation parameters to obtain a trained second CLIP model; the output of the neural network model is optimized by using the first CLIP model and the trained second CLIP model to obtain the optimal distillation parameters, and the trained second CLIP model is optimized by using the optimal distillation parameters to obtain the optimal second CLIP model. The delay during reasoning is reduced and the processing speed of the visual question answering task is improved, that is, the problem that the network structure for processing the visual question answering task in the existing solution is relatively complex, resulting in a slow processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] Figure 1 A flowchart of a visual question answering method provided according to an embodiment of the present application is shown;

[0025] Figure 2 A flowchart of another visual question answering method provided according to an embodiment of the present application is shown;

[0026] Figure 3 A structural block diagram of a visual question answering processing device provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0031] Pre-trained model: refers to a model trained on a large-scale unlabeled dataset.

[0032] Fine-tuning: refers to training the model using a small amount of high-quality data based on the pre-trained model, so that the trained model can meet the requirements of specific tasks or specific fields.

[0033] Reinforcement learning: Reinforcement learning means that the intelligent agent adjusts its behavior strategy by interacting with the environment and obtaining feedback from the environment to obtain the maximum reward.

[0034] Low resources and low latency: Low resources means less computing resources are used during training and inference, and low latency means low latency in generating answers during inference.

[0035] Distillation learning: Distillation learning is a machine learning technique that aims to improve the performance of a student model by transferring knowledge from a complex model (called the teacher model) to another relatively simple model (called the student model).

[0036] Visual question answering task: given a picture and a question text about the image, the task goal is to accurately answer the question.

[0037] PPL: PPL is the abbreviation of Perplexity, which is one of the indicators to measure the quality of a language model. PPL is used to evaluate the degree of fit of a language model to a given set of test data. A lower PPL value means that the model predicts the test data more accurately, and the higher the fluency, the better the model fits the data; a higher PPL value means that the model has poorer predictive ability, that is, the model fits the data poorly.

[0038] Embedding layer: It is an important component in deep learning models, used to map high-dimensional sparse discrete data (such as words, categories, etc.) into low-dimensional dense continuous vector space. The purpose of this mapping is to better represent and learn the semantic relationship between data, thereby improving the performance and generalization ability of the model. In natural language processing tasks, the embedding layer is often used to map words into a vector space, where each word is represented as a vector of fixed dimension. These vectors can capture the semantic and grammatical relationships between words, thereby improving the performance of the model in tasks such as text classification, sentiment analysis, and machine translation.

[0039] CLIP model: It is a pre-trained neural network for matching images and texts released by OpenAI in early 2021. It can understand the correspondence between images and texts and has achieved SOTA results in many tasks in the field of multimodal research. It uses contrastive learning to narrow the distance between similar images and texts, and uses 400 million image-text pairs for training to achieve the alignment function of images and texts. Using a pre-trained model can greatly reduce the resources we need for training.

[0040] As introduced in the background technology, the prior art mostly uses two network structures to extract the information of the image and text respectively, and then interacts the information of the two in some way to obtain the answer. Although the prior art has achieved certain results, it requires a large network structure and a lot of labeled data to train the network structure in order to achieve good results. In other words, the performance on the visual question answering task is positively correlated with the size of the model network structure and the amount of data used. The larger the network structure and the more data, the better the performance, but correspondingly, the more computing resources are used. At the same time, a larger network structure will lead to a larger reasoning delay when the actual application deploys the service, which is not conducive to the improvement of user experience and the promotion of the product. That is, the network structure of the existing solution for processing the visual question answering task is relatively complex, resulting in a slower processing speed. In order to solve the problem that the network structure of the existing solution for processing the visual question answering task is relatively complex, resulting in a slower processing speed, the embodiments of the present application provide a method, device, medium and system for processing visual question answering.

[0041] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0042] In this embodiment, a method for processing visual question answering is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] Figure 1 is a flowchart of a method for processing visual question answering according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0044] Step S101, using a neural network model to distill configuration information and parameters of a first CLIP model to obtain distillation parameters, wherein the configuration information of the first CLIP model includes the number of layers and the vector dimension of each layer, and the first CLIP model is used to score the matching degree between the image and the text to obtain a matching score;

[0045] Specifically, the above-mentioned matching degree scoring can be implemented in the following way: the image is encoded into a vector of fixed dimension, the answer returned by the visual question answering task is also encoded into a vector of the same fixed dimension, and then the cosine distance is used for scoring.

[0046] In one embodiment of the present application, in step S101, in which a neural network model is used to distill the configuration information and parameters of the first CLIP model to obtain the distillation parameters, the method further includes: using a word segmenter to segment the configuration information of the first CLIP model to obtain a word segmentation result; and using the embedding layer of the neural network model to vectorize the word segmentation result to obtain a configuration information vector.

[0047] Specifically, through the above-mentioned vectorization processing, the semantic and grammatical relationships between words can be captured, thereby improving the performance of the model in tasks such as text classification, sentiment analysis, and machine translation.

[0048] Step S102, training a second CLIP model using the distillation parameters to obtain a trained second CLIP model, wherein the number of parameters of the second CLIP model is less than the number of parameters of the first CLIP model;

[0049] Specifically, in order to reduce the computing resources required for training, the parameters of the first CLIP model can be directly used to generate the parameters of the second CLIP model. As a pre-trained model, the CLIP model has been learned on a lot of data and has a lot of knowledge. Therefore, directly migrating its parameters to the second CLIP model can allow the second CLIP model to converge with fewer resources in subsequent training. The above-mentioned large amount of data is specifically 400 million pairs of images and texts.

[0050] Step S103, using the first CLIP model and the trained second CLIP model to optimize the output of the neural network model to obtain optimal distillation parameters, and using the optimal distillation parameters to optimize the trained second CLIP model to obtain an optimal second CLIP model.

[0051] Specifically, the above-mentioned optimal distillation parameters are used to optimize the second CLIP model after training, ensuring that the loss of effect during the distillation process is acceptable, while achieving a reduction in the number of model parameters and a reduction in the delay during inference.

[0052] In the above steps, the configuration information and parameters of the first CLIP model are distilled by using a neural network model to obtain distillation parameters; the second CLIP model is trained by using the distillation parameters to obtain a trained second CLIP model; the output of the neural network model is optimized by using the first CLIP model and the trained second CLIP model to obtain the optimal distillation parameters, and the trained second CLIP model is optimized by using the optimal distillation parameters to obtain the optimal second CLIP model. The delay during inference is reduced, and the processing speed of the visual question answering task is improved, that is, the problem that the network structure for processing the visual question answering task in the existing solution is relatively complex, resulting in a slow processing speed.

[0053] In one embodiment of the present application, the step S103 uses the first CLIP model and the trained second CLIP model to optimize the output of the neural network model to obtain the optimal distillation parameters, including: obtaining a test data set, wherein the test data set includes multiple test images and multiple test texts; using the first CLIP model to process the test data set to obtain a first PPL; using the trained second CLIP model to process the test data set to obtain a second PPL; based on the first PPL and the second PPL, optimizing the output of the neural network model to obtain the optimal distillation parameters.

[0054] Specifically, the first PPL is used to measure the effect of the first CLIP model on the visual question answering task; the second PPL is used to measure the effect of the trained second CLIP model on the visual question answering task. These two indicators can be used to concretely represent the gap between the trained second CLIP model and the first CLIP model in visual task processing, and can be used to assess whether the subsequent optimization results are complete.

[0055] In one embodiment of the present application, in step S103, based on the first PPL and the second PPL, the output of the neural network model is optimized to obtain the optimal distillation parameters, including: determining a loss value as a difference between the second PPL and the first PPL; when the loss value is greater than or equal to a loss threshold, continuously using the configuration information and parameters of the first CLIP model to optimize the output of the neural network model to obtain the optimized distillation parameters, until the loss value is less than the loss threshold, determining that the currently optimized distillation parameters are the optimal distillation parameters.

[0056] Specifically, after obtaining the above loss value, the gradient descent method is used to optimize the above neural network model. The gradient descent method is a commonly used optimization algorithm used to solve the problem of minimizing the objective function. Its basic idea is to gradually approach the minimum value of the objective function through continuous iteration. The core of this method is to use the gradient information of the objective function, that is, the slope of the function at the current point, to guide the next moving direction and step size. Specifically, the gradient descent method calculates the partial derivative (i.e., gradient) at the current position, and then iterates in the direction of the negative gradient to gradually approach the minimum point. If the loss function is a convex function, the solution obtained by the gradient descent method is the global optimal solution; if it is not a convex function, a local optimal solution may be obtained.

[0057] In a specific implementation process, after determining that the loss value is the absolute value of the difference between the first PPL and the second PPL, the method further includes: when the loss value is less than the loss threshold, determining that the optimized distillation parameter is the optimal distillation parameter.

[0058] Specifically, the trained second CLIP model is optimized using the optimal distillation parameters to obtain an optimal second CLIP model. Due to the optimization, the effect of the optimal second CLIP model is not much different from that of the first CLIP model.

[0059] In one embodiment of the present application, after optimizing the trained second CLIP model by using the optimal distillation parameters in step S103 to obtain the optimal second CLIP model, the method further includes: training the optimal second CLIP model by using the training set of the first CLIP model to obtain a final second CLIP model.

[0060] Specifically, the visual question answering task is to ask a question about a picture, and then the model returns an answer. The first CLIP model has a large number of parameters, and the visual question answering task effect is better than the best second CLIP model, so the best second CLIP model can be further supervised and optimized. The above answer is in text form, and the above visual question answering task effect represents the degree of match between the picture and the answer.

[0061] In one embodiment of the present application, after optimizing the trained second CLIP model using the optimal distillation parameters in step S103 to obtain the optimal second CLIP model, the method further includes: converting the weights and activation values ​​of the optimal second CLIP model into INT8 format. This can further reduce the inference delay.

[0062] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the visual question answering processing method of the present application will be described in detail below in combination with specific embodiments.

[0063] This embodiment relates to a specific visual question answering processing method, such as Figure 2 As shown, including:

[0064] Using a neural network model, distilling the configuration information and parameters of the first CLIP model to obtain distillation parameters, the configuration information of the first CLIP model includes the number of layers and the vector dimension of each layer, and the first CLIP model is used to score the matching degree between the image and the text to obtain a matching score;

[0065] The second CLIP model is trained using the distillation parameters to obtain a trained second CLIP model, wherein the number of parameters of the second CLIP model is less than the number of parameters of the first CLIP model;

[0066] The first CLIP model and the trained second CLIP model are used to optimize the output of the neural network model to obtain an optimal distillation parameter, and the trained second CLIP model is optimized using the optimal distillation parameter to obtain an optimal second CLIP model;

[0067] The training set of the first CLIP model is used to train the optimal second CLIP model to obtain the final second CLIP model;

[0068] Convert the weights and activations of the best second CLIP model to INT8 format.

[0069] By using a neural network model, the configuration information and parameters of the first CLIP model are distilled to obtain distillation parameters; the distillation parameters are used to train the second CLIP model to obtain the trained second CLIP model; the output of the neural network model is optimized by using the first CLIP model and the trained second CLIP model to obtain the optimal distillation parameters, and the trained second CLIP model is optimized by using the optimal distillation parameters to obtain the optimal second CLIP model. The delay during inference is reduced and the processing speed of the visual question answering task is improved, that is, the problem that the network structure for processing the visual question answering task in the existing solution is relatively complex, resulting in a slow processing speed.

[0070] The embodiment of the present application also provides a processing device for visual question and answer. It should be noted that the processing device for visual question and answer in the embodiment of the present application can be used to execute the processing method for visual question and answer provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0071] The following is an introduction to the visual question answering processing device provided in an embodiment of the present application.

[0072] Figure 3 is a structural block diagram of a visual question answering processing device provided according to an embodiment of the present application. Figure 3 As shown, the device comprises:

[0073] A first processing unit 31 is used to use a neural network model to perform distillation processing on configuration information and parameters of a first CLIP model to obtain distillation parameters, wherein the configuration information of the first CLIP model includes the number of layers and the vector dimension of each layer, and the first CLIP model is used to score the matching degree between the image and the text to obtain a matching score;

[0074] A second processing unit 32 is used to train a second CLIP model using the distillation parameters to obtain a trained second CLIP model, wherein the number of parameters of the second CLIP model is less than the number of parameters of the first CLIP model;

[0075] The third processing unit 33 is used to use the first CLIP model and the trained second CLIP model to optimize the output of the neural network model to obtain optimal distillation parameters, and use the optimal distillation parameters to optimize the trained second CLIP model to obtain an optimal second CLIP model.

[0076] In the above device, the configuration information and parameters of the first CLIP model are distilled by using a neural network model to obtain distillation parameters; the second CLIP model is trained by using the distillation parameters to obtain a trained second CLIP model; the output of the neural network model is optimized by using the first CLIP model and the trained second CLIP model to obtain the optimal distillation parameters, and the trained second CLIP model is optimized by using the optimal distillation parameters to obtain the optimal second CLIP model. The delay during inference is reduced and the processing speed of the visual question answering task is improved, that is, the problem that the network structure for processing the visual question answering task in the existing solution is relatively complex, resulting in a slow processing speed.

[0077] In one embodiment of the present application, the third processing unit includes an acquisition module, a first processing module, a second processing module and a first optimization module, the acquisition module is used to acquire a test data set, the test data set includes multiple test images and multiple test texts; the first processing module is used to use the first CLIP model to process the test data set to obtain a first PPL; the second processing module is used to use the trained second CLIP model to process the test data set to obtain a second PPL; the first optimization module is used to optimize the output of the neural network model based on the first PPL and the second PPL to obtain the optimal distillation parameters.

[0078] In one embodiment of the present application, the third processing unit includes a first determination module and a second optimization module, the first determination module is used to determine that the loss value is the difference between the above-mentioned second PPL and the above-mentioned first PPL; the second optimization module is used to continuously adopt the above-mentioned configuration information and parameters of the above-mentioned first CLIP model when the above-mentioned loss value is greater than or equal to the loss threshold, optimize the output of the above-mentioned neural network model, and obtain the optimized distillation parameters, until the above-mentioned loss value is less than the above-mentioned loss threshold, determine that the current optimized distillation parameters are the above-mentioned optimal distillation parameters.

[0079] In one embodiment of the present application, the third processing unit includes a second determination module. After determining that the loss value is the absolute value of the difference between the above-mentioned first PPL and the above-mentioned second PPL, the second determination module is used to determine that the above-mentioned optimized distillation parameters are the above-mentioned optimal distillation parameters when the above-mentioned loss value is less than the above-mentioned loss threshold.

[0080] In one embodiment of the present application, the above-mentioned device also includes a fourth processing unit and a fifth processing unit. In the process of using a neural network model to distill the configuration information and parameters of the first CLIP model to obtain the distillation parameters, the fourth processing unit is used to use a word segmenter to segment the above-mentioned configuration information of the above-mentioned first CLIP model to obtain a word segmentation result; and the fifth processing unit is used to use the embedding layer of the above-mentioned neural network model to vectorize the above-mentioned word segmentation result to obtain a configuration information vector.

[0081] In one embodiment of the present application, the above-mentioned device also includes a sixth processing unit. After optimizing the trained second CLIP model by using the above-mentioned optimal distillation parameters to obtain the optimal second CLIP model, the sixth processing unit is used to train the above-mentioned optimal second CLIP model by using the training set of the above-mentioned first CLIP model to obtain the final second CLIP model.

[0082] In one embodiment of the present application, the above-mentioned device also includes a seventh processing unit. After optimizing the above-mentioned trained second CLIP model using the above-mentioned optimal distillation parameters to obtain the optimal second CLIP model, the seventh processing unit is used to convert the weights and activation values ​​of the above-mentioned optimal second CLIP model into INT8 format.

[0083] The visual question answering processing device includes a processor and a memory. The first processing unit, the second processing unit, and the third processing unit are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0084] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem that the network structure of the existing solution for processing visual question answering tasks is relatively complex, resulting in a slow processing speed.

[0085] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0086] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the visual question-answering processing method.

[0087] Specifically, the processing methods for visual question answering include:

[0088] Step S101, using a neural network model to distill configuration information and parameters of a first CLIP model to obtain distillation parameters, wherein the configuration information of the first CLIP model includes the number of layers and the vector dimension of each layer, and the first CLIP model is used to score the matching degree between the image and the text to obtain a matching score;

[0089] Specifically, the above-mentioned matching degree scoring can be implemented in the following way: the image is encoded into a vector of fixed dimension, the answer returned by the visual question answering task is also encoded into a vector of the same fixed dimension, and then the cosine distance is used for scoring.

[0090] Step S102, training a second CLIP model using the distillation parameters to obtain a trained second CLIP model, wherein the number of parameters of the second CLIP model is less than the number of parameters of the first CLIP model;

[0091] Specifically, in order to reduce the computing resources required for training, the parameters of the first CLIP model can be directly used to generate the parameters of the second CLIP model. As a pre-trained model, the CLIP model has been learned on a lot of data and has a lot of knowledge. Therefore, directly migrating its parameters to the second CLIP model can allow the second CLIP model to converge with fewer resources in subsequent training. The above-mentioned large amount of data is specifically 400 million pairs of images and texts.

[0092] Step S103, using the first CLIP model and the trained second CLIP model to optimize the output of the neural network model to obtain optimal distillation parameters, and using the optimal distillation parameters to optimize the trained second CLIP model to obtain an optimal second CLIP model.

[0093] Specifically, the above-mentioned optimal distillation parameters are used to optimize the second CLIP model after training, ensuring that the loss of effect during the distillation process is acceptable, while achieving a reduction in the number of model parameters and a reduction in the delay during inference.

[0094] Optionally, the first CLIP model and the trained second CLIP model are used to optimize the output of the neural network model to obtain optimal distillation parameters, including: obtaining a test data set, wherein the test data set includes multiple test images and multiple test texts; using the first CLIP model to process the test data set to obtain a first PPL; using the trained second CLIP model to process the test data set to obtain a second PPL; based on the first PPL and the second PPL, optimizing the output of the neural network model to obtain the optimal distillation parameters.

[0095] Optionally, based on the first PPL and the second PPL, the output of the neural network model is optimized to obtain the optimal distillation parameters, including: determining a loss value as a difference between the second PPL and the first PPL; when the loss value is greater than or equal to a loss threshold, continuously using the configuration information and parameters of the first CLIP model to optimize the output of the neural network model to obtain optimized distillation parameters, until the loss value is less than the loss threshold, determining the currently optimized distillation parameters as the optimal distillation parameters.

[0096] Optionally, after determining that the loss value is the absolute value of the difference between the first PPL and the second PPL, the method further includes: when the loss value is less than the loss threshold, determining that the optimized distillation parameter is the optimal distillation parameter.

[0097] Optionally, in the process of using a neural network model to distill the configuration information and parameters of the first CLIP model to obtain the distillation parameters, the method further includes: using a word segmenter to segment the configuration information of the first CLIP model to obtain a word segmentation result; and using the embedding layer of the neural network model to vectorize the word segmentation result to obtain a configuration information vector.

[0098] Optionally, after optimizing the trained second CLIP model by using the optimal distillation parameters to obtain the optimal second CLIP model, the method further comprises: training the optimal second CLIP model by using the training set of the first CLIP model to obtain a final second CLIP model.

[0099] Optionally, after optimizing the trained second CLIP model by using the optimal distillation parameters to obtain the optimal second CLIP model, the method further includes: converting the weights and activation values ​​of the optimal second CLIP model into INT8 format.

[0100] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes the visual question answering processing method when running.

[0101] The embodiment of the present invention provides a device, which includes a processor, a memory, and a program stored in the memory and can be run on the processor. When the processor executes the program, at least the following steps are implemented: using a neural network model to distill the configuration information and parameters of a first CLIP model to obtain distillation parameters, the configuration information of the first CLIP model includes the number of layers and the vector dimensions of each layer, and the first CLIP model is used to score the matching degree of an image and a text to obtain a matching score; using the distillation parameters to train a second CLIP model to obtain a trained second CLIP model, the number of parameters of the second CLIP model is less than the number of parameters of the first CLIP model; using the first CLIP model and the trained second CLIP model to optimize the output of the neural network model to obtain the optimal distillation parameters, and using the optimal distillation parameters to optimize the trained second CLIP model to obtain the optimal second CLIP model. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0102] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with at least the following method steps: using a neural network model to distill the configuration information and parameters of a first CLIP model to obtain distillation parameters, the configuration information of the first CLIP model including the number of layers and the vector dimensions of each layer, the first CLIP model being used to score the degree of matching between an image and a text to obtain a matching score; using the distillation parameters to train a second CLIP model to obtain a trained second CLIP model, the number of parameters of the second CLIP model being less than the number of parameters of the first CLIP model; using the first CLIP model and the trained second CLIP model to optimize the output of the neural network model to obtain optimal distillation parameters, and using the optimal distillation parameters to optimize the trained second CLIP model to obtain an optimal second CLIP model.

[0103] The present application also provides a visual question answering processing system, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include means for executing any one of the above methods.

[0104] By using a neural network model, the configuration information and parameters of the first CLIP model are distilled to obtain distillation parameters; the distillation parameters are used to train the second CLIP model to obtain the trained second CLIP model; the output of the neural network model is optimized by using the first CLIP model and the trained second CLIP model to obtain the optimal distillation parameters, and the trained second CLIP model is optimized by using the optimal distillation parameters to obtain the optimal second CLIP model. The delay during inference is reduced and the processing speed of the visual question answering task is improved, that is, the problem that the network structure for processing the visual question answering task in the existing solution is relatively complex, resulting in a slow processing speed.

[0105] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0106] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0107] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0110] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0111] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0112] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0113] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0114] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0115] 1) The visual question answering processing method of the present application adopts a neural network model to distill the configuration information and parameters of the first CLIP model to obtain distillation parameters; adopts the above distillation parameters to train the second CLIP model to obtain the trained second CLIP model; adopts the above first CLIP model and the above trained second CLIP model to optimize the output of the above neural network model to obtain the optimal distillation parameters, and adopts the above optimal distillation parameters to optimize the above trained second CLIP model to obtain the optimal second CLIP model. The delay during reasoning is reduced and the processing speed of the visual question answering task is improved, that is, the problem that the network structure for processing the visual question answering task in the existing solution is relatively complex, resulting in a slow processing speed is solved.

[0116] 2) The visual question answering processing device of the present application uses a neural network model to distill the configuration information and parameters of the first CLIP model to obtain distillation parameters; uses the above distillation parameters to train the second CLIP model to obtain the trained second CLIP model; uses the above first CLIP model and the above trained second CLIP model to optimize the output of the above neural network model to obtain the optimal distillation parameters, and uses the above optimal distillation parameters to optimize the above trained second CLIP model to obtain the optimal second CLIP model. The delay during reasoning is reduced and the processing speed of the visual question answering task is improved, that is, the problem that the network structure for processing the visual question answering task in the existing solution is relatively complex, resulting in a slow processing speed.

[0117] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for processing visual question answering, characterized in that: include: Using a neural network model, distilling configuration information and parameters of a first CLIP model to obtain distillation parameters, wherein the configuration information of the first CLIP model includes the number of layers and the vector dimension of each layer, and the first CLIP model is used to score the degree of matching between the image and the text to obtain a matching score; Training a second CLIP model using the distillation parameters to obtain a trained second CLIP model, wherein the number of parameters of the second CLIP model is less than the number of parameters of the first CLIP model; The first CLIP model and the trained second CLIP model are used to optimize the output of the neural network model to obtain optimal distillation parameters, and the trained second CLIP model is optimized using the optimal distillation parameters to obtain an optimal second CLIP model.

2. The method according to claim 1, characterized in that The first CLIP model and the trained second CLIP model are used to optimize the output of the neural network model to obtain optimal distillation parameters, including: Acquire a test data set, wherein the test data set includes a plurality of test images and a plurality of test texts; Processing the test data set using the first CLIP model to obtain a first PPL; Processing the test data set using the trained second CLIP model to obtain a second PPL; Based on the first PPL and the second PPL, the output of the neural network model is optimized to obtain the optimal distillation parameter.

3. The method according to claim 2, characterized in that Based on the first PPL and the second PPL, optimizing the output of the neural network model to obtain the optimal distillation parameter includes: Determine a loss value as a difference between the second PPL and the first PPL; When the loss value is greater than or equal to the loss threshold, the configuration information and parameters of the first CLIP model are continuously used to optimize the output of the neural network model to obtain optimized distillation parameters, until the loss value is less than the loss threshold, the currently optimized distillation parameters are determined to be the optimal distillation parameters.

4. The method according to claim 3, characterized in that After determining that the loss value is an absolute value of a difference between the first PPL and the second PPL, the method further includes: When the loss value is less than the loss threshold, the optimized distillation parameter is determined to be the optimal distillation parameter.

5. The method according to claim 1, characterized in that: In the process of using the neural network model to distill the configuration information and parameters of the first CLIP model to obtain the distillation parameters, the method further includes: Using a word segmenter to segment the configuration information of the first CLIP model to obtain a word segmentation result; The word segmentation result is vectorized by using the embedding layer of the neural network model to obtain a configuration information vector.

6. The method according to claim 1, characterized in that After optimizing the trained second CLIP model by using the optimal distillation parameter to obtain the optimal second CLIP model, the method further includes: The optimal second CLIP model is trained using the training set of the first CLIP model to obtain a final second CLIP model.

7. The method according to any one of claims 1 to 6, characterized in that After optimizing the trained second CLIP model by using the optimal distillation parameter to obtain the optimal second CLIP model, the method further includes: Convert the weights and activation values ​​of the optimal second CLIP model to INT8 format.

8. A visual question answering processing device, characterized in that: include: a first processing unit, configured to use a neural network model to perform distillation processing on configuration information and parameters of a first CLIP model to obtain distillation parameters, wherein the configuration information of the first CLIP model includes the number of layers and the vector dimension of each layer, and the first CLIP model is used to score the degree of matching between the image and the text to obtain a matching score; A second processing unit is used to train a second CLIP model using the distillation parameters to obtain a trained second CLIP model, wherein the number of parameters of the second CLIP model is less than the number of parameters of the first CLIP model; The third processing unit is used to optimize the output of the neural network model by using the first CLIP model and the trained second CLIP model to obtain an optimal distillation parameter, and optimize the trained second CLIP model by using the optimal distillation parameter to obtain an optimal second CLIP model.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A visual question answering processing system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 7.

Images