A complex scene video question answering method and system of a cloud service robot

Abstract

The application belongs to the technical field of service robot video question and answer, and provides a complex scene video question and answer method and system of a cloud service robot. The complex scene video question and answer method comprises: obtaining question text information and a complex scene video as information interaction data to be queried; extracting all verbs and nouns from the question text information, and extracting object action information and object static information from the video data to be interacted; fusing the verbs and the object action information to obtain video framework features; fusing the nouns and the object static information to obtain behavior action features; comparing the two features of the video framework features and the behavior action features with a prior data set for comparative learning, matching out text information and locating video frame information from the prior data set, performing multi-modal fusion to obtain fusion prediction results; and matching the information interaction data to be queried with the fusion prediction results, selecting a group of fusion prediction results with the highest similarity as visual question and answer output results.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of service robot video question answering, and particularly relates to a complex scene video question answering method and system of a cloud service robot. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Service robots have been widely used in various real-world scenarios such as welcoming, contactless disinfection and distribution, etc. In the above scenarios, the question and answer function based on visual human-computer interaction plays an indispensable role. Video question answering (VideoQA) technology has emerged and been widely used in the field of service robots. VideoQA technology can automatically answer questions in life scenarios according to the video content recorded by service robots, and greatly promotes the development of online education, scene analysis, video content retrieval, etc. by developing interactive artificial intelligence.

[0004] In the continuous video question answering process, as the service robot continuously interacts with the outside world and collects semantic information and entity positioning for each frame of video content, the robot will accumulate a large number of subtle deviations. When the deviation accumulates to a certain saturation, the question answering effect for the video will be significantly reduced. Moreover, because the working scene of the service robot is complex, such as a hospital, a large amount of video information is transmitted at the same time, and the robot has difficulty in accurately cleaning the noise and focusing on the main information, resulting in that the question answering result excessively depends on prior knowledge, the degree of video information capture and differentiation in the actual scene is not enough, the service performance robustness is poor, and the scene is considered single. Traditional single optimization algorithm also has the disadvantage of poor stability, and is difficult to be widely used.

[0005] Video question answering aims to provide correct answers for complex videos and related questions, and has high requirements for the understanding ability of video and language. The inventors found in the development process that the existing video question answering algorithm system describes this task as a multi-modal fusion process, establishes the entire question to the video content, ignores the semantic details of the rich nouns and verbs, and has the disadvantages of poor stability and huge cost, so it is difficult to apply theoretical analysis to practical evaluation. SUMMARY

[0006] In order to solve the technical problems existing in the background art, the present application provides a complex scene video question answering method and system of a cloud service robot, which can fuse the separated nouns and verbs with frame-level and motion-level features respectively, improve the accuracy of human-computer interaction question answering of actual service robots, understand different aspects and granularities of videos, from fine-grained to coarse-grained reasoning problems in the time and space fields, so that the experience of the service robot is better.

[0007] To achieve the above object, the present application adopts the following technical solutions:

[0008] The first aspect of the present application provides a complex scene video question and answer method of a cloud service robot, which comprises:

[0009] Obtaining question text information and complex scene video as information interaction data to be queried;

[0010] Extracting all verbs and nouns from the question text information, and extracting object action information and object static information from the video data to be interacted;

[0011] Fusing the verbs and the object action information to obtain video framework features, and fusing the nouns and the object static information to obtain behavior action features;

[0012] Comparing the two features of the video framework features and the behavior action features with a priori data set for contrast learning, matching the text information and locating the video frame information from the a priori data set, and then performing multi-modal fusion to obtain a plurality of groups of fusion prediction results;

[0013] Matching the information interaction data to be queried with the fusion prediction results, selecting a group of fusion prediction results with the highest similarity as a visual question and answer output result.

[0014] As an implementation mode, in the a priori data set, the priori data of robot visual question and answer under different scenes is also feature-encoded.

[0015] As an implementation mode, the process of comparing the two features of the video framework features and the behavior action features with the a priori data set for contrast learning comprises:

[0016] Obtaining video-question sample pairs in the a priori data set and video-question sample pairs to be queried;

[0017] Comparing the visual information contained in the a priori data set with the video information in the video-question sample pairs to be queried.

[0018] As an implementation mode, a text adapter is used to extract all verbs and nouns from the question text information.

[0019] As an implementation mode, the a priori data set contains video-question sample pairs under different scenes.

[0020] As an implementation mode, in the process of comparing the two features of the video framework features and the behavior action features with the a priori data set for contrast learning, cross-entropy loss and contrast loss are used to learn the sample pairs in the a priori data set.

[0021] The second aspect of the present application provides a complex scene video question and answer system of a cloud service robot, which comprises:

[0022] An interaction data acquisition module is configured to acquire question text information and a complex scene video as information interaction data to be queried.

[0023] An information extraction module is configured to extract all verbs and nouns from the question text information and extract object action information and object static information from the video data to be interacted.

[0024] An information fusion module is configured to fuse the verbs and the object action information to obtain video framework features and fuse the nouns and the object static information to obtain behavior action features.

[0025] A contrast learning module is configured to perform contrast learning on the two features of the video framework features and the behavior action features and a priori data set, match text information and locate video frame information from the a priori data set, and perform multi-modal fusion to obtain a plurality of groups of fusion prediction results.

[0026] A result matching module is configured to match the information interaction data to be queried with the fusion prediction results, select a group of fusion prediction results with the highest similarity as a visual question and answer output result.

[0027] As an implementation form, in the contrast learning module, the process of performing contrast learning on the two features of the video framework features and the behavior action features and the a priori data set comprises:

[0028] Acquiring a video-question sample pair in the a priori data set and a video-question sample pair to be queried.

[0029] Comparing visual information contained in the a priori data set with video information in the video-question sample pair to be queried.

[0030] The third aspect of the present application provides a cloud service robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the complex scene video question and answer method of the cloud service robot of the cloud service robot when executing the program.

[0031] The fourth aspect of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the steps in the complex scene video question and answer method of the cloud service robot of the cloud service robot.

[0032] Compared with the prior art, the present application has the following beneficial effects:

[0033] The system fuses separated nouns and verbs with frame-level and motion-level features, respectively, where the nouns and verbs can come from the question or the answer. In addition, to accommodate benchmarks of different granularities, including word-level and sentence-level, we design an online residual structure adapter to obtain single word embeddings (nouns and verbs) or temporal sequential embeddings (words in a sentence) under different model requirements.

[0034] In order to improve the accuracy and robustness of the service robot video question and answer system, the application constructs a high-quality service robot VideoQA system by means of a contrast learning algorithm, and improves the understanding ability of the question by introducing a contrast-based sampling method, wherein an unrelated question is selected as a negative sample to break the surface correlation between the question and the answer. Based on the designed architecture, the model is suitable for multiple choice and open QA scenarios. At the same time, the system uses an efficient pre-trained language model to retrieve common sense knowledge as a candidate answer to further improve the performance of the model. It can improve the accuracy of human-computer interaction question and answer of the actual service robot, understand different aspects and granularities of the video, from fine-grained to coarse-grained reasoning problems in the time and space fields, so that the experience of the service robot is better.

[0035] Advantages of the additional aspects of the application will be given in part in the following description, part will become apparent from the following description, or be known by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0036] The drawings accompanying the specification of this application form a part thereof, serve to further provide a further understanding of the application, and together with the description of the illustrative embodiments of the application and the explanation thereof, to explain the application, and do not constitute an improper limitation of the application.

[0037] Figure 1 is a working example of a video question and answer model of a cloud service robot in an embodiment of the application;

[0038] Figure 2 is a frame schematic diagram of retrieving answers in a pre-training model in an embodiment of the application;

[0039] Figure 3 is an LSTM model structure schematic diagram in an embodiment of the application;

[0040] Fig. 4(a) is a first adapter mode schematic diagram of language modeling in an embodiment of the application;

[0041] Fig. 4(b) is a second adapter mode schematic diagram of language modeling in an embodiment of the application;

[0042] Fig. 4(c) is a third adapter mode schematic diagram of language modeling in an embodiment of the application;

[0043] Figure 5(a) is a hyperparameter experiment result 1 in the NExTQA validation set in an embodiment of the present application;

[0044] Figure 5(b) is a hyperparameter experiment result 2 in the NExTQA validation set in an embodiment of the present application;

[0045] Figure 5(c) is a hyperparameter experiment result 3 in the NExTQA validation set in an embodiment of the present application;

[0046] Figure 5(d) is a hyperparameter experiment result 4 in the NExTQA validation set in an embodiment of the present application;

[0047] Figure 6 is a residual adaptation module structure schematic diagram in an embodiment of the present application;

[0048] Figure 7 is a structure schematic diagram of a typical model and a question sampling model based on contrast learning in the visual question answering method of the cloud service robot in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The present application will be further described below in conjunction with the accompanying drawings and embodiments.

[0050] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application pertains.

[0051] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of a feature, step, operation, device, component and / or combinations thereof.

[0052] The service robot can answer natural questions about the video, which is a powerful proof of the cognitive ability of the robot. In complex scene interaction, the amount of information contained in the video stream is very considerable, and through corresponding information extraction, the robot can even extract key information from each frame of video and give answers and corresponding services. At present, due to the low utilization rate of visual information of most service robot bodies, the service robot simply collects semantic information of the object when interacting with the human, and cannot utilize the large amount of important information in the video stream and make accurate answers in complex scenes, which makes the human-computer interaction ability of most service robots low, and the error of the answer result is large. Most robots can only be applied to specific semantic small scenes, which greatly limits the development of service robots. Compared with text and image, video has more rich and high-quality visual information in the time dimension. How to represent the video through a reasonable video modeling method and extract appearance information and motion information, and at the same time find the relevant segment in the video according to the question and guide the process of answering the question, and how to reduce the calculation cost in the process of processing all video frames to learn the spatio-temporal information, are the key difficulties of the current service robot video question answering technology. Multi-modal information alignment and fusion problem. In order to improve the generalization of the model in processing complex structured data in environments such as family and hospital, and improve the accurate answer of the service robot to the video stream question in the complex scene, the present application proposes to construct a multi-task cooperation and parameter sharing video question answering function for visual information, to obtain the prior data (video plus question description), scene information, entity and spatial relationship in the dialogue video mode, better improve the targeting and humanization of the robot service, and make the robot question answering service function more intelligent and better understand the visual information intention.

[0053] The video question answering system of the service robot is a learning task combining computer vision and natural language processing compared with the traditional voice question answering system, and is a powerful test platform for developing new AI functions. This task needs to learn to reason about objects, relationships and events in the spatio-temporal visual and language fields. In the reasoning process, the video is abstracted as a dynamic stream of interactive objects. At each stage of the video event stream, these objects will interact with each other, and their interaction is reasoned according to the overall context of the query and the video. Computer vision mainly processes given images, including image recognition, image classification and other tasks. Natural language processing mainly processes and understands the content in the form of natural language text, including machine translation, information retrieval, text summary generation and other tasks. Video question answering is to process given video content and questions, generate natural language answers after certain visual question answering technology processing, which is a combination of the two.

[0054] Embodiment one

[0055] Reference Figure 1The embodiment provides a complex scene video question and answer method of a cloud service robot, and comprises the following steps:

[0056] Obtaining question text information and a complex scene video as information interaction data to be queried;

[0057] Extracting all verbs and nouns from the question text information and extracting object action information and object static information from the video data to be interacted;

[0058] Fusing the verbs and the object action information to obtain video framework features and fusing the nouns and the object static information to obtain behavior action features;

[0059] Comparing the two features of the video framework features and the behavior action features with a priori data set for comparative learning, matching text information and locating video frame information from the priori data set, and then performing multi-modal fusion to obtain a plurality of groups of fusion prediction results;

[0060] Matching the information interaction data to be queried with the fusion prediction results, selecting a group of fusion prediction results with the highest similarity as a visual question and answer output result.

[0061] In the specific implementation process, the two features of the video framework features and the behavior action features are compared with the priori data set for comparative learning, the text information is matched and the video frame information is located from the priori data set, and then multi-modal fusion is performed to obtain a plurality of groups of fusion prediction results. This step is realized by using a residual adaptation module as shown in Figure 6 .

[0062] As an interactive task, the visual question and answer task, especially the question and answer in a real scene, often needs to introduce priori and common sense knowledge. The model not only needs to complete the representation learning of the visual and language modalities, but also needs to integrate external knowledge and reason the logical relationship. Specifically, a priori statistical model under different scenes is constructed, and feature encoding is performed on VideoQA priori text data. The task overview is given a video V and a language question Q. VideoQA aims to predict the correct answer a by combining information V and Q. Generally, the VideoQA task is formulated as follows:

[0063]

[0064] wherein, represents a modeling function with trainable weights θ.

[0065] The model further extracts the nouns and verbs of the sentence to extract more rich semantic information from the video content. Specifically, the input question Q is first divided into nouns and verbs by the language processing tool NLTK 2, and then further input into the BERT encoder to obtain single word representation. Then, the text adapter represents different granularities while projecting the extracted motion level and frame level features into the same D-dimensional space, respectively, linearly transformed, and then GELU activated and normalized:

[0066]

[0067]

[0068] wherein D represents the feature dimension space, F a represents the single word representation after BERT encoding, F m represents the noun level word embedding, and GELU is an activation function. The output value of the prior data statistics can be converted into a normalized distribution with a range of [0, 1] through the Norm function.

[0069] Among them, the embodiment provides three adapter modes as shown in FIG. 4(a), FIG. 4(b) and FIG. 4(c).

[0070] After obtaining the visual features and language representations, the model connects them into combined features, and then fuses them through a typical multi-modal transformer with self-attention mechanism to obtain noun representations with frame-level features:

[0071]

[0072] wherein d k represents the dimension of the visual feature. Q F , K F and V F represent the query, key and value features.

[0073] In order to obtain the time fusion feature FT for the following answer prediction, the system aggregates the visual question representation through two layers of linear transformation and attention weighting, and then performs attention weighting:

[0074]

[0075] wherein FT represents the time fusion feature, represents the visual question representation.

[0076] For the base features of nouns and verbs, the multi-modal features are obtained through a linear transformation after fusion:

[0077]

[0078] In the answer prediction stage, according to the overall setting in the VideoQA system, the question and the candidate answer are first connected, and the global query representation FG and the prediction distribution s are obtained by a language modeling method:

[0079]

[0080] where FT represents the temporal question-level fusion feature, FNV represents the temporal noun and verb feature, represents Hadamard product, and λ represents a weighting parameter.

[0081] In the contrastive learning module, the contrastive loss L is used to ensure that the predicted answer of the negative sample is far away from the true answer:

[0082]

[0083] where k is the index of the true answer in the answer set A.

[0084] In addition, the cross-entropy loss LP is used to optimize the multiple-choice and open prediction process, and when the predicted answer a is equal to the benchmark true value answer, yi is set to 1, otherwise 0.

[0085]

[0086] Therefore, the total loss function of the model is:

[0087] L a =L p +L c (10)

[0088] During the training process of the model, a large amount of raw data is generated, and these raw data have a large amount of missing and noise, which seriously affects the quality of the data and causes certain difficulties in mining effective information. Some methods such as data cutting can be used to improve the quality of the data.

[0089] Experiments are carried out on public data sets NExT-QA data set, MSRVTT data set and MSVD data set. When the distribution of training and test segmentation answers is significantly different, the TGIF-QA-R data set is proposed to evaluate the robustness of the VideoQA model. The model is evaluated by calculating the precision. In this embodiment, the hyperparameter experiment results 1-4 of the NExTQA validation set are respectively as shown in Figures 5(a)-5(d) .

[0090] The cloud robot service comparison model based on video question answering is trained under the same parameter pre-training model as a plurality of baseline models. For all implemented VideoQA learning methods, we initialize the video and text description embedding through the pre-training model, as shown in Figure 2 . Wherein, the pre-training model can adopt, for example,Figure 3 The LSTM model shown.

[0091] The video-question pair is randomly sampled and fixed before model training. The model is trained using the training data of the above disclosed data set, and the model is adjusted and evaluated using the relationships in the validation data and test data. The experiment compares the scores of multiple existing models on the same data set, and the evaluation indicators are based on the prediction accuracy, as follows: overall indicator ACC A , temporal reasoning indicator ACC T , causal reasoning indicator ACC C , and description indicator ACC D .

[0092] The structure of the typical model in the visual question answering method of the cloud service robot in the embodiment of the application and the question sampling model based on contrast learning is shown in Figure 7 . Taking the NExT-QA data set as an example, the baseline model and the model are compared in terms of question answering scores on the NExT-QA test set and the training set, and the performance of all models thereon is shown in Table 1 and Table 2 as follows:

[0093] Table 1 Comparison of question answering scores of each model on the NExT-QA training set

[0094]

[0095] Table 2 Comparison of question answering scores of each model on the NExT-QA test set

[0096]

[0097] The experimental results are shown in the above table, which clearly prove that, compared with the traditional VideoQA learning method, the model achieves better performance on two data sets, and the cloud robot service video question answering system based on visual question answering is more suitable for solving visual question answering problems, and gives accurate answers by combining video and text description information.

[0098] Embodiment Two

[0099] The embodiment provides a complex scene video question answering system of a cloud service robot, which comprises:

[0100] An interactive data acquisition module is configured to acquire question text information and complex scene video as information interaction data to be queried;

[0101] An information extraction module is configured to extract all verbs and nouns from the question text information, and extract object action information and object static information from the video data to be interacted;

[0102] An information fusion module is configured to fuse the verb and object action information to obtain video framework features, and fuse the noun and object static information to obtain behavior action features;

[0103] A contrast learning module is configured to perform contrast learning on the video framework features and the behavior action features with a priori data set, match text information and locate video frame information from the priori data set, and perform multi-modal fusion to obtain a plurality of groups of fusion prediction results;

[0104] A result matching module is configured to match the information interaction data to be queried with the fusion prediction results, select a group of fusion prediction results with the highest similarity as a visual question and answer output result.

[0105] In the contrast learning module, the process of performing contrast learning on the video framework features and the behavior action features with the priori data set comprises:

[0106] Obtaining video-question sample pairs in the priori data set and the video-question sample pair to be queried;

[0107] Comparing visual information contained in the priori data set with video information in the video-question sample pair to be queried.

[0108] Embodiment three

[0109] The embodiment provides a cloud service robot, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the complex scene video question and answer method of the cloud service robot of the cloud service robot when executing the program.

[0110] Embodiment four

[0111] The embodiment provides a computer readable storage medium, which stores a computer program, and the program implements the steps in the complex scene video question and answer method of the cloud service robot of the cloud service robot when executed by a processor.

[0112] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems) and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a machine that implements the steps in the flowcharts and / or block diagrams. Figure 1 one flow or multiple flows and / or blocks Figure 1means for performing the function specified in the block or blocks.

[0113] The above descriptions are only the preferred embodiment of the application, not intended to limit the application. The application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall fall within the protection scope of the application.

Claims

1. A complex scene video question-answering method of a cloud service robot, characterized in that, The method comprises the steps of: acquiring question text information and complex scene video as information interaction data to be queried; extracting all verbs and nouns from the question text information and object action information and object static information from the video data to be interacted; fusing the verbs and object action information to obtain video framework features; fusing the nouns and object static information to obtain behavior action features, and obtaining single word embedding or time sequence embedding under different model requirements through an online residual structure adapter; comparing the video framework features and the behavior action features with a priori data set for contrast learning, matching text information and locating video frame information from the priori data set, and then performing multi-modal fusion to obtain a plurality of groups of fusion prediction results; matching the information interaction data to be queried with the fusion prediction results, and selecting a group of fusion prediction results with the highest similarity as a visual question and answer output result; wherein a sampling method based on comparison is introduced to improve the understanding of the question, and an unrelated question is selected as a negative sample to break the surface correlation between the question and the answer. 2.The complex-scenario video Q&A method of a cloud service robot according to claim 1, wherein, In the priori data set, the priori data of robot visual question and answer in different scenes are also feature encoded. 3.The complex-scenario video Q&A method of a cloud service robot according to claim 1, wherein, The process of comparing the video framework features and the behavior action features with the priori data set for contrast learning comprises the steps of: acquiring video-question sample pairs in the priori data set and video-question sample pairs to be queried; comparing visual information contained in the priori data set with video information in the video-question sample pairs to be queried. 4.The complex-scenario video Q&A method of a cloud service robot according to claim 1, wherein, The text adapter is used to extract all verbs and nouns from the question text information. 5.The complex-scenario video Q&A method of a cloud service robot according to claim 1, wherein, The priori data set contains video-question sample pairs in different scenes. 6.The complex-scenario video Q&A method of a cloud service robot according to claim 1, wherein, In the process of comparing the video framework features and the behavior action features with the priori data set for contrast learning, cross-entropy loss and contrast loss are used to learn the sample pairs in the priori data set. 7.A complex scene video question and answer system of a cloud service robot, characterized in that, The method comprises the steps of: an interaction data acquisition module for acquiring question text information and complex scene video as information interaction data to be queried; an information extraction module for extracting all verbs and nouns from the question text information and object action information and object static information from the video data to be interacted; an information fusion module for fusing the verbs and object action information to obtain video framework features; fusing the nouns and object static information to obtain behavior action features, and obtaining single word embedding or time sequence embedding under different model requirements through an online residual structure adapter; a contrast learning module for comparing the video framework features and the behavior action features with a priori data set for contrast learning, matching text information and locating video frame information from the priori data set, and then performing multi-modal fusion to obtain a plurality of groups of fusion prediction results; a result matching module for matching the information interaction data to be queried with the fusion prediction results, and selecting a group of fusion prediction results with the highest similarity as a visual question and answer output result; Among them, the understanding ability of the question is improved by introducing a contrast-based sampling method, and an unrelated question is selected as a negative sample to break the surface correlation between the question and the answer. 8.The complex-scenario video Q&A system of a cloud service robot according to claim 7, wherein, In the contrast learning module, the process of contrast learning of the video frame feature and the behavior action feature with the prior data set comprises: Obtaining video-question sample pairs in the prior data set and video-question sample pairs to be queried; Comparing the visual information contained in the prior data set with the video information in the video-question sample pairs to be queried. 9.A cloud service robot, characterized by, The computer program stored on the memory and executable on the processor, when the processor executes the program, implements the steps in the complex scene video question answering method of the cloud service robot of any one of claims 1-6.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps in the complex scene video question answering method of the cloud service robot of any one of claims 1-6.

Images