Video sign language question and answer method, system, device and storage medium

Abstract

The application discloses a video sign language question and answer method, system, device and storage medium, which are one-to-one corresponding solutions, wherein, in the solutions, a sign language recognition and translation technology utilizes self-supervised pre-training technology to enhance the representation ability of a model, realizes sign language video translation, fully utilizes training data, and improves recognition accuracy; and a large language model and knowledge retrieval technology are utilized to perform generative question and answer under the constraint of a knowledge base, while a user's intention is understood through multiple interactions, context is maintained, and intelligent conversation under a complex task is completed; in addition, in a sign language generation process, a first step is to convert a text into a gloss conforming to a sign language word order, and a second step is to convert the gloss into an action sequence and drive a digital person; in this step, an action smoothing and transition generation technology is used to process the action sequence of the sign language word, reduce the shaking problem of the action, and obtain a more smooth action sequence for driving the digital person.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video sign language question and answer, and in particular to a video sign language question and answer method, system, device and storage medium. BACKGROUND

[0002] In modern society, the deaf community is severely affected in the realization of basic rights and social participation due to limited information access and communication ability. With the development of technology, sign language recognition and translation, natural language question and answer systems, and sign language generation technology have made some progress, but still face some challenges. These challenges mainly come from the complexity of sign language and the diversity of application scenarios. For example, the system of sign language linguistics is not strong, the numerous sign language dialects make data collection and translation difficult, and it is not possible to provide knowledge answers in professional fields in complex scenarios.

[0003] Sign language is a complex form of communication that involves not only hand movements but also facial expressions, body postures, and other modalities of information. The system needs to process and drive these multi-modal data simultaneously to achieve comprehensive and accurate recognition of sign language. By introducing prior knowledge, the semantic and contextual information of sign language can be better understood, thereby improving the accuracy and efficiency of question and answer. For example, the system can use linguistic knowledge, sign language grammar rules, and other prior knowledge in the conversation to optimize the accuracy of each module of the system.

[0004] The video sign language question and answer system is divided into three key modules: a sign language recognition and translation subsystem that translates sign language videos into text sentences to understand the sign language expressions of the hearing-impaired population; a natural language question and answer subsystem that matches the text recognition results with an established sign language question and answer knowledge base to output a text answer; and a sign language synthesis subsystem that converts the natural language text answer into a digital human sign language video based on sign language grammar rules.

[0005] (1) Sign language recognition and translation subsystem.

[0006] Video sign language recognition translation technology can be mainly divided into two categories, one is deep learning, and the other is machine vision. Among them, the deep learning technology is to train a large number of sign language video data, and the model can learn the characteristics and rules of sign language gestures, so as to realize the recognition of sign language in video. Recurrent neural network (RNN) and convolutional neural network (CNN) are two commonly used models. RNN is suitable for processing sign language with time sequence characteristics, which can capture the continuity and variability of sign language gestures in time, and more accurately recognize sign language. CNN mainly processes image data, extracts key features of sign language gestures such as hand shape, contour and motion trajectory, and identifies sign language through feature extraction. Machine vision technology captures gesture actions in video, and extracts information such as hand position, shape and motion trajectory through image processing technology, and inputs the extracted information into a deep learning model for sign language recognition and classification.

[0007] (2) Natural language question and answer subsystem.

[0008] Question and answer system is divided into two technical modes of natural language generation and template matching. Natural language generation (NLG) usually uses seq2seq (sequence to sequence) model or Transformer (transform neural network) model to automatically construct complete answers. NLG is responsible for converting the system action selected by the dialogue strategy module into natural language for feedback. Template matching technology matches questions with templates by defining a series of templates and rules, finds the most suitable template, and generates answers according to the template.

[0009] In general, template generation is simple and direct, and is directly applicable to a series of different practical application task scenarios. However, such technical mode needs to establish a large template library to improve the accuracy to meet the various ways of users, and the method of NLG has higher requirements for the quality of data set in the process of model training, and the accuracy of answer generation is slightly inferior to that of template generation.

[0010] (3) Sign language synthesis subsystem.

[0011] Sign language synthesis consists of two parts, one is sign language word conversion, which constructs a large sign language word library, uses rule matching to divide the text sentence of natural language, and maps the word to sign language word, and the other is sign language digital human driving, which transmits the sequence parameters of sign language word to digital human for driving display.

[0012] However, the defects of the existing scheme are:

[0013] (1) The existing sign language recognition and translation technology needs a large amount of image processing and data analysis. Hand movement has high flexibility and rapidity, often accompanied by occlusion and self-occlusion. The traditional two-dimensional image processing method is difficult to accurately capture the three-dimensional information of hand movement, and the traditional machine learning and deep learning model cannot fully learn the characteristics and rules of gestures, so it is difficult to control the accuracy of recognition.

[0014] (2) The existing generative natural language question and answer technology cannot control the accuracy of question and answer due to the lack of knowledge base constraint, and is prone to "random answering". Template matching needs to build all possible question and answer libraries, which cannot control the accuracy of the answer while the construction cost is huge, and cannot achieve multi-round question and answer.

[0015] (3) The existing sign language generation technology is mainly based on spoken language to sign language word conversion word library for rule matching. The sign language generation result is oral translation and does not contain sign language sequence, which makes it difficult for deaf people to read the result after fitting the sign language word of the digital person.

[0016] Therefore, the present application is proposed. SUMMARY

[0017] The purpose of the present application is to provide a video sign language question and answer method, system, device and storage medium, which not only can complete the smooth conversion between natural language and sign language, but also can efficiently and accurately understand user demand and provide actual answer to the pain point problem of deaf people group information acquisition with fault, and bring substantial convenience to deaf people group.

[0018] The purpose of the present application is achieved by the following technical solutions:

[0019] A video sign language question and answer method, comprising:

[0020] Extracting a skeleton point sequence from an input sign language video, and using a sign language recognition and translation model based on self-supervised pre-training and task fine-tuning to perform sign language recognition and translation to obtain a question text;

[0021] Generating a question and answer under the constraint of a knowledge base by a large language model combined with the question text; in the process of generative question and answer, through multi-round interaction with the large language model, finally generating an answer text corresponding to the question text;

[0022] Converting the answer text into a sign language word text conforming to the sign language sequence, composing a sign language word text sequence, searching and matching a sign language sequence word library to obtain a corresponding action sequence, and processing by combining action smoothing and transition generation technology, driving a digital person to generate a corresponding sign language video.

[0023] A video sign language question and answer system for implementing the foregoing method, comprising:

[0024] A sign language recognition and translation subsystem is configured to extract a skeleton point sequence from an input sign language video and use a sign language recognition and translation model based on self-supervised pre-training and task fine-tuning to perform sign language recognition and translation to obtain a question text;

[0025] A multi-round knowledge retrieval and question answering subsystem is configured to perform generative question answering by a large language model in combination with the question text under the constraint of a knowledge base; in the generative question answering process, the large language model is interacted with through multiple rounds to finally generate an answer text corresponding to the question text;

[0026] A sign language synthesis subsystem is configured to convert the answer text into a sign language gloss text conforming to a sign language syntax, form a sign language gloss text sequence, search and match a sign language sequence glossary to obtain a corresponding action sequence, and process the action sequence using action smoothing and transition generation techniques to drive a digital human to generate a corresponding sign language video.

[0027] A processing device includes one or more processors; a memory for storing one or more programs;

[0028] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0029] A readable storage medium stores a computer program, which, when executed by a processor, implements the aforementioned method.

[0030] As can be seen from the technical solutions provided by the above-mentioned application, the video sign language question and answer scheme based on multi-modal data driving and prior knowledge modeling can construct a complete and efficient information processing closed loop, from accurate input recognition of sign language, to deep processing and accurate answering relying on strong knowledge reserve, to output in the form of natural and fluent sign language, providing a full range and one-stop technical architecture for sign language communication and information transmission; in general, in the video sign language question and answer scheme provided by the application: (1) the sign language recognition and translation technology uses self-supervised pre-training technology to enhance the representation ability of the model, realizes sign language video translation and completes the full use of training data, and improves the recognition accuracy; (2) the large language model and knowledge retrieval technology are used to perform generative question answering under the constraint of a knowledge base, while the user's intention is understood and the context is maintained through multiple interactions (multi-round dialogue) to complete intelligent dialogue under complex tasks; (3) in the sign language generation process, the first step is to convert the text into a gloss (sign language gloss text) conforming to the sign language syntax, and the second step is to convert the gloss into an action sequence and drive the digital human, in which the action smoothing and transition generation technology is used to process the action sequence of the sign language gloss to reduce the shaking problem of the action, and a more smooth action sequence is obtained to drive the digital human. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description only show some of the embodiments of the present application, and other drawings can be obtained by those of ordinary skill in the art without any creative effort.

[0032] Figure 1 A flow chart of a video sign language question and answer method provided by the embodiment of the present application.

[0033] Figure 2 A schematic diagram of the overall architecture of a video sign language question and answer method provided by the embodiment of the present application.

[0034] Figure 3 A schematic diagram of the architecture of a sign language recognition and translation technology provided by the embodiment of the present application.

[0035] Figure 4 A schematic diagram of the technical route of a natural language question and answer technology provided by the embodiment of the present application.

[0036] Figure 5 A schematic diagram of the technical route of a sign language synthesis technology provided by the embodiment of the present application.

[0037] Figure 6 A schematic diagram of the technical route of a text2gloss translation model training technology provided by the embodiment of the present application.

[0038] Figure 7 A schematic diagram of a processing device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort belong to the protection scope of the present application.

[0040] First, the terms possibly used in the present text will be explained as follows:

[0041] The terms “include”, “contain”, “have”, “possess” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example: including a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.) should be interpreted as not only including the explicitly listed technical feature element, but also including other technical feature elements not explicitly listed but known in the art.

[0042] The term "consisting of" shall mean excluding any element not specified in the claim. If the term is used in the context of a claim, the term shall close the claim, excluding from the claim any element not specified in the claim, except for impurities ordinarily associated with the specified elements. If the term is only used in a clause of a claim, it only limits that clause to the specific listed elements excluding any element not specifically named in that clause, but does not exclude elements named in other clauses of the claim from the overall claim.

[0043] Unless specifically stated otherwise, the terms "mounting", "connected", "connecting", "fixed", "fixedly connected", and the like, should be understood in their broadest sense, such as: can be fixed connection, can also be detachable connection, or integral connection; can be mechanical connection, can also be electrical connection; can be direct connection, can also be indirect connection through intermediate medium, can be internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in this paper can be understood according to the specific circumstances.

[0044] A video sign language question and answer method, system, device and storage medium provided by the present application are described in detail below. The contents not described in detail in the embodiments of the present application belong to the prior art known to those skilled in the art. If no specific conditions are specified in the embodiments of the present application, the conventional conditions or the conditions recommended by the manufacturer are used. If no manufacturer is specified for the instruments used in the embodiments of the present application, they are all conventional products that can be obtained by market purchase.

[0045] Embodiment one

[0046] The embodiments of the present application provide a video sign language question and answer method, as shown in Figure 1 The method mainly includes the following steps:

[0047] Step 1, extracting the skeleton point sequence from the input sign language video, and using the sign language recognition and translation model based on self-supervised pre-training and task fine-tuning to perform sign language recognition and translation, and obtaining the question text.

[0048] Preferably, the steps of obtaining the sign language recognition translation model based on self-supervised pre-training and task fine-tuning are as follows: (1) collecting unannotated sign language video data and annotated sign language video data, and selecting a sign language pre-training model; (2) using the unannotated sign language video data to pre-train the sign language pre-training model, including: extracting a skeleton point sequence from the unannotated sign language video data through a human pose extractor; performing mask modeling on the skeleton point sequence and inputting it into the sign language pre-training model, and performing self-supervised pre-training combined with the output skeleton sequence vector of the sign language pre-training model to obtain a self-supervised pre-training model; (3) using the annotated sign language video data to fine-tune the self-supervised pre-training model, including: extracting a skeleton point sequence from the annotated sign language video data through a human pose extractor and inputting it into the self-supervised pre-training model, and performing fine-tuning training combined with a sign language recognition translation large language model through a skeleton sequence vector and language text alignment mechanism to obtain a fine-tuned self-supervised pre-training model and a fine-tuned sign language recognition translation large language model, which constitute a sign language recognition translation model; wherein the language text involved in the skeleton sequence vector and language text alignment mechanism is the annotated text in the annotated sign language video data.

[0049] In the embodiment of the application, the human pose extractor can be realized by conventional technology, which will not be described here.

[0050] Step 2: The large language model generates a question text corresponding answer text under the constraint of the knowledge base. In the process of generative question answering, through multiple rounds of interaction with the large language model, the question text corresponding answer text is finally generated.

[0051] Preferably, the large language model includes a knowledge retrieval large language model and a question answering large language model; the question text is input into the knowledge retrieval large language model, the knowledge retrieval large language model retrieves the N knowledge with the highest matching degree with the question text in the knowledge base, and inputs the N knowledge into the question answering large language model to obtain N answers corresponding; based on the embedding technology, the question text and the N answers corresponding are embedded into the same vector space, and input into the knowledge retrieval large language model to continue the new round of interaction; and the output of the question answering large language model is controlled in the way of reinforcement learning to control the answering process of the new round of interaction; finally, the question text corresponding answer text is generated.

[0052] Preferably, the knowledge retrieval large language model and the question and answer large language model are each trained by a pre-collected data set; the collected data set includes a question and knowledge pair data set, a question and answer pair data set, and a triple data set of questions, high-quality answers, and suboptimal answers; wherein the question and knowledge pair data set is vectorized to construct a vectorized knowledge base; the knowledge retrieval large language model is trained using the vectorized knowledge base; the question and answer pair data set is responsible for fine-tuning the question and answer large language model, so that the fine-tuned question and answer large language model outputs corresponding answers according to input questions; at the same time, the fine-tuned question and answer large language model is optimized in a reinforcement learning manner combined with the triple data set, so that the optimized question and answer large language model outputs answers that can be aligned with the answers in the triple data set.

[0053] Step 3: The answer text is converted into a sign language word text conforming to the sign language order, a sign language word text sequence is formed, a corresponding action sequence is obtained by searching a matching sign language sequence word library, and after processing combined with action smoothing and transition generation technology, a corresponding sign language video is generated to drive the digital person.

[0054] Preferably, the conversion of the answer text into the sign language word text conforming to the sign language order is realized by a text2gloss translation model, which converts natural language text into sign language word text, adopts an encoder-decoder architecture, the encoder is responsible for encoding the answer text into an intermediate representation vector, and the decoder is responsible for decoding the intermediate representation vector into a sign language word text conforming to the sign language order.

[0055] Preferably, the text2gloss translation model is trained for the translation task of natural text to sign language word text based on a spoken language text-sign language word sequence text corpus through transfer learning and large model technology.

[0056] Preferably, the obtaining of the corresponding action sequence by searching the matching sign language sequence word library and the processing combined with the action smoothing and transition generation technology to drive the digital person to generate the corresponding sign language video include: (1) querying each sign language word text in the sign language word text sequence in the sign language sequence word library to obtain the corresponding action sequence, splicing all the action sequences corresponding to the sign language word texts to obtain the action sequence corresponding to the sign language word text sequence; (2) performing action smoothing and transition generation processing on the action sequence corresponding to the sign language word text sequence to obtain a final action sequence; (3) importing the final action sequence into a rendering engine to drive the digital person in the rendering engine to render the corresponding sign language video.

[0057] In order to more clearly show the technical solutions provided by the present application and the technical effects produced, the method provided by the embodiments of the present application is described in detail below with specific examples.

[0058] I. Overall summary of the solution.

[0059] Considering that sign language recognition translation, sign language question and answer, and sign language synthesis technology are important technological innovations in the field of artificial intelligence for the communication difficulties of hearing-impaired people, aiming to break through the language barrier through technical means and promote information barrier-free communication. In view of the problems existing in the prior art, the embodiment of the present application provides a video sign language question and answer method based on multi-modal data driving and prior knowledge modeling, which breaks through the technical chain of sign language recognition translation, knowledge question and answer large language model, and generates sign language sequence driving digital person, realizes the seamless integration and collaborative operation of the three key links.

[0060] In the aspect of video sign language recognition and translation technology, first, a large amount of unlabeled video data is used to pre-train a sign language model (for example, SignBERT sign language pre-training model) for self-supervised learning. In the self-supervised pre-training framework of the skeleton point sequence, the prior knowledge of sign language is used to implicitly learn the context relationship in the skeleton point sequence by designing different mask strategies for the input skeleton point sequence and reconstructing the content of the mask. Then, a large amount of labeled video data is used to fine-tune the model output skeleton sequence vector based on the self-supervised pre-training model, align the sign language video text, and perform sign language recognition and translation large language model fine-tuning training. The purpose is to extract the features of visual and text modalities through multi-modal learning, align and map them to a shared hidden space, so that the model can understand sign language and generate related text translation or context understanding. This method combines the self-supervised pre-training technology and the powerful semantic understanding ability of the sign language recognition and translation large language model, and is suitable for processing complex sign language video data.

[0061] The multi-round knowledge retrieval question and answer large language model based on the large language model can better understand and process complex dialogue situations, including but not limited to long-term memory retention, sentiment analysis, etc. By using advanced retrieval algorithms and matching models, the accuracy and efficiency of knowledge retrieval are improved. Not only can it answer directly related questions, but also can make more coherent and demand-oriented answers based on previous communication content, thus showing strong knowledge reserves and deep understanding ability. Whether it is a sign language related professional problem or a wide range of questions, it can quickly understand the core of the problem and give accurate, detailed and practical answers.

[0062] Text-driven digital human sign language generation technology combines natural language processing with fine modeling of sign language actions and efficient generation algorithms. Based on large language translation models, an encoder-decoder architecture is used. The encoder is responsible for converting the question and answer model answer text into an intermediate representation (vector sequence), while the decoder generates sign language sequence text based on this representation. This architecture allows the model to understand the source language while flexibly generating the target language, enabling an efficient translation process. Thus, the input text information can be smoothly, naturally and accurately converted into lively digital human sign language expression.

[0063] As shown in Figure 2 The overall architecture of the above-mentioned method of the present application is shown, which realizes the complete process from sign language recognition to text generation and then to sign language video synthesis. The related processes are realized by corresponding models, Figure 2 The training and inference processes are shown in detail in the following.

[0064] II. Details of the scheme.

[0065] 1. Sign language recognition and translation technology architecture (sign language recognition and translation subsystem).

[0066] In the embodiments of the present application, a sign language video translation method based on a sign language pre-training model (for example, a SignBERT sign language pre-training model) is proposed, which realizes end-to-end translation of sign language video to text through two-stage training. In the first stage, a self-supervised pre-training method is used to train a sign language pre-training model using unlabeled sign language video data: by designing a diversified skeleton point sequence mask strategy and constructing a reconstruction task, the model can implicitly learn the context-dependent relationship of sign language actions. In the second stage, the skeleton sequence vector of the labeled data is extracted based on the model trained in the first stage, and through the alignment mechanism of the skeleton sequence vector and the language text, the sign language recognition and translation large language model is fine-tuned. This framework realizes efficient translation from sign language video to natural language.

[0067] (1) Generalization processing of sign language recognition data.

[0068] The quality of sign language videos is an important factor affecting the quality of pre-trained models. To ensure the quality of sign language data, high-resolution camera equipment should be selected, a uniform and simple background should be used, and multiple recordings should be made to increase data robustness. The video should be edited to remove unnecessary parts, and the format, resolution, and frame rate should be unified to facilitate subsequent processing and analysis. After obtaining the sign language video data, filtering and quality control of the data should be performed. This includes invalid video removal: removing invalid video segments such as blurring; image quality inspection: using automated or semi-automated tools to analyze the clarity, brightness, and contrast of the video to ensure that all videos meet the standard in terms of visual effects; duration and frame rate inspection: using automated tools to help check the duration and frame rate of each video to ensure consistency and unify the dataset format.

[0069] (2) Sign language video translation model training based on self-supervised pre-training model.

[0070] During sign language recognition and translation, the sign language video of the deaf person is obtained through video capture or upload, and is translated into text content that can be understood by the deaf community. With the rapid development of deep neural network algorithms and the collection of large-scale data, self-supervised pre-training large models have shown strong representation capabilities and have helped to realize the application of deep models in real-world scenarios. Therefore, the invention uses self-supervised pre-training technology in the sign language recognition part to enhance the representation capability of the model and realize sign language video translation.

[0071] As shown in Figure 3 , the unlabeled video data is masked modeled using the human pose extractor skeleton point sequence, input into the sign language pre-training model (e.g., SignBERT) to learn the general representation of sign language, and output the skeleton sequence vector. This pre-training method can greatly improve the performance of the model on limited labeled data. Figure 3 The right side shows the principle of self-supervised pre-training, i.e., masking the skeleton point sequence and then reconstructing it through the sign language pre-training model, and combining the reconstruction result for self-supervised training. T is the length of the skeleton point sequence, Figure 3 The gesture feature extraction, Transformer encoder, and hand model perception decoder shown on the right side all belong to the sign language pre-training model.

[0072] The self-supervised pre-training model output above is trained on an unlabeled data set, outputting a skeleton sequence vector with context-dependent relationships of sign language actions, and by aligning the skeleton sequence vector with the corresponding text semantics, a sign language recognition translation large language model is used for fine-tuning training, and a cross-entropy loss function is used in the training process to enhance the model's semantic understanding of sign language. In the labeled sign language video data, each sign language video has corresponding labeled text, which is used for fine-tuning training to enable the fine-tuned model to map sign language videos to natural language text. In the fine-tuning stage, a cross-modal alignment technique is integrated, mainly using a human pose extractor, a sign language pre-training model, and a sign language video recognition translation large model for training, aiming to enable the model to understand sign language and generate related text translation or context understanding through multi-modal learning. This method combines the powerful semantic understanding ability of the pose extraction technology and the sign language recognition translation large language model, and is suitable for processing complex sign language video data. Considering that the method provided by the present application can be combined with related schemes for training, further description is omitted.

[0073] After fine-tuning, the obtained sign language recognition translation model adopts a multi-modal learning architecture that fuses video and text information, enabling it to understand sign language and generate semantic output. The input sign language video is converted into a fixed-length skeleton point sequence by a human pose extractor, representing the dynamic information in the video. These skeleton point sequences contain information such as hand gestures and facial expressions, which are abstract representations of sign language. The sequence is then input into the sign language recognition translation model, which outputs a skeleton sequence vector from the fine-tuned sign language model, and then outputs the video text content as the question text from the fine-tuned sign language recognition translation large language model.

[0074] In scene applications (inference stage), a deaf person records or uploads a sign language video through a camera, which is transmitted to a backend server through a network. The video data is first preprocessed, and the preprocessed video is extracted by a human pose extractor to obtain the corresponding skeleton point sequence, which is simultaneously input into the sign language recognition translation model obtained by training above to predict the question text corresponding to the sign language video. The entire model parameters are imported by fine-tuning the weights and freezing the weights, and the generated question text will be used as the input of the next large language model.

[0075] 2. Knowledge retrieval and question answering architecture based on large language model (multi-round knowledge retrieval and question answering subsystem).

[0076] In the knowledge retrieval and question answering large language models, a retrieval enhancement generation method based on reinforcement learning is combined. In the retrieval enhancement generation method, a reinforcement learning algorithm is designed, and a reward for trusted alignment is defined to enable the question answering large language model to actively guide the topic based on the information obtained by retrieval to achieve multi-round dialogue. Figure 4As shown, the knowledge retrieval and question answering architecture based on large language models and the training and inference process are shown. The core is to train a vectorization model (called knowledge retrieval large language model) by combining retrieval mechanism and knowledge vector large language model, efficient use of context window, and end-to-end joint optimization of retrieval and generation; the knowledge retrieval large language model recalls relevant fragments from the external knowledge base (such as database, document set) according to the user query, splices the retrieved context fragments with the input prompts of the question and answer large language model, and provides rich domain knowledge support for the question and answer large language model as enhanced generation basis. The question and answer large language model generates accurate and traceable answers based on the retrieved reliable reference information, avoiding hallucination problems.

[0077] (1) Data set construction.

[0078] The question and answer system constructs three types of data sets: retrieval subsystem data set, question and answer subsystem data set, and human preference data set.

[0079] First, the data set of the retrieval subsystem (knowledge retrieval large language model) focuses on the problem and knowledge pair, which is to train the retrieval ability of the model, that is, to quickly and accurately find the information related to the user's problem from a large amount of data.

[0080] The data set of the question and answer subsystem (question and answer large language model) is constructed in the form of question and answer pairs. The purpose of this data set is to enable the model to directly answer the user's specific question, rather than just providing relevant knowledge. This requires the model not only to understand the literal meaning of the question, but also to grasp the underlying intent behind the question.

[0081] The human preference data set focuses more on learning the user's preferences and satisfaction. Through the triplets of question, high-quality answer and sub-optimal answer, the model learns how to choose the best option from multiple possible answers, and according to the feedback data, the data set is constructed to optimize the model, so that the model can be self-optimized and adjusted according to the actual reflection, so as to continuously improve the quality of question and answer.

[0082] These different types of data sets together constitute a database of an efficient natural language question and answer system, enabling the system not only to provide fast and accurate information retrieval, but also to directly provide accurate answers and optimize according to user preferences. Through these carefully designed and trained models, the natural language question and answer system can provide professional and personalized services in various fields, greatly improving the efficiency and quality of services.

[0083] (2) Construction of question-knowledge binary tuple (question and knowledge pair) knowledge base.

[0084] The knowledge automatic disassembly tool driven by AI (artificial intelligence) as the core processes the document type unstructured knowledge from multiple channels and different forms, automatically performs semantic understanding and knowledge disassembly through an external large language model tool (such as a GPT model), accurately extracts multiple core information of the document, and generates a knowledge representation of a structured question-knowledge binary tuple, and then a person confirms and labels the disassembled knowledge to realize the standardization and unification of the knowledge format. The GPT model is a generative pre-training language model.

[0085] (3) Knowledge retrieval large language model.

[0086] The knowledge retrieval large language model is a core technology for efficient semantic search. Through a self-supervised learning contrastive prediction (CPC) method, the position relationship of a text segment in a context is predicted, the semantic rules behind the data are captured, the text is mapped to a dense vector, and the semantic similarity is calculated in the vector space.

[0087] The retrieval mechanism is combined with the knowledge vector to form a knowledge retrieval large language model. In the process of generating answers and guiding questions in multiple rounds, the retrieval link is introduced. The knowledge retrieval large language model can effectively find and select the best answer or option based on the context information, and can retrieve relevant knowledge information from outside the knowledge reserve, and dynamically retrieve external knowledge for the question and answer large language model.

[0088] (4) Knowledge recall.

[0089] When the user asks a question, the knowledge retrieval large language model performs preliminary retrieval and knowledge recall (symmetric recall and asymmetric recall), performs rough sorting on the candidate set of preliminary recall, performs fine sorting on the candidate set after rough sorting, sorts according to the matching degree of the binary tuple, selects the top N (i.e. the top N with the highest matching degree) knowledge, and obtains the top N retrieval results and the scores of each retrieval item through multi-path recall of a variety of recall algorithms, thereby accurately retrieving the relevant knowledge answers to the question.

[0090] (5) Multi-round question and answer process.

[0091] The question and answer large language model is fine-tuned and trained in advance using high-quality question and answer pair data, the weights of the parameters in the question and answer large language model are changed, the question and answer large language model can capture rich semantic information, and the context learning method of embedding technology is used to improve the performance of the model, the question and answer are embedded into the same vector space, so that the question and answer large language model can capture the semantic relationship between them. Based on this, the question and answer large language model can understand the context relationship of the question, make inferences and judgments accordingly, and accurately understand the intentions and needs of the deaf person, and simultaneously reply to the question in combination with the results of knowledge recall.

[0092] (6) Dialogue machine management.

[0093] The fine-tuned question and answer large language model tracks the dialogue history and context information, thereby dynamically guiding the adjustment of the dialogue strategy. Based on the dialogue state management technology, the state machine dialogue management maintains the dialogue context. The historical dialogue is recorded, including the input of the question, the output of the question and answer large language model, and the current state of the dialogue. According to the history of the dialogue and the analysis of the context of the dialogue content, the next action of the dialogue is determined. At the same time, according to the purpose of the dialogue and the needs of the deaf, the appropriate dialogue strategy is selected, and the dialogue strategy is adjusted according to the user's feedback. By defining the dialogue strategy (for example, after a certain number of rounds of questioning and answering, the historical dialogue analysis can be stopped, and the dialogue can be ended if the user does not continue to ask questions after a certain period of time), the answer process in different situations is designed, such as continuing to provide specific information or ending the dialogue.

[0094] (7) Human feedback reinforcement learning.

[0095] Through the evaluation results of the binary knowledge pair by artificial evaluation, the human feedback reinforcement learning is used to align the question and answer large language model with human preferences, a reward function is learned using human preferences, and the learned reward is optimized using reinforcement learning to align the model, thereby continuously improving the accuracy of the question and answer pair generation and achieving continuous iteration of the question and answer large language model.

[0096] The training process of human feedback reinforcement learning is divided into three stages. First, the pre-trained question and answer large language model is preliminarily adapted to the task through supervised fine-tuning. Then, the preference data (such as satisfaction scores) of the model output by humans is collected, and a reward model (RM) is trained to learn the evaluation criteria of humans. Finally, the question and answer large language model is optimized through a reinforcement learning algorithm (such as PPO) to maximize the reward score of the question and answer large language model. The entire process is continuously iteratively optimized through human feedback, and ultimately the output of the question and answer large language model is more in line with human values and preferences. The subjective preferences of humans are converted into quantifiable reward signals, and the alignment of the question and answer large language model with human values is achieved.

[0097] The knowledge retrieval large language model and the question and answer large language model are collectively referred to as the knowledge retrieval enhanced generation large language model (hereinafter referred to as the large language model). The training process is divided into two stages: knowledge retrieval and answer generation. In the knowledge retrieval stage, a large amount of knowledge document data is used to train text vectors through contrastive learning, a vectorized knowledge base is constructed, and the question and answer large language model is trained by mapping semantically related questions and answers to similar vector spaces. In the answer generation stage, the question and answer large language model is fine-tuned based on question and answer pair data, and a conditional generation task is used to make the model learn to integrate retrieval knowledge to generate accurate answers. Combined with an end-to-end iterative enhancement strategy (such as iterative retrieval-generation), the knowledge retrieval and generation process is dynamically optimized, and the efficiency of the model's knowledge utilization is improved.

[0098] The inference process of the knowledge retrieval enhanced generation large language model is a dynamic retrieval and generation collaborative process. When the user inputs a question, the knowledge retrieval large language model first recalls the most relevant knowledge fragments from the knowledge base, and then concatenates these retrieval results with the question text as a context prompt to input the question and answer large language model for answer synthesis. The entire process adopts a "retrieval-generation" cascading architecture, in which retrieval ensures the factual accuracy of the answer (provides the latest or specialized knowledge), and the question and answer large language model is responsible for understanding the context and outputting a fluent and natural answer, while the retrieval content is fused with the pre-trained knowledge through the attention mechanism, finally realizing intelligent question and answer that is both accurate and conforms to language habits.

[0099] 3. Sign language video synthesis technology architecture (sign language synthesis subsystem).

[0100] Through the encoder-decoder structure, the encoder is used to convert the spoken language text into an intermediate vector representation, and the decoder generates the sign language text according to this representation. In the inference process, various factors such as the semantics, syntax, context information of the spoken language text and the grammatical rules of the sign language are considered to transcribe the sign language text into a sequence of sign language text, and drive the digital human.

[0101] As shown in Figure 5 , it is a sign language video synthesis workflow diagram, mainly converting text information into sign language video so that the deaf can understand. Its foundation is the translation of spoken language text to sign language text, 3D (three-dimensional) motion capture and parametric model technology. In order to generate coherent and natural sign language videos, first, a sign language isolated word sequence with sign language semantics is generated through a translation model, and then a large amount of sign language motion data is collected through 3D motion capture technology to establish a database containing various sign language vocabulary actions. Using this sign language word action library, rule-based action sequence synthesis is adopted to convert isolated words into action sequences through an action sequence dictionary, which can quickly match the vocabulary in the text to the corresponding sign language action, and then use SmoothNet (a lightweight neural network designed for jitter problems in human pose estimation in videos) to splice the action sequence. During the action sequence splicing process, action smoothing and transition generation techniques are also involved, which help to simulate the smooth transition of natural human motion and avoid harsh switching between actions. Finally, through real-time rendering technology, the system can convert the synthesized sign language actions into digital human animation output in real time, so that the deaf can intuitively see the sign language communication content through the screen.

[0102] (1) text2gloss translation model.

[0103] Figure 6For the text2gloss translation model training roadmap, first use a large amount of fine-labeled text2gloss data (spoken text and sign language word sequence text corpus) for fine-tuning training to obtain a spoken text to sign language text translation model (text2gloss).

[0104] As shown in Figure 6 The text2gloss translation model uses an encoder-decoder architecture, with the encoder responsible for converting the answer text output by the question-answering subsystem into an intermediate representation (vector sequence), and the decoder generating sign language word text that conforms to the sign language syntax based on this representation. This architecture allows the model to understand the answer text while flexibly generating sign language words, enabling an efficient translation process.

[0105] In addition, the text2gloss translation model is also configured with an attention mechanism, which allows the model to focus on different parts of the text during translation and dynamically adjust weights based on context information, making the translation results more accurate when dealing with long sentences and complex sentence structures.

[0106] The input of the text2gloss translation model is the answer text output by the question-answering model, and the output is sign language word text that conforms to the sign language syntax, ultimately forming a sign language word text sequence (gloss sequence).

[0107] (2) gloss2pose (sign language word text to action sequence)

[0108] First, construct a sign language word-action sequence vocabulary table to convert the input gloss sequence into an action sequence through rule matching, and combine action smoothing and transition generation techniques for subsequent digital human driving. Each gloss in sign language is a separate word, and words are independent of each other, so it is also called an isolated word sequence. Multiple glosses corresponding to actions can be directly spliced to form an action sequence for a sign language sentence.

[0109] To achieve the above goal, first, each gloss needs to correspond to an action, for which a motion capture device is used to record motion capture data for commonly used sign language words, each motion capture data including precise body, hand, and facial movements to ensure the accuracy of the motion capture data. For the input gloss sequence, each gloss in the sequence is queried in the vocabulary table to obtain the corresponding action sequence, and multiple action sequences are spliced to obtain the action sequence of the entire sentence. The splicing method chosen is spherical linear interpolation, a method for smooth rotation transition in three-dimensional space, widely used in computer graphics, game development, robot control, etc., and can generate very robust and smooth action sequences.

[0110] To address the jitter problem, a motion smoothing method based on neural network, SmoothNet, is used to de-jitter the motion sequence. SmoothNet introduces an independent time-only dependent refinement network, the core of which is to learn the long-term temporal relationship of each joint without considering the noise correlation between joints. This enables the network to efficiently capture the natural smoothing characteristics of body movement, significantly improving the temporal smoothing of existing estimators without complex spatiotemporal models. Finally, the generated motion sequence is passed into a rendering engine (e.g., Unity) to drive the digital human motion.

[0111] (3) Synthesize sign language videos.

[0112] Import the motion sequence into a rendering engine such as Unity, and bind and debug it with the digital human model. Through the animation controller in the rendering engine, adjust the animation playback parameters and binding relationship to ensure correct display of the animation on the digital human model. Use the real-time rendering function in the rendering engine to generate high-quality sign language animations. Real-time lighting, shadows, materials, and post-processing effects can be applied to enhance visual effects and user experience. By adjusting the rendering settings and applying material optimization, ensure that the digital human image is realistic and the motion is delicate, achieving the best visual presentation effect.

[0113] By applying the above-mentioned scheme provided by the embodiments of the present application, a complete and efficient information processing closed loop can be constructed, from accurate input recognition of sign language, to deep processing and accurate answers relying on strong knowledge reserves, to natural and fluent sign language output, providing a full-service, one-stop technical architecture for sign language communication and information transmission. Through the continuous iteration of the above three parts of the model, the development of current intelligent sign language technology is promoted, and the technical limitations are broken. Mainly to solve the following scene demand pain points: 1) Communication barrier: breaks down the communication barriers between hearing-impaired people and hearing people, enabling both parties to communicate without barriers. 2) Difficulty in information acquisition: through sign language synthesis technology, hearing-impaired people can more easily obtain information from video and audio content, expanding information acquisition channels. 3) Professional service needs: in complex scenarios such as medical treatment and legal consultation, professional translation services can be provided to meet the special needs of hearing-impaired people. In the long run, the present application can reduce the cost of relying on manual translation, improve service efficiency, reduce economic losses caused by communication barriers, significantly improve the social participation and quality of life of the deaf community, promote information equality, and reduce social isolation. At the same time, the application of video sign language question and answer technology is also a positive practice of technological ethics and social responsibility, promoting the construction of a more inclusive and harmonious social environment.

[0114] Those skilled in the art can clearly understand that the above-mentioned embodiments can be implemented by software, or can be implemented by means of software and necessary universal hardware platforms based on the description of the above embodiments. Based on such understanding, the technical solutions of the above-mentioned embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0115] Embodiment two

[0116] The present application also provides a video sign language question and answer system, which is mainly used to implement the method provided by the above-mentioned embodiments, and mainly includes:

[0117] A sign language recognition and translation subsystem is used to extract a skeleton point sequence from an input sign language video, and use a sign language recognition and translation model based on self-supervised pre-training and task fine-tuning to perform sign language recognition and translation, and obtain a question text;

[0118] A multi-round knowledge retrieval and question and answer subsystem is used to perform generative question and answer based on the question text, using a large language model and under the constraint of a knowledge base; in the generative question and answer process, through multi-round interaction with the large language model, an answer text corresponding to the question text is finally generated;

[0119] A sign language synthesis subsystem is used to convert the answer text into a sign language word text conforming to a sign language order, form a sign language word text sequence, and then obtain a corresponding action sequence by searching and matching a sign language sequence word library, and after processing by combining action smoothing and transition generation technology, drive a digital person to generate a corresponding sign language video.

[0120] The overall architecture of the system can also be seen in the previously described Figure 2 In addition, the related technical details have been described in the previous embodiments, and therefore will not be described again.

[0121] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional modules is taken as an example, and in actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the internal structure of the system is divided into different functional modules to complete all or part of the functions described above.

[0122] Embodiment three

[0123] The present application also provides a processing device, such as Figure 7As shown, it mainly comprises: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors realize the method provided by the foregoing embodiments.

[0124] Further, the processing device further comprises at least one input device and at least one output device; in the processing device, the processor, the memory, the input device and the output device are connected through a bus.

[0125] In the embodiments of the present application, the specific types of the memory, the input device and the output device are not limited; for example:

[0126] The input device can be a touch screen, an image acquisition device, a physical button or a mouse, etc.

[0127] The output device can be a display terminal.

[0128] The memory can be a random access memory (RAM), and can also be a non-volatile memory, such as a disk memory.

[0129] Embodiment four

[0130] The present application also provides a readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method provided by the foregoing embodiments is realized.

[0131] In the embodiments of the present application, the readable storage medium as the computer readable storage medium can be arranged in the foregoing processing device, for example, as the memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, etc. various media which can store program codes.

[0132] The above is only the preferred specific implementation of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims. The information disclosed in the background section of this paper only aims to deepen the understanding of the overall background of the present application, and should not be regarded as acknowledging or implying in any form that the information constitutes the prior art known to those skilled in the art.

Claims

1. A method of video sign language question answering, the method comprising: Comprise: extracting a skeleton point sequence from an input sign language video, and using a sign language recognition and translation model based on self-supervised pre-training and task fine-tuning to perform sign language recognition and translation, obtaining a question text; generating a question text under the constraint of a knowledge base by a large language model; In the process of generating a question text, a large language model is used for multi-round interaction to finally generate an answer text corresponding to the question text; wherein the large language model comprises a knowledge retrieval large language model and a question and answer large language model, and the knowledge retrieval large language model and the question and answer large language model are trained by a pre-collected data set; the collected data set comprises a question and knowledge pair data set, a question and answer pair data set, and a three-tuple data set of a question, a high-quality answer and a sub-optimal answer; wherein the question and knowledge pair data set is vectorized to construct a vectorized knowledge base; the knowledge retrieval large language model is trained using the vectorized knowledge base; the question and answer pair data set is used to fine-tune the question and answer large language model, so that the fine-tuned question and answer large language model outputs the corresponding answer according to the input question; at the same time, the fine-tuned question and answer large language model is optimized in a reinforcement learning manner combined with the three-tuple data set, so that the optimized question and answer model outputs an answer that aligns with the answer in the three-tuple data set; convert the answer text into a sign language word text sequence that conforms to the sign language order, search for a corresponding action sequence in a sign language sequence word library, and process the action sequence using action smoothing and transition generation technology to drive a digital human to generate a corresponding sign language video.

2. The method of claim 1, wherein, The steps of using a sign language recognition and translation model based on self-supervised pre-training and task fine-tuning are as follows: Collecting unannotated sign language video data and annotated sign language video data, and selecting a sign language pre-training model; Using unannotated sign language video data to pre-train the sign language pre-training model, including: extracting a skeleton point sequence from the unannotated sign language video data by a human pose extractor; performing mask modeling on the skeleton point sequence and inputting it into the sign language pre-training model, and performing self-supervised pre-training on the output skeleton point sequence of the sign language pre-training model to obtain a self-supervised pre-training model; Using annotated sign language video data to fine-tune the self-supervised pre-training model, including: extracting a skeleton point sequence from the annotated sign language video data by a human pose extractor and inputting it into the self-supervised pre-training model, and fine-tuning the sign language recognition and translation large language model by a skeleton point sequence and language text alignment mechanism to obtain a fine-tuned self-supervised pre-training model and a fine-tuned sign language recognition and translation large language model, which form a sign language recognition and translation model; wherein the language text involved in the skeleton point sequence and language text alignment mechanism is the annotated text in the annotated sign language video data.

3. The method of claim 1, wherein, The generating a question text under the constraint of a knowledge base by a large language model comprises: The problem text is input into a knowledge retrieval large language model, which retrieves N pieces of knowledge with the highest matching degree with the problem text in a knowledge base, and inputs the N pieces of knowledge into a question and answer large language model to obtain N corresponding answers; A context learning method based on embedding technology embeds the problem text and the N pieces of knowledge into the same vector space and inputs them into the knowledge retrieval large language model for a new round of interaction; and the output of the question and answer large language model is controlled in a reinforcement learning manner to control the answering process of the new round of interaction; Finally, the answer text corresponding to the problem text is generated.

4. The method of claim 1, wherein, Further comprising: The text2gloss translation model converts the answer text into a sign language word text conforming to the sign language syntax by using an encoder-decoder architecture, in which the encoder is responsible for encoding the answer text into an intermediate representation vector, and the decoder is responsible for decoding the intermediate representation vector into a sign language word text conforming to the sign language syntax.

5. The method of claim 4, wherein, The text2gloss translation model is trained for the translation task from natural text to sign language word text based on a corpus of spoken text and sign language word sequence text through transfer learning and large model technology.

6. The method of claim 1, wherein, The corresponding action sequence is obtained by searching a matching sign language sequence library, and after processing combined with action smoothing and transition generation technology, a digital person generates a corresponding sign language video, including: Each sign language word text in the sign language word text sequence is queried in the sign language sequence library to obtain the corresponding action sequence, and all sign language word texts are spliced to obtain the action sequence corresponding to the sign language word text sequence; The action sequence corresponding to the sign language word text sequence is processed by action smoothing and transition generation to obtain the final action sequence; The final action sequence is imported into a rendering engine to drive a digital person in the rendering engine to render a corresponding sign language video.

7. A video sign language question answering system, characterized by, For implementing the method of any one of claims 1-6, comprising: A sign language recognition and translation subsystem is used to extract a skeleton point sequence from an input sign language video, and a sign language recognition and translation model based on self-supervised pre-training and task fine-tuning is used for sign language recognition and translation to obtain a problem text; A multi-round knowledge retrieval and question answering subsystem is used to generate a question and answer by a large language model in combination with a problem text under the constraint of a knowledge base; in the process of generative question answering, a new round of interaction is carried out through interaction with the large language model, and finally an answer text corresponding to the problem text is generated; A sign language synthesis subsystem is used to convert the answer text into a sign language word text conforming to the sign language syntax, form a sign language word text sequence, and then obtain a corresponding action sequence by searching a matching sign language sequence library, and after processing combined with action smoothing and transition generation technology, a digital person generates a corresponding sign language video.

8. A processing device, characterized by Comprising: One or more processors; Memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-6.

9. A readable storage medium, storing a computer program, characterized in that, When the computer program is executed by the processor, the method of any one of claims 1-6 is implemented.

Images