Method and system for realizing sign language basic level treatment question and answer based on large model

Through large-scale sign language recognition and natural language processing technology, combined with intelligent question-and-answer system, the problems of low efficiency and limited coverage of traditional sign language translation services are solved, real-time accurate conversion of sign language videos and multi-round dialogue services are realized, and listening is improved The convenience of information acquisition and communication for persons with disabilities.

CN119025657AInactive Publication Date: 2024-11-26浪潮智慧城市科技有限公司
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411198298.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional sign language translation services are inefficient, costly and limited in coverage, making it difficult to meet the information acquisition and communication needs of hearing-impaired people, especially how to convert sign language into text in real time and accurately.

Method used

Using large-scale sign language recognition, natural language processing and intelligent question-answer engines, a sign language recognition model, semantic analysis model and intelligent question-answer system are built through neural network algorithms and knowledge graphs, real-time conversion of sign language videos into text, and multiple rounds of conversations are provided. and contextual understanding services.

Benefits of technology

It realizes the fast and accurate identification and translation of sign language videos, reduces communication delays and misunderstandings, provides an intuitive and easy-to-use interactive interface, improves the convenience of life and social participation of hearing-impaired people, meets the needs of diversified information query, and supports personalized push Serve.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025657A_ABST
    Figure CN119025657A_ABST
Patent Text Reader

Abstract

The invention discloses a sign language basic level treatment question and answer implementation method and system based on a large model, belongs to the technical field of deep learning, and aims to solve the technical problem of how to accurately convert sign languages into texts in real time. Comprising the following steps: constructing a sign language recognition model based on a neural network algorithm; constructing a semantic analysis model based on a natural language processing technology; constructing an intelligent question and answer engine based on a knowledge graph, and integrating policies and regulations, common sense of life and government affair service to construct a knowledge base; a sign language video input by a user is obtained through a user interface, the sign language video serves as input, a corresponding text is predicted and output through a sign language recognition model, semantic understanding and reasoning analysis are conducted on the text through a voice analysis model, a user question is obtained, and an answer matched with the user question is obtained based on an intelligent question and answer engine and a knowledge base. The answers are output in a text or voice form; and optimizing the sign language recognition model, the semantic analysis model, the intelligent question and answer engine and the knowledge base.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and specifically to a method and system for implementing question and answer in grassroots governance through sign language based on a large model. Background Art

[0002] With the advancement of science and technology and the development of society, the construction of smart cities pays more and more attention to the construction of barrier-free environment. However, traditional sign language translation services have problems such as low efficiency, high cost and limited coverage, which makes it difficult to meet the growing information acquisition and communication needs of hearing-impaired people. Therefore, it is particularly important to develop a system that can convert sign language into text in real time and accurately and has intelligent question-answering functions.

[0003] How to convert sign language into text in real time and accurately is a technical problem that needs to be solved. Summary of the invention

[0004] The technical task of the present invention is to address the above shortcomings and provide a large-model-based sign language grassroots governance question-answering implementation method and system to solve the technical problem of how to convert sign language into text in real time and accurately.

[0005] In a first aspect, the present invention provides a method for implementing a question-answering method for grassroots governance through sign language based on a large model, comprising the following steps:

[0006] Sign language recognition: A sign language recognition model is constructed based on a neural network algorithm. The sign language recognition model takes a sign language video as input and is used to perform feature extraction, sequence modeling, and classification recognition on the input sign language video to obtain text;

[0007] Natural language processing: A semantic analysis model is built based on natural language processing technology. The semantic analysis model takes text as input and is used to perform semantic understanding and reasoning analysis on the input text to obtain user questions.

[0008] Intelligent question-answering engine construction: Build an intelligent question-answering engine based on knowledge graphs, and integrate policies and regulations, common sense, and government services to build a knowledge base. The intelligent question-answering engine and the knowledge base work together to provide multi-round dialogue and context understanding services, and are used to match answers to user questions;

[0009] User interaction: The user interface obtains the sign language video input by the user, uses the sign language video as input, predicts the corresponding text output through the sign language recognition model, performs semantic understanding and reasoning analysis on the text through the speech analysis model, obtains the user's question, and obtains the answer that matches the user's question based on the intelligent question-answering engine and knowledge base, and outputs the answer in the form of text or speech;

[0010] Data analysis optimization: Optimize the sign language recognition model, semantic analysis model, intelligent question-answering engine, and knowledge base.

[0011] Preferably, the sign language recognition model is a neural network model constructed based on a convolutional neural network and a recurrent neural network.

[0012] Preferably, when a sign language video is used as input and the corresponding text is predicted and output by a sign language recognition model, the text is displayed in real time through a user interface to facilitate the user to adjust the input sign language video.

[0013] Preferably, the intelligent question-answering engine is provided with an adaptive module, which is used to adjust and update the information in the knowledge base according to answers and user feedback.

[0014] In a second aspect, the present invention provides a large-model-based sign language grassroots governance question-answering system, which is used to realize real-time conversion between sign language and text through a large-model-based sign language grassroots governance question-answering implementation method as described in any one of the first aspects, wherein the system includes a sign language recognition module, a natural speech processing module, an intelligent question-answering engine construction module, a user interaction module, and a data analysis and optimization module;

[0015] The sign language recognition module is used to perform the following: construct a sign language recognition model based on a neural network algorithm, wherein the sign language recognition model takes a sign language video as input and performs feature extraction, sequence modeling, and classification recognition on the input sign language video to obtain text;

[0016] The natural language processing module is used to perform the following: construct a semantic analysis model based on natural language processing technology. The semantic analysis model takes text as input and is used to perform semantic understanding and reasoning analysis on the input text to obtain user questions;

[0017] The intelligent question-answering engine construction module is used to perform the following: construct an intelligent question-answering engine based on the knowledge graph, and integrate policies and regulations, common sense and government services to build a knowledge base. The intelligent question-answering engine and the knowledge base work together to provide multi-round dialogue and context understanding services, and are used to match answers to user questions;

[0018] The user interaction module is used to perform the following: obtain the sign language video input by the user through the user interface, use the sign language video as input, predict and output the corresponding text through the sign language recognition model, perform semantic understanding and reasoning analysis on the text through the speech analysis model, obtain the user's question, and obtain the answer matching the user's question based on the intelligent question-answering engine and the knowledge base, and output the answer in the form of text or speech;

[0019] The data analysis and optimization module is used to perform the following: optimize the sign language recognition model, semantic analysis model, intelligent question-answering engine, and knowledge base.

[0020] Preferably, the sign language recognition model is a neural network model constructed based on a convolutional neural network and a recurrent neural network.

[0021] Preferably, when a sign language video is used as input and the corresponding text is predicted and output by a sign language recognition model, the user interaction module is used to display the text in real time through a user interface to facilitate the user to adjust the input sign language video.

[0022] Preferably, the intelligent question-answering engine is provided with an adaptive module, which is used to adjust and update the information in the knowledge base according to answers and user feedback.

[0023] The method and system for implementing the question-answering of grassroots governance through sign language based on a large model of the present invention have the following advantages:

[0024] 1. Real-time and accuracy: Using deep learning technology, we can achieve fast and accurate recognition and translation of sign language videos, reducing communication delays and misunderstandings;

[0025] 2. Comprehensiveness and personalization: Integrate multi-field knowledge bases to meet the diverse information query needs of the hearing-impaired, while supporting personalized push services;

[0026] 3. Convenience and efficiency: Provide an intuitive and easy-to-use interactive interface and an instant response question-and-answer service to improve the convenience of life and social participation of the hearing-impaired;

[0027] 4. Scalability and flexibility: The design is flexible and easy to integrate other functional modules and expand application scenarios to meet changing needs in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] The present invention is further described below in conjunction with the accompanying drawings.

[0030] Figure 1 This is a flowchart of a method for implementing a question-and-answer method for grassroots governance using sign language based on a large model in Example 1;

[0031] Figure 2 This is a bottom-level framework diagram of a large-model-based sign language grassroots governance question-and-answer system in Example 1;

[0032] Figure 3 This is a technical roadmap for a large-model-based sign language grassroots governance question-answering system in Example 1;

[0033] Figure 4This is a diagram showing the technical elements utilized in Example 1, a large-model-based sign language grassroots governance question-and-answer system. DETAILED DESCRIPTION

[0034] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0035] The embodiments of the present invention provide a method and system for implementing question and answer in grassroots governance through sign language based on a large model, which is used to solve the technical problem of how to convert sign language into text in real time and accurately.

[0036] Embodiment 1:

[0037] The present invention discloses a method for implementing question and answer in grassroots governance through sign language based on a large model, which includes five steps: sign language recognition, natural language processing, intelligent question and answer engine construction, user interaction, and data analysis and optimization.

[0038] Step S100 sign language recognition: construct a sign language recognition model based on a neural network algorithm. The sign language recognition model takes a sign language video as input and is used to perform feature extraction, sequence modeling and classification recognition on the input sign language video to obtain text.

[0039] Among them, the sign language recognition model is a neural network model built based on convolutional neural network and recurrent neural network.

[0040] Step S200 Natural language processing: A semantic analysis model is constructed based on natural language processing technology. The semantic analysis model takes text as input and is used to perform semantic understanding and reasoning analysis on the input text to obtain user questions.

[0041] Step S300: Building an intelligent question-and-answer engine: Building an intelligent question-and-answer engine based on a knowledge graph, and integrating policies and regulations, common sense, and government services to build a knowledge base. The intelligent question-and-answer engine and the knowledge base work together to provide multi-round dialogue and context understanding services, and are used to match answers to user questions.

[0042] The intelligent question-answering engine of this embodiment is configured with an adaptive module, which is used to adjust and update the information in the knowledge base according to answers and user feedback.

[0043] Step S400 user interaction: obtain the sign language video input by the user through the user interface, use the sign language video as input, predict and output the corresponding text through the sign language recognition model, perform semantic understanding and reasoning analysis on the text through the speech analysis model, obtain the user's question, and obtain the answer that matches the user's question based on the intelligent question and answer engine and knowledge base, and output the answer in the form of text or voice.

[0044] In this embodiment, when a sign language video is used as input and the corresponding text is predicted and output by the sign language recognition model, the text is displayed in real time through the user interface to facilitate the user to adjust the input sign language video.

[0045] Step S500: Data analysis optimization: optimizing the sign language recognition model, semantic analysis model, intelligent question-answering engine, and knowledge base.

[0046] In this embodiment, big data analysis technology is used to monitor user behavior, question and answer trends and various models, and the sign language recognition model, question and answer engine and interaction logic are continuously optimized to improve overall performance.

[0047] The method of this embodiment can automatically recognize and translate sign language videos, and combined with the knowledge base and intelligent question-answering engine, provide accurate, convenient and efficient information query and government services for the hearing-impaired, break down communication barriers and promote social integration.

[0048] Embodiment 2:

[0049] The present invention discloses a sign language grassroots governance question and answer system based on a large model, comprising a sign language recognition module, a natural speech processing module, an intelligent question and answer engine construction module, a user interaction module and a data analysis and optimization module.

[0050] The sign language recognition module is used to perform the following: construct a sign language recognition model based on a neural network algorithm, the sign language recognition model takes a sign language video as input, and is used to perform feature extraction, sequence modeling and classification recognition on the input sign language video to obtain text.

[0051] Among them, the sign language recognition model is a neural network model built based on convolutional neural network and recurrent neural network.

[0052] The natural language processing module is used to perform the following: construct a semantic analysis model based on natural language processing technology. The semantic analysis model takes text as input and is used to perform semantic understanding and reasoning analysis on the input text to obtain user questions.

[0053] The intelligent question-answering engine construction module is used to perform the following: build an intelligent question-answering engine based on the knowledge graph, and integrate policies and regulations, common sense and government services to build a knowledge base. The intelligent question-answering engine and the knowledge base work together to provide multi-round dialogue and context understanding services, and are used to match answers to user questions.

[0054] The intelligent question-answering engine of this embodiment is configured with an adaptive module, which is used to adjust and update the information in the knowledge base according to answers and user feedback.

[0055] The user interaction module is used to perform the following: obtain the sign language video input by the user through the user interface, use the sign language video as input, predict and output the corresponding text through the sign language recognition model, perform semantic understanding and reasoning analysis on the text through the speech analysis model, obtain the user's question, and obtain the answer that matches the user's question based on the intelligent question and answer engine and knowledge base, and output the answer in the form of text or speech.

[0056] In this embodiment, when a sign language video is used as input and the corresponding text is predicted and output by the sign language recognition model, the text is displayed in real time through the user interface to facilitate the user to adjust the input sign language video.

[0057] The data analysis and optimization module is used to perform the following: optimize the sign language recognition model, semantic analysis model, intelligent question-answering engine, and knowledge base.

[0058] In this embodiment, the data analysis and optimization module uses big data analysis technology to monitor user behavior, question and answer trends and various models, continuously optimizes the sign language recognition model, question and answer engine and interaction logic, and improves overall performance.

[0059] The system of this embodiment can execute the method disclosed in Example 1 to achieve real-time conversion between sign language and text.

[0060] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A method for implementing question-answering in grassroots governance through sign language based on a large model, characterized in that: The steps include: Sign language recognition: A sign language recognition model is constructed based on a neural network algorithm. The sign language recognition model takes a sign language video as input and is used to perform feature extraction, sequence modeling, and classification recognition on the input sign language video to obtain text; Natural language processing: A semantic analysis model is built based on natural language processing technology. The semantic analysis model takes text as input and is used to perform semantic understanding and reasoning analysis on the input text to obtain user questions. Intelligent question-answering engine construction: Build an intelligent question-answering engine based on knowledge graphs, and integrate policies and regulations, common sense, and government services to build a knowledge base. The intelligent question-answering engine and the knowledge base work together to provide multi-round dialogue and context understanding services, and are used to match answers to user questions; User interaction: The user interface obtains the sign language video input by the user, uses the sign language video as input, predicts the corresponding text output through the sign language recognition model, performs semantic understanding and reasoning analysis on the text through the speech analysis model, obtains the user's question, and obtains the answer that matches the user's question based on the intelligent question-answering engine and knowledge base, and outputs the answer in the form of text or speech; Data analysis optimization: Optimize the sign language recognition model, semantic analysis model, intelligent question-answering engine, and knowledge base.

2. The method for implementing question-answering in grass-roots governance using sign language based on a large model according to claim 1 is characterized in that: The sign language recognition model is a neural network model constructed based on convolutional neural network and recurrent neural network.

3. The method for implementing question-answering in grass-roots governance using sign language based on a large model according to claim 1 is characterized in that: When a sign language video is used as input and the corresponding text is predicted and output by a sign language recognition model, the text is displayed in real time through a user interface to facilitate the user to adjust the input sign language video.

4. The method for implementing question-answering in grass-roots governance through sign language based on a large model according to claim 1 is characterized in that: The intelligent question-answering engine is configured with an adaptive module, which is used to adjust and update the information in the knowledge base based on answers and user feedback.

5. A sign language grassroots governance question-answering system based on a large model, characterized by: Used to realize real-time conversion between sign language and text through a large-model-based sign language grassroots governance question-answering implementation method as described in any one of claims 1 to 4, the system includes a sign language recognition module, a natural speech processing module, an intelligent question-answering engine construction module, a user interaction module, and a data analysis and optimization module; The sign language recognition module is used to perform the following: construct a sign language recognition model based on a neural network algorithm, wherein the sign language recognition model takes a sign language video as input and performs feature extraction, sequence modeling, and classification recognition on the input sign language video to obtain text; The natural language processing module is used to perform the following: construct a semantic analysis model based on natural language processing technology. The semantic analysis model takes text as input and is used to perform semantic understanding and reasoning analysis on the input text to obtain user questions; The intelligent question-answering engine construction module is used to perform the following: construct an intelligent question-answering engine based on the knowledge graph, and integrate policies and regulations, common sense and government services to build a knowledge base. The intelligent question-answering engine and the knowledge base work together to provide multi-round dialogue and context understanding services, and are used to match answers to user questions; The user interaction module is used to perform the following: obtain the sign language video input by the user through the user interface, use the sign language video as input, predict and output the corresponding text through the sign language recognition model, perform semantic understanding and reasoning analysis on the text through the speech analysis model, obtain the user's question, and obtain the answer matching the user's question based on the intelligent question-answering engine and the knowledge base, and output the answer in the form of text or speech; The data analysis and optimization module is used to perform the following: optimize the sign language recognition model, semantic analysis model, intelligent question-answering engine, and knowledge base.

6. The large-model-based sign language grassroots governance question-answering system according to claim 4 is characterized in that: The sign language recognition model is a neural network model constructed based on convolutional neural network and recurrent neural network.

7. The large-model-based sign language grassroots governance question-answering system according to claim 5 is characterized in that: When a sign language video is used as input and the corresponding text is predicted and output by a sign language recognition model, the user interaction module is used to display the text in real time through a user interface so that the user can adjust the input sign language video.

8. The large-model-based sign language grassroots governance question-answering system according to claim 5 is characterized in that: The intelligent question-answering engine is configured with an adaptive module, which is used to adjust and update the information in the knowledge base based on answers and user feedback.

Citation Information

Patent Citations

  • Interaction method and device, terminal equipment and storage medium

    CN110807388A

  • Sign language interaction engine construction method, sign language interaction method, sign language interaction device and sign language interaction equipment

    CN116719414A

  • Multi-round question and answer method and question and answer system based on large language model and knowledge graph

    CN118396107A