Information interaction method and device based on sign language recognition and synthesis
Through sign language recognition and synthesis technology, deaf and dumb users can interact with 3D digital people, solving the problem that deaf and dumb people cannot interact with the voice recognition system and improving the user experience.
Patent Information
- Application Number
- CN202110519861.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-05-12
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-05-12
AI Technical Summary
Deaf and mute people cannot interact with information with 3D digital people using voice recognition and speech synthesis, resulting in the inability to enjoy financial services and affect the user experience.
The information interaction method based on sign language recognition and synthesis is adopted, and the user's sign language gesture is obtained, and the semantic information is determined using a preset sign language recognition model. Combined with the intelligent customer service system and sign language gesture database, response gestures are generated and displayed to the user through 3D digital people.
It realizes information interaction between deaf and dumb users and 3D digital people, improves the user experience of deaf and dumb users, and enables them to conduct intelligent Q&A services with digital people through sign language gestures.
Smart Images

Figure CN113515191B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence, specifically to the field of sign language recognition and synthesis technology, and more specifically to an information interaction method, device, equipment, medium and program product based on sign language recognition and synthesis. Background Art
[0002] With the development of digitalization and informatization, virtual intelligent customer service technology can replace manual customer service to provide users with intelligent question and answer services and has been widely used.
[0003] In one example, a financial institution uses 3D digital human images to simulate real customer service to provide users with intelligent question-and-answer services, and interacts with customers via audio and video through technologies such as speech recognition, speech synthesis, and natural language processing.
[0004] However, due to the hearing and pronunciation difficulties of deaf-mute people, this form is not suitable for deaf-mute people. Deaf-mute people cannot interact with 3D digital people and cannot enjoy financial services, which in turn affects the experience of deaf-mute users. Summary of the invention
[0005] In view of the above problems, the present disclosure provides an information interaction method, apparatus, device, medium and program product based on sign language recognition and synthesis.
[0006] According to a first aspect of the present disclosure, there is provided an information interaction method based on sign language recognition and synthesis, comprising: acquiring a first target sign language gesture;
[0007] Determining semantic information according to the first target sign language gesture and a preset sign language recognition model;
[0008] Determine response information according to the semantic information and a preset intelligent customer service system;
[0009] Determine a second target sign language gesture according to the response information and a preset sign language gesture database;
[0010] The second target sign language gesture is displayed by a 3D digital human.
[0011] According to an embodiment of the present disclosure, determining semantic information according to the target sign language gesture and a preset sign language recognition model includes:
[0012] Acquire video data of the first target sign language gesture;
[0013] The semantic information of the first target sign language gesture is determined according to the video data and the preset sign language recognition model.
[0014] According to an embodiment of the present disclosure, the preset sign language recognition model is provided with a standard sign language video data set, and the determining of the semantic information of the first target sign language gesture according to the video data and the preset sign language recognition model includes:
[0015] Calculating the similarity between the video data of the first target sign language gesture and the standard sign language video dataset;
[0016] Determine semantic information corresponding to the first target sign language gesture according to the similarity.
[0017] According to an embodiment of the present disclosure, determining semantic information corresponding to the first target sign language gesture according to the similarity includes:
[0018] The semantic information corresponding to the standard sign language gesture with the highest similarity is output as the semantic information corresponding to the first target sign language gesture.
[0019] According to an embodiment of the present disclosure, the preset sign language recognition model is obtained by training with a deep learning algorithm, and the preset sign language recognition model is obtained by training with a deep learning algorithm including:
[0020] Obtain a standard sign language video dataset;
[0021] Performing frame processing on the standard sign language video dataset to obtain a processed standard sign language video dataset;
[0022] The processed standard sign language video data set and the semantic information corresponding to the standard sign language are used as training sample data, and a deep learning algorithm is used for training to obtain a preset sign language recognition model.
[0023] According to an embodiment of the present disclosure, determining a second target sign language gesture according to the response information and a preset sign language gesture database includes:
[0024] Determining a sign language vocabulary according to the response information;
[0025] Determining gesture motion data corresponding to the response information according to the sign language vocabulary and a preset sign language gesture database;
[0026] Determine time axis information corresponding to the response information according to the gesture motion data and a preset sign language gesture database;
[0027] The second target sign language gesture is determined according to the gesture motion data and the timeline information.
[0028] According to an embodiment of the present disclosure, the preset sign language gesture database is obtained based on the standard sign language video data set, including:
[0029] Determine a standard sign language gesture motion dataset and a timeline information set according to the standard sign language video dataset;
[0030] Determine a standard sign language dataset according to the gesture motion dataset and the timeline information set of the standard sign language;
[0031] A preset sign language gesture database is established according to the standard sign language data set.
[0032] According to an embodiment of the present disclosure, the gesture motion data set of the standard sign language includes data sets of three spatial dimensions of human body model nodes and motion displacement corresponding to each sign language motion.
[0033] According to an embodiment of the present disclosure, there is a corresponding relationship between the gesture motion data set of the standard sign language and the timeline information set.
[0034] According to an embodiment of the present disclosure, determining response information according to the semantic information and a preset intelligent customer service system includes:
[0035] Inputting the semantic information into a preset intelligent customer service system;
[0036] Output response information corresponding to the semantic information.
[0037] According to an embodiment of the present disclosure, the 3D digital human displays the second target sign language gesture, including:
[0038] determining a gesture movement sequence corresponding to the response information according to the gesture movement data corresponding to the response information and the response information;
[0039] The second target sign language gesture is displayed according to the gesture motion sequence and the timeline information.
[0040] A second aspect of the present disclosure provides an information interaction device based on sign language recognition and synthesis, comprising: an acquisition module, configured to acquire a first target sign language gesture;
[0041] A first determination module, configured to determine semantic information according to the first target sign language gesture and a preset sign language recognition model;
[0042] A second determination module is used to determine response information according to the semantic information and a preset intelligent customer service system;
[0043] A third determination module is used to determine a second target sign language gesture according to the response information and a preset sign language gesture database; and
[0044] A display module is used to display the second target sign language gesture through a 3D digital human.
[0045] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned information interaction method based on sign language recognition and synthesis.
[0046] The fourth aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned information interaction method based on sign language recognition and synthesis.
[0047] The fifth aspect of the present disclosure also provides a computer program product, including a computer program, which implements the above-mentioned information interaction method based on sign language recognition and synthesis when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:
[0049] Figure 1 A diagram schematically illustrates an application scenario of an information interaction method, apparatus, device, medium, and program product based on sign language recognition and synthesis according to an embodiment of the present disclosure;
[0050] Figure 2 A flowchart of an information interaction method based on sign language recognition and synthesis according to an embodiment of the present disclosure is schematically shown;
[0051] Figure 3 Schematically shows a flow chart of another information interaction method based on sign language recognition and synthesis according to an embodiment of the present disclosure;
[0052] Figure 4 A structural block diagram of an information interaction device based on sign language recognition and synthesis according to an embodiment of the present disclosure is schematically shown; and
[0053] Figure 5 A block diagram of an electronic device suitable for implementing an information interaction method based on sign language recognition and synthesis according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0054] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.
[0055] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.
[0056] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.
[0057] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).
[0058] The embodiments of the present disclosure provide an information interaction method based on sign language recognition and synthesis, including: acquiring a first target sign language gesture; determining semantic information according to the first target sign language gesture and a preset sign language recognition model; determining response information according to the semantic information and a preset intelligent customer service system; determining a second target sign language gesture according to the response information and a preset sign language gesture database; and displaying the second target sign language gesture through a 3D digital human.
[0059] Figure 1 The following schematically illustrates an application scenario diagram of an information interaction method, apparatus, device, medium, and program product based on sign language recognition and synthesis according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of the information interaction method based on sign language recognition and synthesis to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, environments or scenarios.
[0060] like Figure 1As shown, the information interaction method based on sign language recognition and synthesis according to this embodiment can be used in electronic devices, and the electronic device can be any one of terminal devices 101, 102, and 103. Terminal devices 101, 102, and 103 are all provided with camera devices, which can obtain video images of users. Network 104 is used to provide a medium for communication links between terminal devices 101, 102, and 103 and server 105. Network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.
[0061] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send video information, etc.
[0062] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.
[0063] Server 105 can be a server that provides various services, such as uploading video information to the background management server by users using terminal devices 101, 102, 103 (only as an example). The background management server can analyze and process the received video information data of the user and feed back the processing results to the terminal device.
[0064] It should be noted that the information interaction method based on sign language recognition and synthesis provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the information interaction device based on sign language recognition and synthesis provided in the embodiment of the present disclosure can generally be set in the server 105. The information interaction method based on sign language recognition and synthesis provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the information interaction device based on sign language recognition and synthesis provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.
[0065] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0066] In one example, a financial institution uses 3D digital human images to simulate real customer service to provide users with intelligent question-and-answer services, and uses technologies such as speech recognition, speech synthesis, and natural language processing to interact with customers in audio and video. In another example, a 3D digital human interacts with users through an application on a terminal device, but this form is not suitable for deaf-mute people because they have hearing and pronunciation difficulties, and they cannot interact with 3D digital humans.
[0067] To solve the above problems, the embodiments of the present disclosure propose an information interaction method based on sign language recognition and synthesis, so that deaf-mute people can interact with 3D digital humans through sign language gestures.
[0068] The following will be based on Figure 1 The scene described by Figure 2 to Figure 5 The information interaction method based on sign language recognition and synthesis of the disclosed embodiment is described in detail.
[0069] Figure 2 The flowchart of the information interaction method based on sign language recognition and synthesis according to an embodiment of the present disclosure is schematically shown.
[0070] like Figure 2 As shown, the method of this embodiment includes operations S210 to S250. The execution subject of this embodiment can be a background system or server, or a device or equipment that executes the method of this embodiment. In this embodiment, the background system is used as the execution subject for description.
[0071] In operation S210, a first target sign language gesture is acquired.
[0072] In one example, the background system can obtain a first target sign language gesture, which is a video image. The background system can obtain the user's first target sign language gesture through a terminal device equipped with a camera.
[0073] In operation S220, semantic information is determined according to the first target sign language gesture and a preset sign language recognition model.
[0074] In one example, the sign language recognition model is a pre-trained model that can recognize semantic information of sign language gestures, and the first target sign language gesture acquired in operation S210 is input into the preset sign language recognition model, and semantic information corresponding to the first target sign language gesture can be output. Through operation S210, the user's sign language gesture can be recognized and the user's sign language gesture can be converted into semantic information corresponding to the sign language gesture.
[0075] In operation S230, response information is determined according to the semantic information and the preset intelligent customer service system.
[0076] In one example, the intelligent customer service system is set with a preset question-and-answer logic, which can automatically match answer information related to the user's question. According to the semantic information obtained in operation S220 and the preset intelligent customer service system, the answer information corresponding to the semantic information can be determined.
[0077] In operation S240, a second target sign language gesture is determined according to the response information and a preset sign language gesture database.
[0078] In this embodiment, the second target sign language gesture can be determined based on the response information of operation S230 and a preset sign language gesture database. The second target sign language gesture corresponds to the response information. The preset sign language gesture database contains gesture actions of standard sign language and semantic information corresponding to the gesture actions of standard sign language. Specifically, the response information can be decomposed into multiple words, and the multiple words are retrieved in the sign language gesture database. The sign language gestures corresponding to each word are combined to obtain the second target sign language gesture.
[0079] In operation S250 , a second target sign language gesture is displayed through the 3D digital human.
[0080] The second target sign language gesture obtained in operation S240 obtains action parameters of the second target sign language gesture according to a preset sign language gesture database, and the 3D digital human displays the second target sign language gesture to the user according to the action parameters of the sign language gesture to answer the user's question.
[0081] According to the information interaction method based on sign language recognition and synthesis provided by the embodiment of the present disclosure, a first target sign language gesture is obtained; semantic information is determined according to the first target sign language gesture and a preset sign language recognition model; response information is determined according to the semantic information and a preset intelligent customer service system; a second target sign language gesture is determined according to the response information and a preset sign language gesture database; the second target sign language gesture is displayed by a 3D digital human to complete the recognition of the sign language, and the response information is determined according to the semantic information of the sign language recognition, and the response information is synthesized into a sign language gesture to be displayed to the user, so that the deaf-mute user can interact with the 3D digital human for information, thereby improving the user experience of the deaf-mute user.
[0082] Figure 3 The following schematically shows a flowchart of another information interaction method based on sign language recognition and synthesis according to an embodiment of the present disclosure.
[0083] like Figure 3 As shown, the method of this embodiment includes operations S310 to S360.
[0084] In operation S310, a first target sign language gesture is acquired.
[0085] The specific scheme and principle of this operation S310 can be found in Figure 2The step S210 shown is not repeated any more.
[0086] In operation S320, semantic information is determined according to the first target sign language gesture and a preset sign language recognition model.
[0087] According to an embodiment of the present disclosure, a preset sign language recognition model is obtained by training with a deep learning algorithm. The preset sign language recognition model is provided with a standard sign language video data set. The training steps of the sign language recognition model specifically include:
[0088] Step 1: Obtain a standard sign language video dataset.
[0089] The standard sign language video dataset can be obtained by sign language experts wearing motion capture equipment. Specifically, sign language experts record and generate the standard sign language video dataset based on "Chinese Sign Language" (revised edition) and "International Common Sign Language Vocabulary".
[0090] The second step: performing frame processing on the standard sign language video dataset to obtain a processed standard sign language video dataset.
[0091] The standard sign language video dataset is frame-processed at a certain time interval to obtain a processed standard sign language video dataset, which is a plurality of video images, and each video image corresponds to a sign language vocabulary.
[0092] The third step: use the processed standard sign language video dataset and the semantic information corresponding to the standard sign language as training sample data, and use a deep learning algorithm to train to obtain a preset sign language recognition model.
[0093] After the second step, the processed standard sign language video data set and the semantic information corresponding to the standard sign language are used as training sample data, and a deep learning algorithm is used for training to obtain a preset sign language recognition model.
[0094] After obtaining the preset sign language recognition model, operation S320 includes the specific steps of sign language recognition:
[0095] The first step: obtaining video data of the first target sign language gesture.
[0096] The second step is to determine the semantic information of the first target sign language gesture according to the video data and the preset sign language recognition model, which specifically includes:
[0097] The video data of the first target sign language gesture is processed by frame division to obtain multiple video images, and these video images are compared with the images in the standard sign language video data set in the preset sign language recognition model to obtain multiple similarities. The standard sign language gesture closest to the first target sign language gesture is determined according to the similarities. Specifically, the standard sign language gesture corresponding to the highest similarity is taken as the recognition result of the first target sign language gesture, and the semantic information corresponding to the standard sign language gesture with the highest similarity is output as the semantic information corresponding to the first target sign language gesture. Through operation S320, the process of the user transmitting information to the background system through sign language is completed.
[0098] In operation S330, response information is determined according to the semantic information and the preset intelligent customer service system.
[0099] According to the embodiments of the present disclosure, semantic information is input into a preset intelligent customer service system, and response information corresponding to the semantic information is output, wherein the preset intelligent customer service system can automatically match response information related to the user's semantic information according to the semantic information of the user. Technologies such as autonomous learning, natural language understanding, knowledge graphs, and intelligent question and answer involved in the intelligent customer service system are all existing technologies and will not be elaborated on herein.
[0100] The semantic information corresponding to the first target sign language gesture recognized in operation S320 is input into a preset intelligent customer service system, and response information corresponding to the semantic information can be determined, where the response information is text information.
[0101] In operation S340, a second target sign language gesture is determined according to the response information and a preset sign language gesture database.
[0102] According to an embodiment of the present disclosure, the response information obtained in operation S330 needs to be converted into a sign language gesture. Before converting the response information into a sign language gesture, a sign language gesture database needs to be established in advance. The steps for establishing the sign language gesture database are as follows:
[0103] The first step is to determine the gesture motion dataset and timeline information set of the standard sign language based on the standard sign language video dataset.
[0104] According to an embodiment of the present disclosure, based on the standard sign language video dataset in operation S320, a gesture motion dataset of each standard sign language can be determined, wherein the gesture motion dataset of the standard sign language includes a dataset of three spatial dimensions of a human body model node and action displacement corresponding to each sign language action.
[0105] The gesture motion data set can be obtained by a sign language expert wearing a motion capture device. Specifically, the motion capture device can obtain a parameter set h for each sign language, including a data set of three dimensions: human model node v, time t, and motion displacement (x, y, z), i.e., h = {[v0, t0, (x, y, z)], [v1, t1, (x1, y1, z1)], ... [vn, tn, (xn, yn, zn)]}, i.e., the sign language is composed of a series of human node movements on a time axis. Among them, the human model node v and the motion displacement (x, y, z) are the gesture motion data of the standard sign language. Multiple groups of gesture motion data constitute the gesture motion data set, and multiple groups of time t constitute the time axis information set.
[0106] The second step: determining a standard sign language dataset based on the gesture motion dataset and timeline information set of the standard sign language.
[0107] In one example, since sign language is composed of a series of human node movements on a time axis, there is a corresponding relationship between the gesture motion dataset of the standard sign language and the time axis information set. The standard sign language dataset can be determined based on the gesture motion dataset of the standard sign language and the time axis information set in the first step.
[0108] Step 3: Establish a preset sign language gesture database based on the standard sign language dataset.
[0109] The standard sign language data set in the second step corresponds one-to-one with the textual meaning of the sign language and is saved in the preset sign language gesture database in the form of (key, value).
[0110] Operation 340 is the following specific steps of converting the response information into a sign language gesture:
[0111] The first step: determine the sign language vocabulary based on the response information.
[0112] The second step: determining the gesture motion data corresponding to the response information based on the sign language vocabulary and the preset sign language gesture database.
[0113] In one example, the response information is decomposed into a plurality of sign language words, and a search is performed in a preset sign language gesture database to determine gesture motion data corresponding to the plurality of sign language words.
[0114] The third step: determine the time axis information corresponding to the response information based on the gesture motion data and the preset sign language gesture database; determine the second target sign language gesture based on the gesture motion data and the time axis information.
[0115] The timeline information corresponding to the response information can be determined based on the gesture motion data determined in the second step, and then the second target sign language gesture can be determined based on the gesture motion data and the timeline information. The second target sign language gesture is the sign language corresponding to the response information.
[0116] After the second target sign language gesture is determined, it needs to be displayed to the user through the 3D digital human. Operations S350 to S360 are the process of the 3D digital human displaying the second target sign language gesture.
[0117] In operation S350, a gesture motion sequence corresponding to the response information is determined based on the gesture motion data corresponding to the response information and the response information.
[0118] In operation S360, a second target sign language gesture is presented according to the gesture motion sequence and the timeline information.
[0119] Since the gesture motion data corresponding to the response information is gesture motion data determined based on a single vocabulary, the gesture motion sequence corresponding to the response information can be further determined in combination with the response information. The gesture motion sequence represents a series of sign language gestures. The 3D digital human can display the animation corresponding to the second target sign language gesture based on the timeline information corresponding to the gesture motion data, thereby completing the information interaction with the deaf-mute user.
[0120] According to the information interaction method based on sign language recognition and synthesis provided by the embodiment of the present disclosure, a first target sign language gesture is obtained; semantic information is determined according to the first target sign language gesture and a preset sign language recognition model; response information is determined according to the semantic information and a preset intelligent customer service system; a second target sign language gesture is determined according to the response information and a preset sign language gesture database; the second target sign language gesture is displayed by a 3D digital human to complete the recognition of the sign language, and the response information is determined according to the semantic information of the sign language recognition, and the response information is synthesized into a sign language gesture to be displayed to the user, so that the deaf-mute user can interact with the 3D digital human for information, thereby improving the user experience of the deaf-mute user.
[0121] Based on the above-mentioned information interaction method based on sign language recognition and synthesis, the present disclosure also provides an information interaction device based on sign language recognition and synthesis. Figure 4 The device is described in detail.
[0122] Figure 4 The structural block diagram of the information interaction device based on sign language recognition and synthesis according to an embodiment of the present disclosure is schematically shown.
[0123] like Figure 4 As shown, the information interaction device 400 based on sign language recognition and synthesis of this embodiment includes an acquisition module 410 , a first determination module 420 , a second determination module 430 , a third determination module 440 and a display module 450 .
[0124] The acquisition module 410 is used to acquire the first target sign language gesture. In one embodiment, the acquisition module 410 can be used to perform the operation S210 described above, which will not be described in detail here.
[0125] The first determination module 420 is used to determine semantic information according to the first target sign language gesture and the preset sign language recognition model. In one embodiment, the first determination module 420 can be used to perform the operation S220 described above, which will not be repeated here.
[0126] The second determination module 430 is used to determine the response information according to the semantic information and the preset intelligent customer service system. In one embodiment, the second determination module 430 can be used to perform the operation S230 described above, which will not be repeated here.
[0127] The third determination module 440 is used to determine the second target sign language gesture according to the response information and the preset sign language gesture database. In one embodiment, the third determination module 440 can be used to perform the operation S240 described above, which will not be repeated here.
[0128] The display module 450 is used to display the second target sign language gesture through the 3D digital human. In one embodiment, the display module 450 can be used to perform the operation S250 described above, which will not be repeated here.
[0129] According to an embodiment of the present disclosure, any multiple modules among the acquisition module 410, the first determination module 420, the second determination module 430, the third determination module 440 and the display module 450 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module 410, the first determination module 420, the second determination module 430, the third determination module 440 and the display module 450 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the acquisition module 410, the first determination module 420, the second determination module 430, the third determination module 440 and the display module 450 may be at least partially implemented as a computer program module, which may perform a corresponding function when executed.
[0130] Figure 5 A block diagram of an electronic device suitable for implementing an information interaction method based on sign language recognition and synthesis according to an embodiment of the present disclosure is schematically shown.
[0131] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 504 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.
[0132] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.
[0133] According to an embodiment of the present disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the I / O interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 504 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 504 as needed.
[0134] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.
[0135] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.
[0136] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the information interaction method based on sign language recognition and synthesis provided by the embodiment of the present disclosure.
[0137] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 501. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.
[0138] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0139] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.
[0140] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).
[0141] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0142] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.
[0143] It should be noted that the information interaction method and device based on sign language recognition and synthesis provided in the embodiments of the present disclosure can be used in the field of artificial intelligence, and can also be used in any field other than artificial intelligence. The application field of the information interaction method and device based on sign language recognition and synthesis provided in the embodiments of the present disclosure is not limited.
[0144] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.
Claims
1. An information interaction method based on sign language recognition and synthesis, comprising: Acquire a first target sign language gesture; Determining semantic information according to the first target sign language gesture and a preset sign language recognition model; Determine response information according to the semantic information and a preset intelligent customer service system; Determine a second target sign language gesture according to the response information and a preset sign language gesture database; The second target sign language gesture is displayed by 3D digital human. The determining a second target sign language gesture according to the response information and a preset sign language gesture database includes: Determining a sign language vocabulary according to the response information; Determining gesture motion data corresponding to the response information according to the sign language vocabulary and a preset sign language gesture database; Determine time axis information corresponding to the response information according to the gesture motion data and a preset sign language gesture database; determining the second target sign language gesture according to the gesture motion data and the time axis information, The preset sign language gesture database is obtained based on a standard sign language video data set, including: Determine a standard sign language gesture motion dataset and a timeline information set according to the standard sign language video dataset; Determine a standard sign language dataset according to the gesture motion dataset and the timeline information set of the standard sign language; A preset sign language gesture database is established based on the standard sign language data set, wherein the gesture motion data set of the standard sign language includes data sets of three spatial dimensions of human body model nodes and action displacement corresponding to each sign language action, and there is a corresponding relationship between the gesture motion data set of the standard sign language and the timeline information set.
2. The method according to claim 1, characterized in that The determining of semantic information according to the target sign language gesture and a preset sign language recognition model includes: Acquire video data of the first target sign language gesture; The semantic information of the first target sign language gesture is determined according to the video data and the preset sign language recognition model.
3. The method according to claim 2, characterized in that The preset sign language recognition model is provided with a standard sign language video data set, and determining the semantic information of the first target sign language gesture according to the video data and the preset sign language recognition model includes: Calculating the similarity between the video data of the first target sign language gesture and the standard sign language video data set; Determine semantic information corresponding to the first target sign language gesture according to the similarity.
4. The method according to claim 3, characterized in that The determining, according to the similarity, semantic information corresponding to the first target sign language gesture includes: The semantic information corresponding to the standard sign language gesture with the highest similarity is output as the semantic information corresponding to the first target sign language gesture.
5. The method according to claim 2 further comprises using a deep learning algorithm to train the preset sign language recognition model, characterized in that: The preset sign language recognition model obtained by training with a deep learning algorithm includes: Obtain a standard sign language video dataset; Performing frame processing on the standard sign language video dataset to obtain a processed standard sign language video dataset; The processed standard sign language video data set and the semantic information corresponding to the standard sign language are used as training sample data, and a deep learning algorithm is used for training to obtain a preset sign language recognition model.
6. The method according to any one of claims 1 to 5, characterized in that: The determining of response information according to the semantic information and a preset intelligent customer service system includes: Inputting the semantic information into a preset intelligent customer service system; Output response information corresponding to the semantic information.
7. The method according to claim 1, characterized in that The 3D digital human displays the second target sign language gesture, including: determining a gesture movement sequence corresponding to the response information according to the gesture movement data corresponding to the response information and the response information; The second target sign language gesture is displayed according to the gesture motion sequence and the timeline information.
8. An information interaction device based on sign language recognition and synthesis, comprising: An acquisition module, used for acquiring a first target sign language gesture; A first determination module, configured to determine semantic information according to the first target sign language gesture and a preset sign language recognition model; A second determination module is used to determine response information according to the semantic information and a preset intelligent customer service system; A third determination module, configured to determine a second target sign language gesture according to the response information and a preset sign language gesture database; as well as A display module is used to display the second target sign language gesture through a 3D digital human. The third determination module is further used to determine the sign language vocabulary according to the response information; determine the gesture motion data corresponding to the response information according to the sign language vocabulary and a preset sign language gesture database; Determine time axis information corresponding to the response information according to the gesture motion data and a preset sign language gesture database; The second target sign language gesture is determined according to the gesture motion data and the timeline information, wherein the preset sign language gesture database is obtained based on a standard sign language video data set: A preset sign language gesture database establishment module is used to determine a standard sign language gesture motion data set and a timeline information set based on the standard sign language video data set; determine a standard sign language data set based on the standard sign language gesture motion data set and the timeline information set; and establish a preset sign language gesture database based on the standard sign language data set, wherein the standard sign language gesture motion data set includes data sets of three spatial dimensions of human body model nodes and action displacements corresponding to each sign language action, and there is a corresponding relationship between the standard sign language gesture motion data set and the timeline information set.
9. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 7.
11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
