Sign language translation method and system

Abstract

The invention provides a sign language translation method and system, and belongs to the technical field of sign language translation.The method comprises the steps that in response to the fact that first equipment is in a first translation mode, gesture features are extracted based on gesture images, and the gesture features and a first translation instruction are sent to second equipment; the second equipment is equipment connected with the first equipment; the first translation instruction is used for instructing the second equipment to translate the gesture features; responding to the condition that the first equipment is in the second translation mode, and sending the gesture image and the second translation instruction to the second equipment; the second translation instruction is used for instructing the second device to translate the gesture image. According to the sign language translation method and system provided by the invention, different translation requirements of the user in different scenes can be met.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of sign language translation, and more specifically, to a sign language translation method and system. Background Art

[0002] Smart glasses for sign language translation have broad development prospects. With the continuous advancement of technology, the performance of smart glasses will continue to improve, and the recognition accuracy and translation speed will continue to increase. At the same time, as society pays more attention to the deaf-mute community, the application scope of this technology will continue to expand. In the future, smart glasses are expected to become an indispensable auxiliary tool for the deaf-mute in their daily life and work.

[0003] However, the current technology of using smart glasses for sign language translation still has problems and shortcomings. Real-time sign language translation requires powerful computing power and a large amount of storage space. As a portable device, smart glasses have limited computing resources and storage capacity, which makes it difficult to meet the needs of complex sign language recognition and translation algorithms. Therefore, existing smart glasses often collect image information through cameras and transmit it to other devices for image recognition and gesture translation to reduce the computing pressure of smart glasses. However, gesture translation usually needs to be carried out in real time, and the method of transmitting gesture images to other devices (such as smartphones) for translation usually takes a certain amount of time, especially when the amount of image data is large. At the same time, the smartphone starts feature extraction only after receiving the gesture image, which is a sequential process. Before extracting features, the image must be received first, and parallel processing cannot be performed. This leads to low efficiency of the entire sign language translation process, which is difficult to meet user needs. Summary of the invention

[0004] The purpose of the present disclosure is to provide a sign language translation method and system to meet different translation needs of users in different scenarios.

[0005] According to a first aspect of an embodiment of the present disclosure, a sign language translation method is provided, which is applied to a first device and includes: In response to the first device being in the first translation mode, the gesture feature is extracted based on the gesture image, and the gesture feature and the first translation instruction are sent to the second device. The second device is a device connected to the first device. The first translation instruction is used to instruct the second device to translate the gesture feature.

[0006] In response to the first device being in the second translation mode, the gesture image and the second translation instruction are sent to the second device. The second translation instruction is used to instruct the second device to translate the gesture image.

[0007] According to a second aspect of the embodiments of the present disclosure, a sign language translation system is provided, which is applied to a first device and includes: The first translation module is used to extract gesture features based on the gesture image in response to the first device being in the first translation mode, and send the gesture features and the first translation instruction to the second device. The second device is a device connected to the first device. The first translation instruction is used to instruct the second device to translate the gesture features.

[0008] The second translation module is used to send the gesture image and the second translation instruction to the second device in response to the first device being in the second translation mode. The second translation instruction is used to instruct the second device to translate the gesture image.

[0009] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned sign language translation method when executing the computer program.

[0010] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned sign language translation method are implemented.

[0011] The beneficial effects of the sign language translation method and system provided by the embodiments of the present disclosure are: meeting the different translation needs of users in different scenarios, significantly improving the convenience and efficiency of sign language communication, and achieving reasonable allocation of resources.

[0012] Specifically, by setting different translation modes, the flexibility and adaptability of sign language translation are enhanced. In the first translation mode, the first device extracts gesture features before sending them to the second device for translation, which reduces the amount of data transmitted and the requirements for transmission bandwidth. It also provides more refined information for the translation work of the second device, which helps to improve translation efficiency. In the second translation mode, gesture images are sent directly, providing the second device with more original data, which can meet more complex translation needs in specific situations.

[0013] The disclosed embodiments take into full account the problem of limited computing resources and storage capacity of the first device such as smart glasses, thereby avoiding performance bottlenecks caused by complex translation algorithms on the first device and enabling efficient translation with the powerful computing power of the second device.

[0014] Compared with the existing method of transmitting images to other devices for translation, the different translation modes of this solution can be selected according to different scenarios, reducing the time delay of translation. Especially when the demand for real-time translation is high, it can better meet user needs. At the same time, by instructing the second device to translate through instructions, more efficient task allocation and collaborative work are achieved, avoiding the inefficiency of sequential processing, improving the efficiency of the entire sign language translation process, and providing a more convenient and efficient communication auxiliary tool for the deaf-mute group in daily life and work. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A flowchart of a sign language translation method provided by an embodiment of the present disclosure; Figure 2 A structural block diagram of a sign language translation system provided in one embodiment of the present disclosure; Figure 3 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0018] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0019] Please refer to Figure 1 , Figure 1 This is a flowchart of a sign language translation method provided in one embodiment of the present disclosure. The method may include S101 to S102.

[0020] S101: In response to the first device being in a first translation mode, extracting gesture features based on a gesture image, and sending the gesture features and a first translation instruction to a second device. The second device is a device connected to the first device. The first translation instruction is used to instruct the second device to translate the gesture features.

[0021] In this embodiment, the first device may include a wearable device, such as smart glasses. Extracting gesture features based on the gesture image includes: Hand features, head features and eye features are extracted from gesture images based on the first sign language recognition model.

[0022] The first translation instruction is used to instruct the second device to translate the gesture feature, including: The first translation instruction instructs the second device to translate the gesture feature based on the first sign language translation model.

[0023] In this embodiment, when the first device (such as smart glasses) is in the first translation mode, first, the camera of the smart glasses collects a gesture image containing a sign language action. Then, the gesture image is processed using the built-in first sign language recognition model to extract hand features, head features, and eye features. Hand features may include features such as the shape of the hand, the position and movement of the fingers, and head features may provide the head posture and facial expression information of the sign language speaker to assist in determining the context and emotional tendency of the sign language. Eye features may reflect information such as the direction of the speaker's attention and eye contact, further enriching the connotation of the sign language expression. Finally, the extracted gesture features and the first translation instruction are sent to the second device connected thereto, instructing the second device to translate the gesture features using the first sign language translation model.

[0024] In this embodiment, the sign language translation method further includes: The gesture weight of the first gesture is determined to be a first weight, and the first gesture is a univocal gesture.

[0025] The gesture weight of the second gesture is determined to be a second weight, the second weight is determined based on the number of meanings of the second gesture, and the second gesture is an ambiguous gesture.

[0026] In this embodiment, extracting gesture features based on gesture images includes: A gesture category of each gesture is determined based on the gesture image, and the gesture category includes a univocal gesture or a polyvocal gesture.

[0027] The gesture features of each unambiguous gesture in the gesture image are extracted, and the first weight of each unambiguous gesture is determined.

[0028] Gesture features of each polysemic gesture in the gesture image are extracted, and a second weight of each polysemic gesture is determined, where the second weight is determined based on the number of meanings of the polysemic gesture.

[0029] A first target feature vector is determined based on the gesture features of each univocal gesture and a first weight of each univocal gesture.

[0030] A second target feature vector is determined based on the gesture features of each of the ambiguous gestures and a second weight of each of the ambiguous gestures.

[0031] The first target feature vector and the second target feature vector are fused to obtain a target gesture feature vector.

[0032] Accordingly, sending the gesture feature and the first translation instruction to the second device includes: The target gesture feature vector and the first translation instruction are sent to the second device, where the first translation instruction is specifically used to instruct the second device to translate the target gesture feature vector.

[0033] In this embodiment, the first device can establish a communication connection with the second device to ensure that the gesture features and the translation instructions can be sent to the second device quickly and stably. Wireless communication technologies such as Bluetooth and Wi-Fi can be used.

[0034] For example, in a school classroom, deaf and mute students wear smart glasses, and the teacher uses sign language to assist in the lecture. The smart glasses are in the first translation mode. When the students see the teacher's sign language, the camera of the glasses collects gesture images and extracts hand, head and eye features. For example, the hand features show that the teacher makes a sign language gesture indicating "question", the head tilts slightly to indicate inquiry, and the eye features show that the teacher is paying attention to the student's response. These features are sent to the connected tablet computer (second device), and the tablet computer translates according to the received gesture features, and displays the translation result "The teacher has a question" on the screen or on the lenses of the smart glasses. Students can understand the teacher's meaning through the displayed text, so as to better participate in classroom learning.

[0035] S102: In response to the first device being in the second translation mode, sending the gesture image and the second translation instruction to the second device. The second translation instruction is used to instruct the second device to translate the gesture image.

[0036] In this embodiment, the sign language translation method further includes: In response to receiving the first trigger instruction, the first device is controlled to be in a first translation mode.

[0037] In response to receiving the second trigger instruction, the first device is controlled to be in a second translation mode.

[0038] In this embodiment, a specific trigger instruction is used to control the first device to switch between different translation modes. When the first device receives the first trigger instruction, the first device enters the first translation mode, and when the second trigger instruction is received, the first device switches to the second translation mode. In this way, the appropriate translation mode can be flexibly selected according to different needs and scenarios, thereby improving the efficiency and accuracy of sign language translation.

[0039] Exemplarily, the first device (such as smart glasses) can be provided with physical buttons, touch sensing areas or specific gesture recognition modules, etc., as input methods for triggering instructions.

[0040] In the conference room scenario, the deaf-mute participant wears smart glasses as the first device. When a simple and fast sign language translation is needed, the participant can press a specific button on the smart glasses (first trigger command) to enter the first translation mode. At this time, the smart glasses extract the participant's sign language gesture features and send them to the connected laptop (second device). The laptop quickly translates the received gesture features and outputs the translation content in the form of voice, making it easier for other participants to understand the opinions of the deaf-mute participant.

[0041] If a more detailed and accurate translation is needed when discussing complex issues, participants can press another button (second trigger command) again to switch to the second translation mode. The smart glasses send the complete gesture image to the laptop, which performs more in-depth analysis and translation to ensure the accuracy of the translation results.

[0042] In this embodiment, the second device translating the gesture image includes: A gesture category of each gesture is determined based on the gesture image, and the gesture category includes a univocal gesture or a polyvocal gesture.

[0043] The gesture features of each unambiguous gesture in the gesture image are extracted, and the first weight of each unambiguous gesture is determined.

[0044] Gesture features of each polysemic gesture in the gesture image are extracted, and a second weight of each polysemic gesture is determined, where the second weight is determined based on the number of meanings of the polysemic gesture.

[0045] A first target feature vector is determined based on the gesture features of each univocal gesture and a first weight of each univocal gesture.

[0046] A second target feature vector is determined based on the gesture features of each of the ambiguous gestures and a second weight of each of the ambiguous gestures.

[0047] The first target feature vector and the second target feature vector are fused to obtain a target gesture feature vector.

[0048] Accordingly, sending the gesture feature and the first translation instruction to the second device includes: The target gesture feature vector and the first translation instruction are sent to the second device, where the first translation instruction is specifically used to instruct the second device to translate the target gesture feature vector.

[0049] In this embodiment, the second weight is determined based on the number of meanings of the polysemous gesture, including: The second weight of the ambiguous gesture is calculated by the first formula based on the number of meanings of the ambiguous gesture, or the second weight of the ambiguous gesture is calculated by the second formula based on the number of meanings of the ambiguous gesture.

[0050] The first formula is:

[0051] in, represents the second weight of the j-th polysemous gesture, Indicates the number of meanings of a polysemous gesture, , b, A, and B are all fixed constants.

[0052] The second formula is:

[0053] in, Indicates the number of meanings x and the second weight The mapping coefficient between .

[0054] Exemplarily, when the number of meanings is greater than or equal to the first number (a) and less than the second number (b), the second weight of the ambiguous gesture is determined to be A. When the number of meanings is greater than the second number (b), the second weight of the ambiguous gesture is determined to be B.

[0055] Alternatively, a mapping coefficient between the number of meanings and the second weight is set. When determining the mapping coefficient, the changing trend of the weight as the number of meanings increases is considered. The more meanings there are, the greater the uncertainty of the polysemous gesture, and its weight needs to be reduced accordingly to reflect this uncertainty. Specifically, nonlinear adjustments are made according to actual conditions. For example, several key nodes of the number of meanings can be determined first, such as the number of meanings of 5, 10, 15, etc., and corresponding weight values ​​are set for these nodes based on experience or experimental results. Then, the mapping coefficients corresponding to other numbers of meanings are determined by interpolation or curve fitting. For example, when the number of meanings is 5, the weight is set to 0.8. When the number of meanings is 10, the weight is 0.6. When the number of meanings is 15, the weight is 0.4. Then, the weight values ​​corresponding to other numbers of meanings are determined by methods such as linear interpolation or quadratic curve fitting to obtain mapping coefficients.

[0056] In this embodiment, determining the first target feature vector based on the gesture features of each univocal gesture and the first weight of each univocal gesture includes: Based on the gesture features of each univocal gesture and the first weight of each univocal gesture, a first target feature vector is determined by a third formula.

[0057] The third formula is: T=

[0058] Where n represents the total number of univocal gestures, represents the gesture feature of the kth univocal gesture, T represents the first target feature vector, In this embodiment, the second target feature vector is determined based on the gesture features of each polysemous gesture and the second weight of each polysemous gesture, including: Based on the gesture features of each ambiguous gesture and the second weight of each ambiguous gesture, a second target feature vector is determined by a fourth formula.

[0059] The fourth formula is: S=

[0060] in, , S represents the second target feature vector, represents the second weight of the j-th polysemous gesture, The gesture features representing the j-th polysemous gesture.

[0061] In this embodiment, the first target feature vector and the second target feature vector are fused to obtain a target gesture feature vector, including: A gesture sequence is determined based on the gesture image, where the gesture sequence includes an arrangement order of a plurality of univocal gestures and a plurality of polyvocal gestures.

[0062] Based on the gesture sequence and the fifth formula, the first target feature vector and the second target feature vector are fused to obtain a target gesture feature vector.

[0063] The fifth formula is:

[0064] in, = k + j, the gesture sequence is Seq = { , ..., }, represents the i-th gesture in the gesture sequence, represents the target gesture feature vector, Indicates when gesture When it is a univocal gesture, the index function of the univocal gesture in the first target feature vector is, Indicates when gesture For a polysemous gesture, the index function of the polysemous gesture in the second target feature vector is: represents the weight of the i-th gesture in the gesture sequence.

[0065] In this embodiment, the first translation instruction is specifically used to instruct the second device to translate the target gesture feature vector, including: The second device translates the target gesture feature vector based on the environmental information.

[0066] Exemplarily, the first weight of a univocal gesture determines its contribution to the first target feature vector. Reasonable setting of the first weight can accurately reflect the importance of the univocal gesture in the expression. For example, for some key and commonly used univocal gestures, a higher weight can be given to make them play a leading role in the translation, thereby improving the accuracy of the translation result.

[0067] The second weight of the polysemous gesture is determined based on the number of meanings, which enables the weight to reflect the uncertainty of the polysemous gesture. When the number of meanings is large, reducing its weight can reduce the uncertainty impact of the polysemous gesture in the translation and avoid translation errors caused by the interference of multiple meanings. At the same time, setting the weight reasonably according to the number of meanings can guide the second device to more accurately combine the context and environmental information to determine the specific meaning of the polysemous gesture during translation, thereby improving the accuracy of the translation.

[0068] When fusing the first target feature vector and the second target feature vector, the weight determines the role of each gesture feature in the fusion process. If the weight setting is unreasonable, some important gesture features may be ignored, or some less important gesture features may excessively affect the translation results, thereby reducing the accuracy of the translation.

[0069] The weight setting will also affect the second device's translation of the target gesture feature vector in combination with the environmental information. Reasonable weights can enable the second device to better adjust the translation of different gesture features according to the environmental information, ensure that the translation result is more in line with the actual situation, and improve the accuracy of the translation.

[0070] For example, the target gesture feature vector includes each gesture feature and gesture weight. A gesture with a lower weight means that its meaning is relatively unclear or there is more ambiguity. Therefore, in order to translate these gestures more accurately, more context and environmental information are needed to assist understanding.

[0071] Contextual information can include previous and subsequent gestures, the overall context of the sentence, and the topic of the communication. By considering these contextual factors, the specific meaning of gestures with lower weights in the current situation can be better inferred.

[0072] Environmental information can provide background knowledge about the communication scene, such as the location where the communication takes place, the participants, surrounding objects, etc. This environmental information can help further determine the meaning of gestures, especially for those polysemous gestures or gestures that may have different interpretations in different contexts.

[0073] For example, if a polysemous gesture with a low weight appears, but based on the previous communication content and the current environment, it can be inferred that its most likely meaning is related to a specific topic, then translation incorporating this contextual and environmental information can more accurately convey its intention.

[0074] In this embodiment, when the first device is in the second translation mode, the first device (such as smart glasses) collects a gesture image containing sign language movements through its camera. Subsequently, the first device sends the complete gesture image together with the second translation instruction to the second device connected thereto. The second translation instruction serves as a signal to inform the second device to translate the gesture image in a specified manner. After receiving the image and the instruction, the second device uses its own computing resources and translation algorithm to analyze and process the gesture image. It will identify the sign language movements, hand postures, body postures and other information in the image, and convert the sign language movements into corresponding text or voice information by comparing and analyzing with the pre-stored sign language database, thereby realizing the function of sign language translation.

[0075] For example, in a shopping mall, a deaf-mute customer wears smart glasses. When the customer wants to ask for product information, the smart glasses are switched to the second translation mode through operation. The smart glasses send the collected gesture image of the customer's sign language movement and the second translation instruction to their own smartphone (second device). The second device translates the gesture and outputs the voice. The service desk staff can accurately understand the customer's needs and respond, thereby providing a better shopping experience for the deaf-mute customer.

[0076] In this embodiment, the sign language translation method further includes: Obtain a gesture image dataset and annotate the gesture image dataset.

[0077] The target detection algorithm is trained based on the annotated gesture image dataset to obtain the first sign language recognition model and the first sign language translation model.

[0078] Exemplarily, sign language images with different gestures are collected, which may include sign language images with different angles, lighting conditions and backgrounds, to obtain a sign language image dataset. The sign language image dataset is annotated, and the gestures in each image are annotated to train the model. A target detection algorithm based on deep learning, such as YOLO, Faster R-CNN, etc., is used to identify and train the sign language image dataset, including feature extraction of the sign language images, to obtain a first sign language recognition model.

[0079] The training process of the first sign language translation model includes: using the first sign language recognition model and weight parameters to extract features from the gesture images obtained by the real-time camera, and based on the gesture features, using deep learning classification algorithms such as convolutional neural networks to classify the corresponding gesture images. According to the judgment results of the gesture action, the corresponding labeling operation is performed, and the labeled gesture action is transferred to the large language model to generate the translation text. Through the voice output module, the translation text is converted into voice output, so that ordinary people can hear what the deaf-mute person means.

[0080] It can be concluded from the above that this embodiment meets the different translation needs of users in different scenarios, significantly improves the convenience and efficiency of sign language communication, and achieves reasonable allocation of resources.

[0081] Specifically, by setting different translation modes, the flexibility and adaptability of sign language translation are enhanced. In the first translation mode, the first device extracts gesture features before sending them to the second device for translation, which reduces the amount of data transmitted and the requirements for transmission bandwidth. It also provides more refined information for the translation work of the second device, which helps to improve translation efficiency. In the second translation mode, gesture images are sent directly, providing the second device with more original data, which can meet more complex translation needs in specific situations.

[0082] The disclosed embodiments take into full account the problem of limited computing resources and storage capacity of the first device such as smart glasses, thereby avoiding performance bottlenecks caused by complex translation algorithms on the first device and enabling efficient translation with the powerful computing power of the second device.

[0083] Compared with the existing method of transmitting images to other devices for translation, the different translation modes of this solution can be selected according to different scenarios, reducing the time delay of translation. Especially when the demand for real-time translation is high, it can better meet user needs. At the same time, by instructing the second device to translate through instructions, more efficient task allocation and collaborative work are achieved, avoiding the inefficiency of sequential processing, improving the efficiency of the entire sign language translation process, and providing a more convenient and efficient communication auxiliary tool for the deaf-mute group in daily life and work.

[0084] In one embodiment of the present disclosure, the sign language translation method further includes: In response to the first device being in the third translation mode, a gesture category of each gesture is determined based on the gesture image, the gesture category including a univocal gesture or a polyvocal gesture.

[0085] In response to the gesture category being a univocal gesture, the gesture features of the gesture image are extracted, and the gesture features and a third translation instruction are sent to the second device. The third translation instruction is used to instruct the second device to translate the gesture features.

[0086] In response to the gesture category being an ambiguous gesture, the gesture image and a fourth translation instruction are sent to the second device. The fourth translation instruction is used to instruct the second device to translate the gesture image.

[0087] In this embodiment, the third translation instruction includes a third environment identifier, and in response to the first device being in the third translation mode and the gesture category being a univocal gesture, further includes: An environmental feature is determined based on the environmental information, and the environmental feature and a third environmental identifier are sent to the second device.

[0088] The environmental information includes environmental information around the first user, the first user is a user using the first device, and the third environmental identifier is used to instruct the second device to translate the gesture feature based on the environmental feature.

[0089] In this embodiment, the fourth translation instruction includes a fourth environment identifier, and in response to the first device being in the third translation mode and the gesture category being an ambiguous gesture, further includes: The environment information and the fourth environment identifier are sent to the second device.

[0090] The fourth environment identifier is used to instruct the second device to translate the gesture feature based on the environment information.

[0091] In this embodiment, the sign language translation method further includes: In response to identifying the second user, and the distance between the second user and the first user is less than the first distance, and the duration meets the first duration, controlling the first device to be in the third translation mode.

[0092] In this embodiment, the sign language translation method further includes: In response to receiving the third trigger instruction, the first device is controlled to be in a third translation mode.

[0093] Exemplarily, in a tourist attraction, deaf-mute tourists wear smart glasses. When tourists encounter a complex sign language communication scenario, a specific gesture is used to trigger a third trigger instruction. The smart glasses enter the third translation mode to recognize and classify the collected sign language gestures. For example, the recognized gesture sequence includes 10 gestures. First, each gesture, such a single-meaning gesture, extracts the gesture features and sends them to the tour guide's mobile phone (second device) for quick translation. If it is a polysemous gesture with multiple meanings, the smart glasses will send the gesture image to the tour guide's mobile phone, which will perform a more comprehensive analysis and translation to help the tour guide better understand the needs of tourists and provide better services for deaf-mute tourists.

[0094] In this embodiment, sending the gesture feature to the second device includes: sending the gesture feature to the second device through a first transmission channel. Sending the gesture image to the second device includes sending the gesture image to the second device through a second transmission channel.

[0095] For example, Chinese Sign Language includes two parts: hand gesture and finger language. Hand gesture is mainly used to simulate the appearance and dynamics of things, while finger language is based on written text and formed according to the rules of Chinese Pinyin. Monosemous gestures refer to those sign language movements with clear and unique meanings. Among them, finger language is the most typical monosemous gesture. There are also some monosemous gestures, such as using specific finger movements to represent the number "1", or using a fixed gesture to represent "good", etc. The meaning of these gestures is relatively simple and not easy to cause ambiguity. Polysemous gestures are some sign language movements with multiple different meanings. Their specific meanings need to be determined in combination with context, facial expressions, body postures and other factors. For example, a specific hand movement may mean different meanings such as "like", "want", and "expect" in different situations.

[0096] In this embodiment, when the first device is in the third translation mode, the gesture category of each gesture is first determined based on the gesture image. If it is determined to be a univocal gesture, the gesture features of the gesture image are extracted. These features may include key information such as the shape, position, and motion trajectory of the hand that can uniquely determine the meaning of the univocal gesture. The gesture features are then sent to the second device, and a third translation instruction is attached to instruct the second device to translate the gesture features. If it is determined to be a polysemous gesture, since its meaning is unclear, the complete gesture image is directly sent to the second device, and the third translation instruction is attached, so that the second device can combine more information to determine its specific meaning and translate it.

[0097] Different modes and corresponding processing logic are set for the sign language translation method, mainly considering the complexity and diversity of sign language. Monosemantic gestures have clear meanings, and extracting their key features for translation is more efficient and accurate, because these features can uniquely determine their meanings. Polysemantic gestures require more contextual information to determine the specific meaning, so the complete gesture image is sent directly to allow the second device to perform a more comprehensive analysis. At the same time, environmental information and environmental identification are introduced because the meaning of sign language may change in different environments, and environmental features can assist the second device in translating gestures more accurately. In addition, setting multiple trigger methods to enter the third translation mode is to facilitate users to flexibly start this mode in different scenarios to meet the needs of complex communication scenarios.

[0098] On the one hand, this embodiment improves the accuracy of sign language translation. The classification and processing of monosemous gestures and polysemous gestures, as well as the translation in combination with environmental information, make the translation results more in line with the actual situation and reduce ambiguity. On the other hand, the user experience is enhanced. Deaf-mute tourists can start the third translation mode in a variety of ways in complex scenarios such as tourist attractions and communicate effectively with others in a timely manner. For the party receiving the translation, such as the tour guide, it can also better understand the needs of deaf-mute tourists and provide better services. At the same time, different transmission channels transmit gesture features and gesture images respectively, which improves the efficiency and stability of data transmission and ensures the smooth progress of the translation process.

[0099] In an embodiment of the present disclosure, the first translation instruction includes a first environment identifier.

[0100] In response to the first device being in the first translation mode, the method further includes: An environmental feature is determined based on the environmental information, and the environmental feature and the first environmental identifier are sent to the second device.

[0101] The environmental information includes environmental information around the first user, the first user is a user using the first device, and the first environmental identifier is used to instruct the second device to translate the gesture feature based on the environmental feature.

[0102] In this embodiment, the environmental information includes environmental images, environmental audio, and spatial distance. When the first device is in the first translation mode, in addition to extracting gesture features and sending them to the second device, the environmental features are determined based on the environmental information around the first user. The environmental features and the first translation instruction with the first environmental identifier are then sent to the second device. After receiving this information, the second device will translate the gesture features more accurately based on the environmental features. Because different environments may affect the specific meaning of sign language, the accuracy and adaptability of the translation can be improved by adding environmental features.

[0103] Exemplarily, the first device may be equipped with an environmental sensor, such as a light sensor, a sound sensor, a depth sensor, etc., for collecting surrounding environment information.

[0104] The second translation instruction includes a second environment identifier.

[0105] In response to the first device being in the second translation mode, the method further includes: The environment information and the second environment identifier are sent to the second device.

[0106] The second environment identifier is used to instruct the second device to translate the gesture feature based on the environment information.

[0107] In this embodiment, when the first device is in the second translation mode, environmental information including environmental images, environmental audio, and spatial distance, as well as a second translation instruction with a second environmental identifier, will be sent to the second device. After receiving this information, the second device translates the gesture image based on the environmental information. The environmental image can provide visual information of the surrounding scene, the environmental audio may contain surrounding sound clues, and the spatial distance can help determine the relative position relationship between the sign language speaker and the surrounding objects or people. These environmental factors help to understand the specific meaning of the sign language more accurately, making the translation more in line with the actual situation.

[0108] Exemplarily, the first device (such as smart glasses) needs to be equipped with a camera to acquire environmental images, a microphone to collect environmental audio, and a distance sensor to measure spatial distance.

[0109] At the train station, a deaf-mute passenger wears the first device (smart glasses). When the passenger asks the staff in sign language, the smart glasses are in the second translation mode. At this time, the smart glasses collect surrounding environmental information, such as the noisy audio of the train station, the environmental images of the crowd and facilities, and the spatial distance from the staff, and send this information and the second environmental identification to the second device (such as a tablet). Based on this environmental information, the second device more accurately translates the passenger's sign language into "Excuse me, which platform does my train stop at?", combined with the specific environment of the train station, so that the translation result is more in line with actual needs, making it easier for staff to provide accurate help to passengers.

[0110] This embodiment significantly improves the accuracy and adaptability of gesture translation by introducing environmental information and environmental identification. In the first translation mode, translation is performed in combination with environmental features to ensure that the translation result is more in line with the actual situation. In the second translation mode, multi-dimensional information such as environmental images, audio, and distance is used to further improve the accuracy of translation. This improvement is particularly suitable for complex and changeable actual environments, making it easier for deaf-mute people to communicate with others, improving communication efficiency and user experience.

[0111] Corresponding to the sign language translation method of the above embodiment, Figure 2 This is a structural block diagram of a sign language translation system provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 2 , the sign language translation system 20 comprises: The first translation module is used to extract gesture features based on the gesture image in response to the first device being in the first translation mode, and send the gesture features and the first translation instruction to the second device. The second device is a device connected to the first device. The first translation instruction is used to instruct the second device to translate the gesture features.

[0112] The second translation module is configured to send the gesture image and the second translation instruction to the second device in response to the first device being in the second translation mode. The second translation instruction is configured to instruct the second device to translate the gesture image.

[0113] In one embodiment of the present disclosure, the sign language translation system 20 further includes: The third translation module is used to determine the gesture category of each gesture based on the gesture image in response to the first device being in the third translation mode, where the gesture category includes a univocal gesture or a polyvocal gesture.

[0114] In response to the gesture category being a univocal gesture, extracting gesture features of the gesture image, sending the gesture features and a third translation instruction to the second device, and instructing the second device to translate the gesture features.

[0115] In response to the gesture category being an ambiguous gesture, the gesture image and a fourth translation instruction are sent to the second device. The fourth translation instruction is used to instruct the second device to translate the gesture image.

[0116] In an embodiment of the present disclosure, the first translation instruction includes a first environment identifier and a first translation module, and is further configured to determine environment characteristics based on environment information, and send the environment characteristics and the first environment identifier to the second device.

[0117] The environmental information includes environmental information around the first user, the first user is a user using the first device, and the first environmental identifier is used to instruct the second device to translate the gesture feature based on the environmental feature.

[0118] In an embodiment of the present disclosure, the second translation instruction includes a second environment identifier and a second translation module, and is further configured to send the environment information and the second environment identifier to a second device.

[0119] The environmental information includes environmental information around the first user, the first user is a user using the first device, and the second environmental identifier is used to instruct the second device to translate the gesture feature based on the environmental information.

[0120] In an embodiment of the present disclosure, the environmental information includes environmental images, environmental audio, and spatial distance.

[0121] In one embodiment of the present disclosure, the sign language translation system 20 further includes: The first trigger module is used to control the first device to be in a first translation mode in response to receiving a first trigger instruction.

[0122] In response to receiving the second trigger instruction, the first device is controlled to be in a second translation mode.

[0123] In one embodiment of the present disclosure, the sign language translation system 20 further includes: The second trigger module is used to control the first device to be in a third translation mode in response to receiving a third trigger instruction.

[0124] See also Figure 3 , Figure 3 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 3 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of the modules in the above-mentioned system embodiments, such as Figure 2 The functions of modules 21 to 22 are shown.

[0125] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0126] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0127] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0128] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of the sign language translation method provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device 300 described in the embodiments of the present disclosure, which will not be repeated here.

[0129] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0130] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0131] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0133] In the several embodiments provided in the present application, it should be understood that the disclosed electronic device and method can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0134] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0135] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0136] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A sign language translation method, characterized in that: Applied to a first device, comprising: In response to the first device being in a first translation mode, extracting gesture features based on the gesture image, and sending the gesture features and a first translation instruction to a second device; the second device is a device connected to the first device; and the first translation instruction is used to instruct the second device to translate the gesture features; In response to the first device being in the second translation mode, the gesture image and the second translation instruction are sent to the second device; the second translation instruction is used to instruct the second device to translate the gesture image.

2. The sign language translation method according to claim 1, characterized in that: Also includes: In response to the first device being in a third translation mode, determining a gesture category of each gesture based on the gesture image, the gesture category comprising a univocal gesture or a polyvocal gesture; In response to the gesture category being the univocal gesture, extracting gesture features of the gesture image; sending the gesture features and a third translation instruction to a second device; the third translation instruction is used to instruct the second device to translate the gesture features; In response to the gesture category being the ambiguous gesture, the gesture image and a fourth translation instruction are sent to a second device; the fourth translation instruction is used to instruct the second device to translate the gesture image.

3. The sign language translation method according to claim 1, characterized in that: The first translation instruction includes a first environment identifier; In response to the first device being in a first translation mode, the method further includes: Determine an environmental feature based on the environmental information, and send the environmental feature and the first environmental identifier to the second device; The environmental information includes environmental information around a first user, the first user is a user using the first device, and the first environmental identifier is used to instruct the second device to translate the gesture feature based on the environmental feature.

4. The sign language translation method according to claim 1, wherein: The second translation instruction includes a second environment identifier; In response to the first device being in the second translation mode, the method further includes: Sending the environment information and the second environment identifier to the second device; The environmental information includes environmental information around a first user, the first user is a user using the first device, and the second environmental identifier is used to instruct the second device to translate the gesture feature based on the environmental information.

5. The sign language translation method according to claim 3 or 4, characterized in that: The environmental information includes environmental images, environmental audio and spatial distance.

6. The sign language translation method according to claim 1, wherein: Also includes: In response to receiving a first trigger instruction, controlling the first device to be in a first translation mode; In response to receiving a second trigger instruction, controlling the first device to be in a second translation mode.

7. The sign language translation method according to claim 2, wherein: Also includes: In response to receiving a third trigger instruction, controlling the first device to be in a third translation mode.

8. A sign language translation system, characterized in that: Applied to a first device, comprising: a first translation module, configured to extract gesture features based on the gesture image in response to the first device being in a first translation mode, and send the gesture features and a first translation instruction to a second device; the second device is a device connected to the first device; the first translation instruction is used to instruct the second device to translate the gesture features; The second translation module is configured to send the gesture image and a second translation instruction to a second device in response to the first device being in a second translation mode; the second translation instruction is configured to instruct the second device to translate the gesture image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images