Model training method and device, electronic equipment and readable storage medium

Abstract

The application discloses a model training method and device, electronic equipment and a readable storage medium, and belongs to the field of artificial intelligence. The method comprises the following steps: acquiring a first text sequence; inputting the first text sequence and a sign language vocabulary group information table into a sign language grammar error detection model to output N processing labels, one processing label corresponding to one first sign language vocabulary, the processing label comprising a retention label indicating retention of a sign language vocabulary or a deletion label indicating deletion of a sign language vocabulary; the sign language vocabulary group information table comprises multiple groups of sign language vocabularies and historical co-occurrence numbers corresponding to each group of sign language vocabularies; based on the N processing labels, the first text sequence is processed to obtain a third text sequence; in the case that the third text sequence is different from a second text sequence, based on a target sign language vocabulary, the sign language grammar error detection model is trained, the target sign language vocabulary being a different sign language vocabulary in the third text sequence and the second text sequence.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence, and particularly relates to a model training method and device, electronic equipment and a readable storage medium. BACKGROUND

[0002] At present, due to the limitations of segmentation technology, in the sign language video segmentation stage, the electronic device will inevitably often cut out some video clips. When the electronic device inputs these sign language video clips into the sign language recognition module for recognition, redundant sign language vocabulary may be obtained, so that the sign language grammar of the sign language video corresponding to the sign language vocabulary is incorrect. In order to alleviate this phenomenon, the electronic device usually uses the rules configured in the model to process the redundant sign language vocabulary.

[0003] However, because the rules in the model are too single, when the electronic device encounters new redundant sign language vocabulary, the rules configured in the current model cannot timely identify the sign language grammar error in the sign language video where the sign language vocabulary is located and correct the error.

[0004] Therefore, the existing model has the problems of poor generalization and low detection accuracy when detecting the redundant grammar of sign language vocabulary. SUMMARY

[0005] The embodiments of the present application provide a model training method, device, electronic equipment and readable storage medium, which can timely identify the sign language grammar error in the sign language video and improve the detection accuracy and generalization of the model.

[0006] In a first aspect, the embodiments of the present application provide a model training method, which comprises: obtaining a first text sequence, the first text sequence comprising N first sign language vocabularies, the N first sign language vocabularies comprising at least one grammatically correct sign language vocabulary and at least one arbitrary sign language vocabulary in a second text sequence, N being an integer greater than 1; inputting the first text sequence and a sign language vocabulary group information table into a sign language grammar error detection model to output N processing labels, one processing label corresponding to one first sign language vocabulary, the processing label comprising a retention label indicating retention of a sign language vocabulary or a deletion label indicating deletion of a sign language vocabulary; the sign language vocabulary group information table comprising a plurality of sign language vocabulary groups and a historical co-occurrence number corresponding to each sign language vocabulary group; processing the first text sequence based on the N processing labels to obtain a third text sequence; in the case that the third text sequence is different from the second text sequence, training the sign language grammar error detection model based on a target sign language vocabulary, the target sign language vocabulary being a different sign language vocabulary in the third text sequence and the second text sequence.

[0007] In a second aspect, an embodiment of the present application provides a model training apparatus, the model training apparatus comprising: an obtaining module, a processing module, and a training module; the obtaining module is configured to obtain a first text sequence, the first text sequence comprising N first sign words, the N first sign words comprising at least one grammatically correct sign word and at least one arbitrary sign word in a second text sequence, N being an integer greater than 1; the processing module is configured to input the first text sequence obtained by the obtaining module and a sign word group information table into a sign grammar error detection model, and output N processing labels, one processing label corresponding to one first sign word, the processing label comprising a retention label indicating a retention sign word or a deletion label indicating a deletion sign word; the sign word group information table comprises a plurality of sign word groups and a historical co-occurrence number corresponding to each sign word group; the processing module is further configured to process the first text sequence based on the N processing labels to obtain a third text sequence; and the training module is configured to train the sign grammar error detection model based on a target sign word in a case where the third text sequence is different from the second text sequence, the target sign word being a different sign word in the third text sequence and the second text sequence.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions being executed by the processor to implement the steps of the method according to the first aspect.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, the readable storage medium storing a program or instructions, the program or instructions being executed by a processor to implement the steps of the method according to the first aspect.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, the chip comprising a processor and a communication interface, the communication interface being coupled to the processor, the processor being configured to run a program or instructions to implement the method according to the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, the program product being stored in a storage medium, the program product being executed by at least one processor to implement the method according to the first aspect.

[0012] In the embodiment of the present application, a first text sequence is obtained, the first text sequence includes N first sign language vocabularies, the N first sign language vocabularies include at least one grammatically correct sign language vocabulary and at least one arbitrary sign language vocabulary in a second text sequence, the second text sequence includes at least one grammatically correct sign language vocabulary, and N is an integer greater than 1; the first text sequence and a sign language vocabulary group information table are input into a sign language grammar error detection model, N processing labels are output, one processing label corresponds to one first sign language vocabulary, and the processing label includes a retention label indicating retention of a sign language vocabulary or a deletion label indicating deletion of a sign language vocabulary; the sign language vocabulary group information table includes multiple sign language vocabularies and a historical co-occurrence number corresponding to each sign language vocabulary; based on the N processing labels, the first text sequence is processed to obtain a third text sequence; in the case that the third text sequence is different from the second text sequence, the sign language grammar error detection model is trained based on a target sign language vocabulary, and the target sign language vocabulary is a different sign language vocabulary in the third text sequence and the second text sequence. In this way, since the first text sequence is composed of at least one grammatically correct sign language vocabulary and an arbitrary sign language vocabulary in the second text sequence, the text sequence training database containing only the grammatically correct second text sequence for training the sign language grammar error detection model can be expanded, so that the electronic device can train the sign language grammar error detection model by expanding the training database to improve the generalization of the sign language grammar error detection model. At the same time, the electronic device introduces a sign language vocabulary group information table, so that the sign language grammar error detection model can obtain the historical co-occurrence number corresponding to each sign language vocabulary according to the sign language vocabulary group information table, and accurately add a processing label to each sign language vocabulary according to the obtained historical co-occurrence number corresponding to each sign language vocabulary, so that the electronic device can delete or retain the sign language vocabulary of the first text sequence according to the processing label to obtain the third text sequence. Finally, the sign language grammar error detection model is trained and updated based on the different sign language vocabularies in the third text sequence and the second text sequence, i.e., the redundant sign language vocabulary detected by the sign language grammar error detection model, thereby improving the detection accuracy of the sign language grammar error detection model. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is one of the flowcharts of the model training method provided by the embodiment of the present application;

[0014] Figure 2 is another flowchart of the model training method provided by the embodiment of the present application;

[0015] Figure 3 is a schematic diagram of the sign language grammar error detection model provided by the embodiment of the present application;

[0016] Figure 4 is a third flowchart of the model training method provided by the embodiment of the present application;

[0017] Figure 5 This is the fourth flowchart illustrating a model training method provided in this application embodiment;

[0018] Figure 6 This is a schematic diagram illustrating the process of obtaining a sign language vocabulary group according to an embodiment of this application;

[0019] Figure 7 This is the fifth flowchart illustrating a model training method provided in this application embodiment;

[0020] Figure 8 This is a flowchart illustrating a sign language recognition model provided in an embodiment of this application;

[0021] Figure 9 This is one of the structural schematic diagrams of a model training device provided in the embodiments of this application;

[0022] Figure 10 This is a second schematic diagram of the structure of a model training device provided in an embodiment of this application;

[0023] Figure 11 This is one of the hardware structure diagrams of an electronic device provided in the embodiments of this application;

[0024] Figure 12 This is a second schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0026] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0027] The model training method, apparatus, electronic device, and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0028] Currently, electronic devices use a sign language segmentation module to cut user-input sign language videos into multiple video segments. These segments are then processed by a sign language recognition module to obtain the corresponding sign language vocabulary for each segment. However, due to limitations in segmentation technology, electronic devices inevitably cut out extra video segments during the segmentation stage. When these segments are input into the sign language recognition module, redundant sign language vocabulary may be obtained, leading to grammatical errors in the resulting sign language expressions. To mitigate this issue, electronic devices typically use pre-configured rules within a model to process the redundant sign language vocabulary.

[0029] However, due to the overly simplistic rules in the model, when electronic devices encounter new redundant sign language words, the currently configured rules cannot promptly identify and correct grammatical errors in the sign language video containing those words. Consequently, existing models suffer from poor generalization and low detection accuracy when detecting redundant sign language words.

[0030] In this embodiment, since the first text sequence is composed of at least one grammatically correct sign language word from the second text sequence and any other sign language word, the training database for training the sign language grammar error detection model, which only contains the grammatically correct second text sequence, can be expanded. This allows the electronic device to train the sign language grammar error detection model by expanding the training database, thereby improving the generalization of the model. Simultaneously, by introducing a sign language word group information table, the electronic device allows the sign language grammar error detection model to obtain the historical co-occurrence count of each sign language word. Based on this count, a processing label is accurately added to each sign language word. This allows the electronic device to delete or retain sign language words from the first text sequence based on the processing label, resulting in the third text sequence. Finally, based on the different sign language words in the third text sequence and the second text sequence—that is, the redundant sign language words detected—the sign language grammar error detection model is trained and updated.

[0031] The execution entity of the model training method provided in this embodiment can be a model training method device, which can be an electronic device, or a control module or processing module within the electronic device. The following description uses an electronic device as an example to illustrate the technical solution provided in this application embodiment.

[0032] This application provides a model training method. Figure 1 A flowchart illustrating a model training method provided in an embodiment of this application is shown, which can be applied to electronic devices. Figure 1 As shown, the model training method provided in this application embodiment may include the following steps 201 to 204.

[0033] Step 201: The electronic device acquires the first text sequence.

[0034] In this embodiment of the application, the first text sequence includes N first sign language words, where N is an integer greater than 1.

[0035] In this embodiment of the application, the aforementioned sign language vocabulary refers to the text vocabulary corresponding to sign language actions.

[0036] In this embodiment of the application, each sign language word in the first text sequence corresponds to a sign language action.

[0037] Example 1: The first text sequence above is “I|woman|child|high|salute”, which includes 5 sign language words.

[0038] In this embodiment of the application, the aforementioned N first sign language words include at least one grammatically correct sign language word in the second text sequence and at least one arbitrary sign language word.

[0039] It is understandable that the above-mentioned grammatically correct sign language vocabulary refers to the sign language vocabulary having the correct word order and semantics in the text sequence.

[0040] For example, the second text sequence described above may be obtained by the electronic device from a preset grammatically correct text sequence library.

[0041] For example, the first text sequence can be obtained directly from a preset text sequence library by the electronic device, or it can be selected from a preset grammatically correct text sequence library in the electronic device, and then an arbitrary sign language word is concatenated to the second text sequence.

[0042] Step 202: The electronic device inputs the first text sequence and the sign language vocabulary information table into the sign language grammar error detection model and outputs N processing labels.

[0043] In this embodiment, each of the N processing tags corresponds to a first sign language word in the first text sequence. That is, the N processing tags correspond one-to-one with the N sign language words in the first text sequence.

[0044] In this embodiment of the application, the processing label includes a retention label indicating the retention of sign language words or a deletion label indicating the deletion of sign language words.

[0045] Exemplarily, the above-mentioned retention tag can be represented by "O", and the above-mentioned deletion tag can be represented by "B-delete".

[0046] Exemplarily, when the processing tag corresponding to a sign language vocabulary is "O", it indicates that the electronic device needs to retain the sign language vocabulary; when the processing tag corresponding to a sign language vocabulary is "B-delete", it indicates that the electronic device needs to delete the sign language vocabulary.

[0047] In the embodiment of the present application, the above-mentioned sign language grammar error detection model is used to detect whether there are sign language vocabulary with grammar errors in the text sequence.

[0048] In the embodiment of the present application, the above-mentioned grammar errors may include any one of the following: redundancy of sign language vocabulary in the text sequence, incorrect word order sorting of sign language vocabulary in the text sequence, or semantic errors of sign language vocabulary in the text sequence.

[0049] In the embodiment of the present application, the above-mentioned sign language grammar error detection model includes: a feature layer and a label adding module.

[0050] Exemplarily, the above-mentioned feature layer is used to extract the feature information of each sign language vocabulary in the first text sequence.

[0051] Exemplarily, the above-mentioned label adding module is used to add a processing label to each of the above-mentioned sign language vocabulary.

[0052] Example 2, taking the first text sequence "I|female|child|tall|salute" as an example, if the processing tags output by the above-mentioned sign language grammar error detection model are "O|O|O|O|B-delete", it means retaining the sign language vocabulary "I|female|child|tall" and deleting the sign language vocabulary "salute".

[0053] In the embodiment of the present application, the above-mentioned sign language vocabulary group information table includes multiple groups of sign language vocabulary and the corresponding historical co-occurrence times of each group of sign language vocabulary.

[0054] In the embodiment of the present application, each group of sign language vocabulary in the above-mentioned multiple groups of sign language vocabulary contains multiple sign language vocabulary.

[0055] It should be noted that the above-mentioned sign language vocabulary refers to multiple sign language vocabulary that may coexist and be easily confused in a sign language video segment. Therefore, the above-mentioned sign language vocabulary group information table can also be called a confusion co-occurrence information table, and the above-mentioned sign language vocabulary group can also be called a confusion co-occurrence vocabulary group, where the confusion co-occurrence vocabulary group includes multiple confusion co-occurrence vocabulary.

[0056] In this embodiment, the historical co-occurrence count for each group of confusing co-occurring words refers to the number of times that group of confusing co-occurring words appears in the text sequence corresponding to the historical sign language video. Generally, the higher the historical co-occurrence count for a group of confusing co-occurring words, the more likely the text sequence corresponding to that group of confusing co-occurring words is to contain grammatical errors of sign language vocabulary redundancy; conversely, the lower the historical co-occurrence count for a group of confusing co-occurring words, the less likely the text sequence corresponding to that group of confusing co-occurring words is to contain grammatical errors of sign language vocabulary redundancy.

[0057] Optionally, in this embodiment, the electronic device inputs the concatenated first text sequence and a sign language vocabulary group information table into a sign language grammar error detection model. This allows the model to obtain the historical co-occurrence count of each sign language vocabulary group in the first text sequence within the corresponding sign language information, based on the input sign language vocabulary group information table. Then, based on the historical co-occurrence count of each sign language vocabulary, the electronic device determines whether to retain or delete each vocabulary. Finally, the electronic device labels each sign language vocabulary with a corresponding processing tag based on the determination result, and outputs the processing tag for each sign language vocabulary.

[0058] Step 203: The electronic device processes the first text sequence based on N processing tags to obtain the third text sequence.

[0059] In this embodiment of the application, the aforementioned third text sequence is a text sequence obtained by the electronic device after retaining or deleting each sign language word in the first text sequence according to the processing tag.

[0060] Example 3, combined with Example 2, takes the first text sequence as “I|Woman|Child|High|Salute” as an example. According to the output processing label “O|O|O|O|B-delete”, the electronic device deletes the sign language word “Salute”, and finally obtains the third text sequence. At this time, the third text sequence includes the four sign language words “I|Woman|Child|High”.

[0061] Optionally, in this embodiment of the application, the electronic device processes each sign language word in the first text sequence according to the sequence of each sign language word and the processing tag corresponding to each sign language word, so as to obtain a third text sequence. For example, if a sign language word in the first text sequence has a retention tag, the sign language word is retained; if the processing tag corresponding to the sign language word is a deletion tag, the sign language word is deleted.

[0062] Step 204: When the third text sequence differs from the second text sequence, the electronic device trains a sign language grammar error detection model based on the target sign language vocabulary.

[0063] In this embodiment of the application, the target sign language vocabulary is different sign language vocabulary in the third text sequence and the second text sequence.

[0064] Optionally, in the embodiments of this application, the phrase "the third text sequence is different from the second text sequence" indicates that the sign language vocabulary in the third text sequence is partially or completely different from the sign language vocabulary in the second text sequence. It is understood that when the third text sequence is different from the second text sequence, it indicates that there are sign language vocabulary words with incorrect sign language grammar in the third text sequence.

[0065] Optionally, in this embodiment of the application, the electronic device can calculate the cross-entropy function based on the target sign language vocabulary, and optimize and train the sign language grammar error detection model based on the cross-entropy function to obtain the trained sign language grammar error detection model.

[0066] In the model training method provided in this application embodiment, since the first text sequence is composed of at least one grammatically correct sign language word from the second text sequence and any other sign language word, the training database of the sign language grammar error detection model, which only contains the grammatically correct second text sequence, can be expanded. This allows the electronic device to train the sign language grammar error detection model by expanding the training database, thereby improving the generalization of the sign language grammar error detection model. Simultaneously, by introducing a sign language word group information table, the electronic device can obtain the historical co-occurrence count of each sign language word based on the information table. Based on this count, a processing label is accurately added to each sign language word, allowing the electronic device to delete or retain sign language words from the first text sequence based on the processing label, thus obtaining the third text sequence. Finally, based on the different sign language words in the third text sequence and the second text sequence—that is, the sign language grammar error detection model trains and updates the model for the detected redundant sign language words.

[0067] Optionally, in the embodiments of this application, combined with Figure 1 ,like Figure 2 As shown, step 202 specifically includes steps 202a to 202c:

[0068] Step 202a: The electronic device inputs the first text sequence and the sign language vocabulary group information table into the sign language grammar error detection model to obtain the historical co-occurrence count of each first sign language word to which the sign language vocabulary group belongs.

[0069] For example, for each sign language word in the first sign language vocabulary, the electronic device queries all the sign language vocabulary groups to which the sign language word belongs from the aforementioned sign language vocabulary group information table, and adds up the historical co-occurrence counts corresponding to all the sign language vocabulary groups to obtain the historical co-occurrence counts corresponding to the sign language vocabulary group to which the sign language word belongs.

[0070] It should be noted that electronic devices can refer to step 202a above to obtain the historical co-occurrence count for each first sign language word.

[0071] Step 202b: If the second sign language vocabulary is a preset sign language vocabulary, and the sign language grammar of the second sign language vocabulary is redundant in the first text sequence, then the deletion label will be used as the processing label corresponding to the second sign language vocabulary; if the sign language grammar of the second sign language vocabulary is not redundant in the first text sequence, then the retention label will be used as the processing label corresponding to the second sign language vocabulary.

[0072] In this embodiment of the application, the aforementioned second sign language vocabulary is one of N first sign language vocabulary.

[0073] In this embodiment, the historical co-occurrence frequency of the sign language vocabulary group to which the aforementioned preset sign language vocabulary belongs is greater than a first threshold. It is understood that the aforementioned preset sign language vocabulary refers to sign language vocabulary prone to grammatical errors.

[0074] For example, the electronic device acquires the feature information of the second sign language vocabulary, combines it with the semantic information in the first text sequence where the second sign language vocabulary is located, and determines whether the sign language grammar of the second sign language vocabulary in the first text sequence is redundant. If it is not redundant, the retained label is used as the processing label corresponding to the second sign language vocabulary; if it is redundant, the deleted label is used as the processing label corresponding to the second sign language vocabulary.

[0075] Step 202c: If the electronic device does not use the preset sign language vocabulary for the second sign language vocabulary, it will retain the tag as the processing tag corresponding to the second sign language vocabulary.

[0076] In one possible embodiment, the electronic device concatenates the first text sequence and the historical co-occurrence counts of the sign language vocabulary group to which each sign language word in the first text sequence belongs, to obtain a first feature sequence. The electronic device inputs this first feature sequence into a label adding module and outputs a processed label corresponding to each sign language word in the first text sequence.

[0077] For example, consider the sign language sequence "I|Woman|Child|High," which includes four sign language words. The electronic device sequentially queries the historical co-occurrence counts of the four sign language words' corresponding confusion co-occurrence word groups in the confusion co-occurrence information table. Since the sign language words "I," "Woman," and "Child" do not belong to any confusion co-occurrence word groups in the table, their historical co-occurrence counts are recorded as 0. However, the sign language word "High" belongs to multiple confusion co-occurrence word groups in the table, such as 22 times for "Salute|High|Grandson," 20 times for "Salute|Grandson|High," and 13 times for "High|Grandson|Salute." Therefore, the historical co-occurrence counts of "High" are added together and recorded as 55. Finally, the electronic device concatenates the sign language sequence and the historical co-occurrence counts for each sign language word to obtain "I|Woman|Child|High|0|0|0|55," which is the first feature sequence mentioned above.

[0078] For example, the label-adding module described above can be a Named Entity Recognition (Bilstm-CRF) model.

[0079] For example, the Bilstm-CRF model described above includes Bilstm layers and CRF.

[0080] For example, such as Figure 3 As shown, the electronic device inputs the first text sequence and the confusion co-occurrence information table into the feature layer. The feature layer concatenates the first text sequence and the historical co-occurrence frequency feature information of the confusion co-occurrence word group to which each sign language word in the first text sequence belongs, to obtain the first feature sequence. Then, the first feature sequence is input into the Bilstm layer in the Bilstm-CRF model. The Bilstm layer extracts and encodes the information from the first feature sequence to obtain the first feature information, which is the feature information of the aforementioned second sign language word. Next, the electronic device inputs the encoded first feature information into the CRF layer. The CRF layer combines the position of the second sign language word in the first text sequence with the sign language words before and after the second sign language word to perform annotation and classification, and finally generates the processing label corresponding to each word in the first text sequence.

[0081] In this way, electronic devices can use the tag-adding module to identify easily confused co-occurring words in sign language text sequences. Then, based on the specific location of the sign language word and its grammatical issues in the text sequence, the accuracy of the sign language word can be determined, thereby improving the accuracy of the model's detection.

[0082] Optionally, in the embodiments of this application, combined with Figure 1 ,like Figure 4 As shown, step 204 above specifically includes steps 204a and 204b:

[0083] Step 204a: The electronic device calculates the cross-entropy loss function based on the historical co-occurrence frequency of the target sign language word to which it belongs and the position of the target sign language word in the first text sequence.

[0084] Step 204b: The electronic device trains a sign language grammar error detection model based on the cross-entropy loss function.

[0085] In this way, electronic devices improve the accuracy of the sign language grammar error detection model by continuously updating the cross-entropy loss function to train the model.

[0086] Optionally, in this embodiment of the application, after obtaining the sign language text sequence, the electronic device will input the sign language text sequence into the sign language grammar error detection (GED) model and output the processing label of each sign language word in the sign language text sequence, so that the electronic device can perform corresponding processing on each sign language word in the sign language text sequence according to the processing label.

[0087] Optionally, in this embodiment of the application, the electronic device needs to build a GED model before using the GED model.

[0088] Optionally, in this embodiment of the application, the construction of the GED model includes two stages: an offline stage and an online stage.

[0089] For example, the aforementioned offline phase specifically includes: mining prior business information, constructing and training the GED model.

[0090] For example, the aforementioned prior information for mining operations includes: a confusion co-occurrence information table, namely the aforementioned sign language vocabulary group information table, and a sign language vocabulary duration database.

[0091] Optionally, in this embodiment of the application, the electronic device can construct the above-mentioned confusion co-occurrence information table by analyzing multiple historical sign language videos.

[0092] Optionally, in the embodiments of this application, combined with Figure 1 ,like Figure 5 As shown, prior to step 202 above, the model training method provided in this application embodiment further includes steps 301 to 303:

[0093] Step 301: The electronic device divides the first historical sign language video into at least two sign language video segments.

[0094] In the embodiments of this application, each of the above-mentioned at least two sign language video clips corresponds to a set of sign language words.

[0095] In the embodiments of this application, each of the above-mentioned at least two sign language video clips corresponds to a sign language action.

[0096] In this embodiment of the application, the electronic device segments the first historical sign language video into at least two sign language video segments by inputting the first historical sign language video into the sign language segmentation module.

[0097] For example, the sign language segmentation module described above can be a temporal action video segmentation model.

[0098] For example, the sign language segmentation module described above can be a sign language segmentation module included in a sign language recognition model.

[0099] Step 302: The electronic device obtains the first value corresponding to each sign language word in the first sign language vocabulary group.

[0100] In this embodiment of the application, the first value corresponding to any sign language word in the first sign language vocabulary group is used to characterize the similarity between any sign language word and the sign language action corresponding to the first sign language vocabulary group.

[0101] In this embodiment of the application, the first sign language vocabulary group may include at least two sign language words.

[0102] For example, an electronic device can group at least two sign language words that meet preset conditions from all the sign language words corresponding to a sign language video segment into a sign language word group.

[0103] For example, the above preset conditions may include at least one of the following: the first value of the sign language words corresponding to a sign language segment is greater than the fourth threshold, and the first value of the sign language words corresponding to a sign language segment is among the top three of the first values ​​of all sign language words.

[0104] Step 303: If a second sign language vocabulary group exists in at least two sign language video segments, the electronic device adds the first sign language vocabulary group to the sign language vocabulary group information table.

[0105] In this embodiment of the application, the maximum value of the first value corresponding to each sign language word in the second sign language vocabulary group is less than the second threshold, and the minimum value is greater than the third threshold.

[0106] Example 4, such as Figure 6As shown, taking the sign language sequence "woman|child|high" in a historical sign language video as an example, the electronic device divides the historical sign language video into three sign language video segments. It calculates the first value of multiple sign language words corresponding to each segment, and records the top three first values ​​as a group of confused co-occurring words, i.e., the aforementioned sign language word groups. The confused co-occurring word group corresponding to the first sign language video segment is "woman|difficult|think," where the first value of "woman" is 0.9, the first value of "difficult" is 0.09, and the first value of "think" is 0.01. The confused co-occurring word group corresponding to the second sign language video segment is "child|what|where," where the first value of "child" is 0.9, the first value of "what|" is 0.07, and the first value of "where" is 0.03. The confused co-occurring word group corresponding to the third sign language video segment is "salute|high|grandson," where the first value of "salute" is 0.6, the first value of "high" is 0.4, and the first value of "grandson" is 0.3. If the second threshold is set to 0.7 and the third threshold is set to 0.2, then since the first value of "salute" in the confusing co-occurrence vocabulary group of the third sign language video clip is less than 0.7 and the first value of "grandson" is greater than 0.2, the confusing co-occurrence vocabulary group of the third sign language video clip is added to the confusing co-occurrence vocabulary list, and the historical co-occurrence count is recorded as 1 time.

[0107] It should be noted that the electronic device can first acquire a large number of historical sign language videos, and then, for each historical sign language video, refer to the process of acquiring sign language vocabulary groups in steps 301 to 302 above to acquire the sign language vocabulary groups in each sign language video.

[0108] Optionally, in this embodiment of the application, the electronic device can construct a sign language vocabulary duration database by analyzing the historical sign language video set in the prior information of the business.

[0109] Optionally, in this embodiment of the application, before step 202 above, the model training method provided in this embodiment of the application may further include the following steps 401 to 402:

[0110] Step 401: The electronic device acquires X sign language video clips of any sign language word.

[0111] In this embodiment of the application, each of the X sign language video clips corresponds to the sign language action for any sign language word.

[0112] For example, the electronic device extracts X sign language video clips corresponding to the sign language actions of any sign language word from a large number of historical sign language video clips.

[0113] Step 402: The electronic device calculates the average duration of X sign language video clips and uses the average duration as the preset sign language action duration for any sign language word, adding it to the sign language word duration library.

[0114] In this embodiment of the application, the electronic device obtains the duration of each sign language video segment from X sign language video clips, and then calculates the average duration of the X sign language video clips.

[0115] For example, if the fourth sign language word is "child", and the average duration of the sign language segment corresponding to the sign language actions of 50 children is 1 second, then the preset duration of the sign language word "child" is 1 second.

[0116] In this embodiment of the application, the above-mentioned sign language vocabulary duration library includes at least one sign language vocabulary and a preset sign language action duration corresponding to each sign language vocabulary.

[0117] In this way, electronic devices can increase the feature parameter information in the model by mining business prior information and using the GED model based on the business prior information, thereby improving the efficiency and accuracy of model detection.

[0118] Optionally, in the embodiments of this application, combined with Figure 1 ,like Figure 7 As shown, after step 204 above, the model training method provided in this application embodiment further includes steps 501 to 503:

[0119] Step 501: The electronic device acquires the first sign language video corresponding to the first text sequence and determines the actual sign language action duration corresponding to each first sign language word.

[0120] In this embodiment of the application, the actual sign language action duration is used to characterize the actual sign language action duration of the sign language words in the corresponding sign language video.

[0121] Optionally, in this embodiment of the application, the electronic device acquires a first sign language video corresponding to a first text sequence, segments the first sign language video according to the sign language action corresponding to each sign language word in the first sign language video, and obtains N sign language video segments. Then, the electronic device determines the duration of the time sign language action corresponding to each first sign language word by acquiring the duration of each sign language video segment.

[0122] Step 502: If the time difference between the actual sign language action duration corresponding to the third sign language word and the preset sign language action duration corresponding to the third sign language word is greater than or equal to the fourth threshold, the electronic device determines that the processing tag corresponding to the third sign language word is a deletion tag.

[0123] Step 503: If the time difference between the actual sign language action duration corresponding to the third sign language word and the preset sign language action duration corresponding to the third sign language word is less than the fourth threshold, the electronic device determines the processing label corresponding to the third sign language word as the reserved label.

[0124] In this embodiment of the application, the aforementioned third sign language vocabulary is one of N first sign language vocabulary.

[0125] Optionally, in this embodiment of the application, the electronic device can determine the preset sign language action duration corresponding to each first sign language word by querying a sign language word duration database.

[0126] It should be noted that the acquisition of the preset sign language action duration for each of the N first sign language words is based on the acquisition method for the preset sign language action duration for the third sign language word.

[0127] In one possible embodiment, if the time difference between the actual sign language action duration corresponding to the third sign language word and the preset sign language action duration corresponding to the third sign language word is greater than or equal to the fourth threshold, it indicates that the sign language action of the third sign language word in the first sign language video is inconsistent with the preset sign language action corresponding to the third sign language word. In other words, it can be considered that they are not the same sign language word. At this time, the processing tag corresponding to the third sign language word should be the deletion tag.

[0128] In this embodiment of the application, if the processing label corresponding to the third sign language vocabulary is a deletion label, then the processing label is determined to be correct; if the processing label corresponding to the third sign language vocabulary is a retention label, then the processing label is determined to be incorrect, and the retention label is corrected and modified to a deletion label.

[0129] In another possible embodiment, if the time difference between the actual sign language action duration corresponding to the third sign language word and the preset sign language action duration corresponding to the third sign language word is less than the fourth threshold, it indicates that the sign language action of the third sign language word in the first sign language video is consistent with the preset sign language action corresponding to the third sign language word, and can be considered as the same sign language word. In this case, the processing tag corresponding to the sign language word should be the reserved tag.

[0130] In this embodiment of the application, if the processing label corresponding to the third sign language vocabulary is a retention label, then the processing label is determined to be correct; if the processing label corresponding to the third sign language vocabulary is a deletion label, then the processing label is determined to be incorrect, and the retention label is corrected and modified to a retention label.

[0131] In this way, electronic devices can further determine whether a sign language word is correct based on the duration of the corresponding sign language action, thereby improving the accuracy of detecting grammatical errors.

[0132] Optionally, in this embodiment of the application, before detecting whether the first text sequence is correct, the electronic device needs to acquire a sign language video, input the sign language video into the sign language recognition model, and output the aforementioned second text sequence.

[0133] In this embodiment of the application, the sign language recognition model includes a sign language segmentation module and a sign language recognition module.

[0134] For example, the sign language segmentation module described above is used to segment a sign language video into multiple sign language video segments.

[0135] For example, the sign language segmentation module described above can be a temporal action video segmentation model.

[0136] For example, the sign language recognition module described above is used to identify the sign language words corresponding to sign language actions in a sign language video clip.

[0137] For example, the sign language recognition module described above can be a discrete sign language recognition model, such as a 3D motion recognition model (poseX3D).

[0138] In one possible embodiment, such as Figure 8 As shown, this application embodiment provides a method for recognizing sign language text sequences in sign language videos, specifically including the following process:

[0139] Step A1: The electronic device acquires the first sign language video.

[0140] For example, a user can speak a sign language phrase into the camera in real time, and all video frames corresponding to this sign language phrase can be used as the first sign language video mentioned above and input into the sign language recognition model. For instance, the sign language text sequence "I|woman|child|high" corresponding to the first sign language video is denoted as the original sign language sequence (OriginGlossSet), which contains 5 sign language words.

[0141] Step A2: The electronic device inputs the first sign language video into the sign language segmentation module, which then segments the first sign language video into N sign language video segments.

[0142] Step A3: The electronic device obtains the online timestamp corresponding to each sign language video clip from the N sign language video clips.

[0143] For example, after step A2 above, the electronic device obtains a set of N sign language video clips, denoted as VideoSet1, and a set of timestamps denoted as TimeSet1. TimeSet1 includes: the timestamp corresponding to each sign language video clip, where each tuple in each timestamp represents the timestamp when the sign language video clip starts playing and the timestamp when it ends playing. For example, if the timestamp corresponding to the first sign language video clip is (0s, 0.6s), it means that the first sign language video clip starts playing from 0s and ends at 0.6s, that is, the duration of the sign language action in the first sign language video clip is 0.6s.

[0144] It should be noted that the timestamps corresponding to each of the above sign language video segments are obtained synchronously when the sign language segmentation module segments the first sign language video.

[0145] Step A4: The electronic device inputs N sign language video clips into the sign language recognition module.

[0146] For example, the electronic device sequentially inputs each of the N sign language video clips in VideoSet1 into the sign language recognition module for recognition, and outputs the sign language vocabulary (gloss) corresponding to the sign language actions in each sign language video clip.

[0147] Step A5: The electronic device assembles the sign language words corresponding to the sign language actions in each acquired sign language video segment into a sign language text sequence, i.e., a gloss sequence.

[0148] For example, after step A4 above, the electronic device obtains N glosses after recognizing N sign language video segments, and the gloss sequence composed of N glosses is denoted as GlossSet1. For example, GlossSet1 is "I|Woman|Child|High|Salute".

[0149] In this way, electronic devices can extract the sign language text sequence from the sign language video input by the user through the sign language recognition model, thereby facilitating the subsequent processing of the sign language translation by the electronic devices.

[0150] It should be noted that the model training method provided in this application can be executed by a model training device, an electronic device, or a functional module or entity within an electronic device. This application uses a model training device to perform model training as an example to illustrate the model training apparatus provided in this application.

[0151] Figure 9 A schematic diagram of a possible structure of the model training apparatus involved in an embodiment of this application is shown. For example... Figure 9 As shown, the model training device 700 may include: an acquisition module 701, a processing module 702, and a training module 703;

[0152] The acquisition module 701 is used to acquire a first text sequence, which includes N first sign language words. The N first sign language words include at least one grammatically correct sign language word and at least one arbitrary sign language word from the second text sequence, where N is an integer greater than 1. The processing module 702 is used to input the first text sequence and the sign language word group information table acquired by the acquisition module 701 into the sign language grammar error detection model and output N processing labels. Each processing label corresponds to one first sign language word. The processing labels include a retention label indicating that the sign language word should be retained or a deletion label indicating that the sign language word should be deleted. The sign language word group information table includes multiple groups of sign language words and the historical co-occurrence counts corresponding to each group of sign language words. The processing module 702 is also used to process the first text sequence based on the N processing labels to obtain a third text sequence. The training module 703 is used to train the sign language grammar error detection model based on target sign language words when the third text sequence is different from the second text sequence. The target sign language words are the different sign language words in the third text sequence and the second text sequence.

[0153] Optionally, in this embodiment, the processing module 702 is specifically used to: input the first text sequence and the sign language vocabulary group information table into the sign language grammar error detection model, and obtain the historical co-occurrence count corresponding to the sign language vocabulary group to which each first sign language word belongs; if the second sign language word is a preset sign language word, and the sign language grammar of the second sign language word is redundant in the first text sequence, then the deletion label is used as the processing label corresponding to the second sign language word; if the sign language grammar of the second sign language word is not redundant in the first text sequence, then the retention label is used as the processing label corresponding to the second sign language word; the historical co-occurrence count corresponding to the sign language vocabulary group to which the preset sign language word belongs is greater than a first threshold; if the second sign language word is not a preset sign language word, then the retention label is used as the processing label corresponding to the second sign language word; wherein, the second sign language word is one of N first sign language words.

[0154] Optionally, in this embodiment, the training module 703 is specifically used to: calculate the cross-entropy loss function based on the historical co-occurrence count of the target sign language word group and the position of the target sign language word in the first text sequence; and train a sign language grammar error detection model based on the cross-entropy loss function.

[0155] Optionally, in the embodiments of this application, combined with Figure 9 ,like Figure 10As shown, the device 700 further includes a segmentation module 704 and an addition module 705. The segmentation module 704 is used to divide the first historical sign language video into at least two sign language video segments before the processing module 702 inputs the first text sequence and sign language vocabulary group information table obtained by the acquisition module 701 into the sign language grammar error detection model and outputs N processing labels. Each sign language video segment corresponds to a set of sign language vocabulary, and each sign language video segment corresponds to a sign language action. The acquisition module 701 is also used to acquire a first value corresponding to each sign language vocabulary in the first sign language vocabulary group. The first value is used to characterize the similarity between any sign language vocabulary and the sign language action corresponding to the first sign language vocabulary group. The addition module 705 is used to add the second sign language vocabulary group to the sign language vocabulary group information table when there is a second sign language vocabulary group in the sign language vocabulary groups corresponding to the at least two sign language video segments segmented by the segmentation module 704. The maximum value of all the first values ​​corresponding to the second sign language vocabulary group is less than a second threshold, and the minimum value is greater than a third threshold.

[0156] Optionally, in this embodiment, the acquisition module 701 is further configured to acquire the first sign language video corresponding to the first text sequence and determine the actual sign language action duration corresponding to each first sign language word before the processing module 702 performs processing on the second training sample based on the first label to obtain the target training sample; the processing module 702 is further configured to determine the processing label corresponding to the third sign language word as a deletion label when the time difference between the actual sign language action duration corresponding to the third sign language word and the preset sign language action duration corresponding to the third sign language word is greater than or equal to a fourth threshold; the processing module 702 is further configured to determine the processing label corresponding to the third sign language word as a retention label when the time difference is less than the fourth threshold; wherein, the third sign language word is one of N first sign language words.

[0157] In the model training apparatus provided in this application embodiment, the apparatus acquires a first text sequence, which includes N first sign language words. The N first sign language words include at least one grammatically correct sign language word and at least one arbitrary sign language word from a second text sequence, where N is an integer greater than 1. The first text sequence and a sign language word group information table are input into a sign language grammar error detection model, which outputs N processing labels. Each processing label corresponds to one first sign language word. The processing labels include a retain label indicating that the sign language word should be retained or a delete label indicating that the sign language word should be deleted. The sign language word group information table includes multiple groups of sign language words and the historical co-occurrence counts corresponding to each group of sign language words. Based on the N processing labels, the first text sequence is processed to obtain a third text sequence. If the third text sequence is different from the second text sequence, the sign language grammar error detection model is trained based on a target sign language word, which is a different sign language word in the third text sequence and the second text sequence. Thus, since the first text sequence consists of at least one grammatically correct sign language word from the second text sequence and any other sign language word, the training database for training the sign language grammar error detection model can be expanded to include only the text sequences containing the grammatically correct second text sequence. This allows the electronic device to train the sign language grammar error detection model by expanding the training database, thereby improving the model's generalization ability. Simultaneously, by introducing a sign language vocabulary group information table, the electronic device can obtain the historical co-occurrence count of each sign language word. Based on this information, the model accurately adds a processing label to each sign language word, allowing it to delete or retain sign language words from the first text sequence based on these labels, resulting in the third text sequence. Finally, based on the different sign language words in the third text sequence compared to the second text sequence—that is, the redundant sign language words detected—the sign language grammar error detection model is trained and updated.

[0158] The model training device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0159] The model training device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0160] The model training device provided in this application embodiment can achieve... Figures 1 to 8 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0161] Optionally, such as Figure 11 As shown, this application embodiment also provides an electronic device 800, including a processor 801 and a memory 802. The memory 802 stores a program or instructions that can run on the processor 801. When the program or instructions are executed by the processor 801, they implement the various steps of the above-described model training method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0162] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0163] Figure 12 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0164] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0165] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 12 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0166] The processor 110 is configured to acquire a first text sequence, which includes N first sign language words, including at least one grammatically correct sign language word and at least one arbitrary sign language word from the second text sequence, where N is an integer greater than 1; the processor 110 is further configured to input the first text sequence and a sign language word group information table into a sign language grammar error detection model, and output N processing labels, each processing label corresponding to one first sign language word, including a retention label indicating the retention of the sign language word or a deletion label indicating the deletion of the sign language word; the sign language word group information table includes multiple groups of sign language words and the historical co-occurrence counts corresponding to each group of sign language words; the processor 110 is further configured to process the first text sequence based on the N processing labels to obtain a third text sequence; the processor 110 is further configured to train a sign language grammar error detection model based on a target sign language word when the third text sequence is different from the second text sequence, where the target sign language word is a different sign language word in the third text sequence and the second text sequence.

[0167] Optionally, in this embodiment, the processor 110 is specifically configured to: input the first text sequence and the sign language vocabulary group information table into the sign language grammar error detection model, and obtain the historical co-occurrence count corresponding to the sign language vocabulary group to which each first sign language word belongs; if the second sign language word is a preset sign language word, and the sign language grammar of the second sign language word is redundant in the first text sequence, then the deletion label is used as the processing label corresponding to the second sign language word; if the sign language grammar of the second sign language word is not redundant in the first text sequence, then the retention label is used as the processing label corresponding to the second sign language word; the historical co-occurrence count corresponding to the sign language vocabulary group to which the preset sign language word belongs is greater than a first threshold; if the second sign language word is not a preset sign language word, then the retention label is used as the processing label corresponding to the second sign language word; wherein, the second sign language word is one of N first sign language words.

[0168] Optionally, in this embodiment of the application, the processor 110 is specifically used to: calculate the cross-entropy loss function based on the historical co-occurrence frequency of the target sign language word to which it belongs and the position of the target sign language word in the first text sequence; and train a sign language grammar error detection model based on the cross-entropy loss function.

[0169] Optionally, in this embodiment, the processor 110 is further configured to, before inputting the first text sequence and the sign language vocabulary group information table into the sign language grammar error detection model and outputting N processing labels, divide the first historical sign language video into at least two sign language video segments, each sign language video segment corresponding to a set of sign language vocabulary and each sign language video segment corresponding to a sign language action; the processor 110 is further configured to obtain a first value corresponding to each sign language vocabulary in the first sign language vocabulary group, the first value being used to characterize the similarity between any sign language vocabulary and the sign language action corresponding to the first sign language vocabulary group; the processor 110 is further configured to, if there is a second sign language vocabulary group in the sign language vocabulary groups corresponding to the at least two sign language video segments, add the second sign language vocabulary group to the sign language vocabulary group information table; wherein, the maximum value among all the first values ​​corresponding to the second sign language vocabulary group is less than a second threshold, and the minimum value is greater than a third threshold.

[0170] Optionally, in this embodiment, the processor 110 is further configured to, before processing the second training sample based on the first label to obtain the target training sample, acquire the first sign language video corresponding to the first text sequence and determine the actual sign language action duration corresponding to each first sign language word; the processor 110 is further configured to, when the time difference between the actual sign language action duration corresponding to the third sign language word and the preset sign language action duration corresponding to the third sign language word is greater than or equal to a fourth threshold, determine the processing label corresponding to the third sign language word as a deletion label; the processor 110 is further configured to, when the time difference is less than the fourth threshold, determine the processing label corresponding to the third sign language word as a retention label; wherein, the third sign language word is one of N first sign language words.

[0171] In the electronic device provided in this application embodiment, the electronic device acquires a first text sequence, which includes N first sign language words. The N first sign language words include at least one grammatically correct sign language word and at least one arbitrary sign language word from a second text sequence, where N is an integer greater than 1. The first text sequence and a sign language word group information table are input into a sign language grammar error detection model, which outputs N processing labels. Each processing label corresponds to one first sign language word. The processing labels include a retention label indicating that the sign language word should be retained or a deletion label indicating that the sign language word should be deleted. The sign language word group information table includes multiple groups of sign language words and the historical co-occurrence counts corresponding to each group of sign language words. Based on the N processing labels, the first text sequence is processed to obtain a third text sequence. If the third text sequence is different from the second text sequence, a sign language grammar error detection model is trained based on a target sign language word, where the target sign language word is a different sign language word in the third text sequence and the second text sequence. Thus, since the first text sequence consists of at least one grammatically correct sign language word from the second text sequence and any other sign language word, the training database for training the sign language grammar error detection model can be expanded to include only the text sequences containing the grammatically correct second text sequence. This allows the electronic device to train the sign language grammar error detection model by expanding the training database, thereby improving the model's generalization ability. Simultaneously, by introducing a sign language vocabulary group information table, the electronic device can obtain the historical co-occurrence count of each sign language word. Based on this information, the model accurately adds a processing label to each sign language word, allowing it to delete or retain sign language words from the first text sequence based on these labels, resulting in the third text sequence. Finally, based on the different sign language words in the third text sequence compared to the second text sequence—that is, the redundant sign language words detected—the sign language grammar error detection model is trained and updated.

[0172] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0173] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0174] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0175] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described model training method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0176] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0177] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described model training method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0178] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0179] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the model training method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0180] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0181] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0182] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A model training method, characterized in that, The method comprises: acquiring a first text sequence, the first text sequence comprising N first sign language words, the N first sign language words comprising at least one grammatically correct sign language word and at least one arbitrary sign language word in a second text sequence, N being an integer greater than 1; inputting the first text sequence and a sign language word group information table into a sign language grammar error detection model to output N processing labels, one processing label corresponding to one first sign language word, the processing label comprising a retention label indicating retention of a sign language word or a deletion label indicating deletion of a sign language word; the sign language word group information table comprising multiple groups of sign language words and a historical co-occurrence number corresponding to each group of sign language words; processing the first text sequence based on the N processing labels to obtain a third text sequence; in the case where the third text sequence is different from the second text sequence, training the sign language grammar error detection model based on a target sign language word, the target sign language word being a different sign language word in the third text sequence and the second text sequence.

2. The method of claim 1, wherein, The method comprises: inputting the first text sequence and a sign language word group information table into a sign language grammar error detection model to obtain a historical co-occurrence number corresponding to a sign language word group to which each first sign language word belongs; in the case where the second sign language word is a preset sign language word, if the second sign language word is grammatically redundant in the first text sequence, the deletion label is taken as the processing label corresponding to the second sign language word; if the second sign language word is not grammatically redundant in the first text sequence, the retention label is taken as the processing label corresponding to the second sign language word; the historical co-occurrence number corresponding to the sign language word group to which the preset sign language word belongs is greater than a first threshold value; in the case where the second sign language word is not a preset sign language word, the retention label is taken as the processing label corresponding to the second sign language word; wherein the second sign language word is one of the N first sign language words.

3. The method of claim 1, wherein, The method comprises: calculating a cross-entropy loss function based on the historical co-occurrence number corresponding to the sign language word group to which the target sign language word belongs and the position of the target sign language word in the first text sequence; training the sign language grammar error detection model based on the cross-entropy loss function.

4. The method of claim 1, wherein, The method further comprises: dividing a first historical sign language video into at least two sign language video segments, each sign language video segment corresponding to a group of sign language words, and each sign language video segment corresponding to a sign language action; acquiring a first value corresponding to each sign language word in the first sign language word group, the first value being used to represent the similarity between any sign language word and the sign language action corresponding to the first sign language word group; in a case where there is a second sign vocabulary group in the sign vocabulary groups corresponding to the at least two sign video clips, adding the second sign vocabulary group to the sign vocabulary group information table; wherein a maximum value of all first values corresponding to the second sign vocabulary group is less than a second threshold value, and a minimum value is greater than a third threshold value.

5. The method of claim 1, wherein, Before the target training sample is obtained by performing processing on the second training sample based on the first label, the method further comprises: obtaining a first sign video corresponding to the first text sequence, and determining an actual sign action duration corresponding to each of the first sign vocabularies; in a case where a time difference between the actual sign action duration corresponding to the third sign vocabulary and the preset sign action duration corresponding to the third sign vocabulary is greater than or equal to a fourth threshold value, determining that a processing label corresponding to the third sign vocabulary is a deletion label; in a case where the time difference is less than the fourth threshold value, determining that the processing label corresponding to the third sign vocabulary is a retention label; wherein the third sign vocabulary is one of the N first sign vocabularies.

6. A model training apparatus characterized by comprising: The model training apparatus comprises an acquisition module, a processing module and a training module. The acquisition module is configured to acquire a first text sequence, the first text sequence comprising N first sign vocabularies, the N first sign vocabularies comprising at least one grammatically correct sign vocabulary and at least one arbitrary sign vocabulary in a second text sequence, N being an integer greater than 1. The processing module is configured to input the first text sequence and a sign vocabulary group information table acquired by the acquisition module into a sign grammar error detection model, output N processing labels, one processing label corresponding to one first sign vocabulary, the processing label comprising a retention label indicating a retained sign vocabulary or a deletion label indicating a deleted sign vocabulary, and the sign vocabulary group information table comprising multiple sign vocabulary groups and a historical co-occurrence number corresponding to each sign vocabulary group. The processing module is further configured to process the first text sequence based on the N processing labels to obtain a third text sequence. The training module is configured to, in a case where the third text sequence is different from the second text sequence, train the sign grammar error detection model based on a target sign vocabulary, the target sign vocabulary being a different sign vocabulary in the third text sequence and the second text sequence.

7. The apparatus of claim 6, wherein, The processing module is specifically configured to: input the first text sequence and a sign vocabulary group information table into a sign grammar error detection model, and acquire a historical co-occurrence number corresponding to a sign vocabulary group to which each first sign vocabulary belongs; in a case where the second sign vocabulary is a preset sign vocabulary, if the second sign vocabulary is grammatically redundant in the first text sequence, the deletion label is taken as a processing label corresponding to the second sign vocabulary; if the second sign vocabulary is not grammatically redundant in the first text sequence, the retention label is taken as the processing label corresponding to the second sign vocabulary; the historical co-occurrence number corresponding to the sign vocabulary group to which the preset sign vocabulary belongs is greater than a first threshold value; In a case that the second sign vocabulary is not a preset sign vocabulary, the reserved label is determined as a processing label corresponding to the second sign vocabulary; The second sign vocabulary is one of the N first sign vocabularies.

8. The apparatus of claim 6, wherein, The training module is specifically configured to: calculate a cross-entropy loss function based on the historical co-occurrence number corresponding to the sign vocabulary group to which the target sign vocabulary belongs and the position of the target sign vocabulary in the first text sequence; train the sign grammar error detection model based on the cross-entropy loss function.

9. The apparatus of claim 6, wherein, The device further comprises a segmentation module and an adding module. The segmentation module is configured to, before the processing module inputs the first text sequence and the sign vocabulary group information table obtained by the obtaining module into the sign grammar error detection model and outputs the N processing labels, divide the first historical sign video into at least two sign video segments, each of which corresponds to a sign vocabulary group, and each of which corresponds to a sign action. The obtaining module is further configured to obtain a first value corresponding to each sign vocabulary in the first sign vocabulary group, the first value being used to represent the similarity between any sign vocabulary and the sign action corresponding to the first sign vocabulary group. The adding module is configured to, in a case that there is a second sign vocabulary group in the sign vocabulary groups corresponding to the at least two sign video segments segmented by the segmentation module, add the second sign vocabulary group to the sign vocabulary group information table. The maximum value of all the first values corresponding to the second sign vocabulary group is less than a second threshold value, and the minimum value is greater than a third threshold value.

10. The apparatus of claim 6, wherein, The obtaining module is further configured to, before the processing module performs processing on the second training sample based on the first label to obtain the target training sample, obtain a first sign video corresponding to the first text sequence and determine an actual sign action duration corresponding to each first sign vocabulary. The processing module is further configured to, in a case that a time difference between the actual sign action duration corresponding to a third sign vocabulary and a preset sign action duration corresponding to the third sign vocabulary is greater than or equal to a fourth threshold value, determine a deletion label as the processing label corresponding to the third sign vocabulary. The processing module is further configured to, in a case that the time difference is less than the fourth threshold value, determine a reserved label as the processing label corresponding to the third sign vocabulary. The third sign vocabulary is one of the N first sign vocabularies.

11. An electronic device, comprising: The device comprises a processor, a memory, and a program or instructions stored on the memory and executable on the processor, and the program or instructions are executed by the processor to implement the steps of the model training method according to any one of claims 1 to 5.

12. A readable storage medium, characterized by, The program or instructions are stored on the readable storage medium and are executed by the processor to implement the steps of the model training method according to any one of claims 1 to 5.

Images