Plug-and-play sign language word positioning and refinement method, system, device and storage medium

Abstract

The application discloses a plug-and-play sign language word positioning fine-tuning method, system, device and storage medium, which can correct the start and end positions of sign language actions on the basis of a traditional method, can make the boundaries of the start and end more accurate, and can make the sign language word positioning method more widely applied to the labeling of a sign language data set, and thus improve the accuracy of continuous sign language recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sign language gloss positioning, and particularly relates to a plug-and-play sign language gloss positioning fine-tuning method, system, device and storage medium. BACKGROUND

[0002] In order to communicate, the hearing-impaired mainly use sign language to express their ideas. In order to help them better integrate into society, researchers use computer algorithms to translate sign language into text, which is convenient for normal people to understand. In order to improve the accuracy of the sign language translation algorithm, researchers introduce sign language gloss (gloss) as auxiliary information. The sign language gloss is the basic ideographic unit of sign language, and its order is consistent with the order of the sign language video, and can well depict the characteristics of the sign language video. Unfortunately, it needs to be manually annotated by professionals, and the annotation cost is very high.

[0003] In order to quickly obtain a data set containing sign language gloss annotation, a single-sample sign language gloss positioning task has emerged. The purpose of the single-sample sign language gloss positioning task is to locate the start and end positions of a query short video corresponding to a sign language gloss in a target long video.

[0004] Most traditional methods are to extract the features of the sign language gloss query video (short video) and the target video (long video) frame by frame, then calculate the cosine similarity of each frame of the short video and each frame of the long video, and then use Dynamic Time Warping (DTW), Longest Path, Temporal Network, Hough Voting and other algorithms to determine the interval of the sign language gloss. Due to the existence of adjacent sign language gloss connection actions, the division of the traditional algorithm at the boundary is not accurate. In addition, the hearing-impaired people sign at different speeds, and if multi-scale information is not introduced, it will lead to inaccurate boundary division, thereby causing the data set annotation to be biased, and further affecting the accuracy of continuous sign language recognition after model training. SUMMARY

[0005] The purpose of the present application is to provide a plug-and-play sign language gloss positioning fine-tuning method, system, device and storage medium, which can make the start and end boundaries more accurate, so that the sign language gloss positioning method can be more widely applied to the annotation of sign language data sets, and further improve the accuracy of continuous sign language recognition.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] A plug-and-play sign language gloss positioning fine-tuning method, comprising:

[0008] Given a sign language word query video and a target video, a cosine similarity graph is formed by calculating the cosine similarity of each frame of the sign language word query video and each frame of the target video using a sign language word positioning algorithm, and the interval with the highest confidence is taken out as a preliminary positioning result;

[0009] The cosine similarity graph is down-sampled in the dimension of the target video by different multiples to form a multi-scale lookup table;

[0010] The start position and the end position in the preliminary positioning result are respectively iteratively optimized using a recurrent neural network. In the current iteration process, the previous part of the feature of the sign language word query video or the next part of the feature, the multi-scale lookup table feature in the neighborhood of the refined start position or end position obtained in the last iteration, and the hidden variable feature of the recurrent neural network of the last iteration are taken to generate the hidden variable feature of the recurrent neural network of the current iteration, and the correction amount of the start position or the end position of the current iteration is obtained. The refined start position or end position obtained in the last iteration is combined to obtain the refined start position or end position obtained in the current iteration. After multiple iterations of the start position and the end position, the optimized positioning result formed by the final optimized start position and the end position is obtained.

[0011] A plug-and-play sign language word positioning refinement system, comprising:

[0012] A preliminary positioning result acquisition module is configured to, given a sign language word query video and a target video, calculate the cosine similarity of each frame of the sign language word query video and each frame of the target video using a sign language word positioning algorithm, form a cosine similarity graph, and take out the interval with the highest confidence as a preliminary positioning result;

[0013] A multi-scale lookup table construction module is configured to

[0014] The cosine similarity graph is down-sampled in the dimension of the target video by different multiples to form a multi-scale lookup table;

[0015] The boundary refinement module is used to use a recurrent neural network to iteratively optimize the start position and the end position in the preliminary positioning result. During the current iteration, the first part of the feature or the second part of the feature of the sign language query video, the multi-scale lookup table feature in the neighborhood of the refined start position or end position obtained in the previous iteration, and the latent variable feature of the recurrent neural network in the previous iteration are taken to generate the latent variable feature of the recurrent neural network in the current iteration, and thereby obtain the correction amount of the start position or end position of the current iteration. Combined with the refined start position or end position obtained in the previous iteration, the refined start position or end position obtained in the current iteration is obtained; after multiple iterative optimizations of the start position and the end position, an optimized positioning result formed by the final optimized start position and end position is obtained.

[0016] A processing device comprising: one or more processors; a memory for storing one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0018] A readable storage medium stores a computer program, which implements the aforementioned method when the computer program is executed by a processor.

[0019] It can be seen from the technical solution provided by the present invention described above that, based on the traditional method, a multi-scale lookup table can be constructed to correct the start and end positions of sign language actions, further improving the performance of the single-sample sign language word localization method, so that it can better serve the scenario of sign language dataset annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A schematic diagram of the overall framework of a plug-and-play sign language word localization and refinement method provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a plug-and-play sign language word localization and refinement system provided by an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0025] Firstly, the terms possibly used in the present application are described as follows:

[0026] The terms "comprise", "contain", "include", "have" or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, the inclusion of a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.) should be interpreted as not only including the explicitly listed technical feature element, but also including other technical feature elements not explicitly listed but known in the art.

[0027] A sign language word positioning and fine-tuning method, system, device and storage medium provided by the present application are described below. The contents not described in detail in the embodiments of the present application belong to the prior art known to those skilled in the art. If the specific conditions are not specified in the embodiments of the present application, the conventional conditions in the art or the recommended conditions of the manufacturer are followed.

[0028] Embodiment one

[0029] The embodiments of the present application provide a sign language word positioning and fine-tuning method, Figure 1 The overall framework of the method is shown, which mainly includes the following three parts:

[0030] 1. Given a sign language word query video and a target video, use a sign language word positioning algorithm to calculate the cosine similarity of each frame of the sign language word query video and each frame of the target video, form a cosine similarity graph, and take the interval with the highest confidence from the cosine similarity graph as the preliminary positioning result.

[0031] This part can be realized by a traditional scheme. In general, the traditional method is used to roughly estimate the interval of sign language action, and the similarity matrix of the sign language word query video and the target video is reserved. Specifically:

[0032] Given a sign language word query video (short video) and a target video (long video), calculate the cosine similarity of each frame of the sign language word query video and each frame of the target video to obtain a cosine similarity graph. The cosine similarity calculation method is as follows:

[0033]

[0034] wherein T short is the total frame number of the sign language word query video, T long is the target video frame number, F t1 represents the feature corresponding to the sign language word query video t1 frame, F t2 represents the feature corresponding to the target video t2 frame, and Sim[t1, t2] represents the cosine similarity of the sign language word query video t1 frame and the target video t2 frame.

[0035] The above provides a calculation method of the cosine similarity of two frames. All the calculated cosine similarities are comprehensively calculated to form a cosine similarity graph (similarity matrix), wherein each row is the cosine similarity of a single frame of the sign language word query video and each frame of the target video.

[0036] In the embodiment of the application, the preliminary positioning result can be obtained by using existing methods such as a dynamic programming algorithm, a graph theory longest path algorithm, or a time network algorithm after the cosine similarity graph is calculated. For example, the interval with the maximum sum of cosine similarities that meets the dynamic programming condition is taken out.

[0037] The dynamic programming algorithm is used, and the formula is as follows:

[0038]

[0039] wherein f(.) is the state of the dynamic programming, that is, the maximum sum of the cosine similarities mentioned later, f(t1, t2) represents the maximum sum of the cosine similarities obtained by taking the sign language word query video from the start position to the t1 frame and the target video from the start position to the t2 frame; f(t1-1, t2-1) represents the maximum sum of the cosine similarities obtained by taking the sign language word query video from the start position to the t1-1 frame and the target video from the start position to the t2-1 frame; f(t1, t2-1) represents the maximum sum of the cosine similarities obtained by taking the sign language word query video from the start position to the t1 frame and the target video from the start position to the t2-1 frame; and f(t1-1, t2) represents the maximum sum of the cosine similarities obtained by taking the sign language word query video from the start position to the t1-1 frame and the target video from the start position to the t2 frame.

[0040] According to the principle of the above formula, f(t1, t2) is obtained only from f(t1-1, t2-1), f(t1, t2-1), and f(t1-1, t2), and the purpose is to maintain the relative order of the internal time of the long video and the short video.

[0041] The start positions of the two videos are enumerated, the maximum f(t1, t2) is selected, the t2 frame in the maximum f(t1, t2) is combined to determine that the interval with the highest confidence is [target video start position: t2], that is, the start position in the preliminary positioning result is the target video start position, and the end position is the t2 frame.

[0042] 2. Downsample the cosine similarity graph in the dimension of the target video by different multiples to form a multi-scale lookup table.

[0043] Traditional solutions do not introduce multi-scale information, resulting in inaccurate boundary demarcation. Therefore, the present invention establishes cosine similarity graphs of different scales to form a lookup table, which is robust to videos of different sign language speeds.

[0044] Let the lookup table of a single scale be L m , the value L at the t1th row and t2th column t m 1t2 Calculated by the following formula:

[0045]

[0046] Among them, 2 m is the step size of average pooling, which means downsampling the cosine similarity map by 2 m times, m is used to control whether to downsample and the multiple of downsampling, m = 0, 1, 2, 3..., the neighborhood range of average pooling is also 2 m , p is the target video frame number in the enumerated neighborhood range, t2′ represents the t2′ frame of the target video after downsampling (for example, if the t2 frame before downsampling becomes t2′=t2 / 2 after downsampling m frame), sim[t1,2 m t2′+p] represents the sign language query video frame t1 and the target video frame 2 m The cosine similarity of the t2′+p frame. For general target videos, a large downsampling factor will increase the computational complexity and cause excessive interference to the iterative refinement module. A small downsampling factor will fail to capture long-distance relationships. Therefore, the downsampling factor needs to be set according to the actual situation.

[0047] By setting the value of m, a multi-scale lookup table formed by several single-scale lookup tables is obtained.

[0048] For example, the value of m can be set to 0, 1, 2, or 3. If m=0, it indicates that no downsampling is performed, and if m=1, 2, or 3, it indicates a similarity graph after downsampling by 2, 4, or 8 times.

[0049] 3. Boundary refinement module.

[0050] This section mainly optimizes the positioning results of traditional methods by introducing a recurrent neural network algorithm to correct the start and end positions.

[0051] In an embodiment of the present invention, a recurrent neural network is used to iteratively optimize the start position and the end position in the preliminary positioning result. During the current iteration, the first or second part of the features of the sign language query video, the multi-scale lookup table features in the neighborhood of the refined start position or end position obtained in the previous iteration, and the latent variable features of the recurrent neural network in the previous iteration are taken to generate the latent variable features of the recurrent neural network in the current iteration, and thereby obtain a correction amount for the start position or end position of the current iteration. The refined start position or end position obtained in the previous iteration is then combined to obtain the refined start position or end position obtained in the current iteration. If the current iteration is the first iteration, the refined start position and end position obtained in the previous iteration are the start position and end position in the preliminary positioning result. After the start position and the end position are iteratively optimized for multiple times, the final optimized start position and end position are obtained as the optimized positioning result.

[0052] To ensure effective optimization, the iterative optimization of the start and end positions uses separate recurrent neural networks, referred to as the first and second recurrent neural networks. The following uses the iterative optimization of the start position in the preliminary positioning results as an example to illustrate the iterative optimization process and the associated recurrent neural network training methods.

[0053] 1) Take out the first part of the sign language query video feature F query , expressed as:

[0054]

[0055] Among them, F short The feature sequence representing the sign language word query video, Represents the first 1 / N frames of the sign language query video (i.e., frames 0 to 1 / N). frame), 1 / N is the set proportional coefficient.

[0056] For example, N may be set to 3, and features of the first 1 / 3 frame of the query short video are extracted as query features.

[0057] 2) Combine the multi-scale lookup table to determine the multi-scale lookup table feature F in the neighborhood of the refined starting position obtained in the previous iteration lookup Specifically, the current iteration is recorded as the kth iteration, and the lookup table of each scale is called one by one to obtain the single-scale lookup table feature corresponding to a neighborhood of the starting position obtained in the previous iteration. Then all the single-scale lookup table features are spliced ​​together to obtain the multi-scale lookup table feature F lookup .

[0058] Among them, the single-scale lookup table feature The way to obtain is expressed as:

[0059]

[0060] Among them, S k-1 is the refined starting position obtained in the previous iteration, is the neighborhood, R is the neighborhood width, Z is the set of integers ds∈Z means ds is an integer, Indicates S k-1 Neighborhood, that is, range [S k-1 -R,S k-1 +R];0: Represents the first 1 / N frames of the sign language word query video; since The value of is not necessarily an integer. When calling the lookup table to find features, a differentiable interpolation algorithm is needed to interpolate the features of the decimal position using the features of the surrounding integer points.

[0061] 3) Boundary refinement.

[0062] The feature F query and feature F lookup Splicing to obtain splicing feature F k-1 , through the Convolutional Gate Recurrent Unit (ConvGRU) using the latent variable feature h of the first recurrent neural network in the previous iteration k-1 and the splicing feature F k-1 Generate the hidden variable feature h of the first recurrent neural network of the current iteration k , combined with the hidden variable feature h of the first recurrent neural network of the current iteration k Get the correction value ΔS of the starting position of the current iteration k-1 , combined with the refined starting position S obtained in the previous iteration k-1 , get the starting position after the current iteration refinement: S k =S k-1 +ΔS k-1 .

[0063] Among them, the latent variable feature h k The generation method is as follows:

[0064] z k =σ(1DConv([h k-1 ,F k-1 ],W z ))

[0065] r k =σ(1DConv([h k-1 ,F k-1 ],W r))

[0066]

[0067] where h k-1 denotes the hidden state of the first recurrent neural network at the last iteration, σ denotes an activation function, z k is an update gate, which controls how much information of the current hidden state needs to be preserved from the history; r k is a reset gate, which controls the degree of dependence on the hidden state at the last time step; W z is a weight parameter in the update gate, W r is a weight parameter in the reset gate, is an intermediate feature, W h is a weight parameter in the calculation of the intermediate feature; since the present application is a refinement in the time dimension rather than in the spatial dimension, the two-dimensional convolution of the ConvGRU is replaced by a one-dimensional convolution 1DConv. The first recurrent neural network obtains h k , and then the time offset ΔS k-1 of the output of each iteration is obtained through the fully connected network at the end of the first recurrent neural network.

[0068] In the embodiment of the present application, the parameters (weights) in all iterations of the first recurrent neural network are shared. During training, the hidden state of the first recurrent neural network at the current iteration is obtained through the fully connected network at the end of the first recurrent neural network, and the starting position offset ΔS k-1 of the current iteration is obtained, so as to obtain the refined starting position S k of the current iteration. The mean square loss between the refined starting position of each iteration and the data label is calculated, and then weighted and added. The weight gradually increases from the first iteration to the last iteration, and the first training loss function is obtained. The first recurrent neural network is optimized by using the first training loss function.

[0069] The first training loss function is represented as:

[0070]

[0071] where γ is a decimal number between 0 and 1, which makes the weight gradually increase from the first iteration to the last iteration, K is the total number of iterations, S gt is the starting position labeled by the data set, S i is the refined starting position obtained by the i-th iteration.

[0072] For the ending position, the iteration optimization is also performed in a similar manner as 1) to 3) above:

[0073] 1) Take the feature of the (N-1) / N frame at the end of the query short video as the query feature Fquery ′.

[0074] 2) Determine the multi-scale lookup table features F in the neighborhood of the refined start position obtained in the last iteration, in combination with the multi-scale lookup table lookup ′; Similarly, call each scale lookup table one by one to obtain the corresponding single-scale lookup table features in the neighborhood of the start position obtained in the last iteration Then splice all single-scale lookup table features to obtain multi-scale lookup table features F lookup ′.

[0075]

[0076] where e k-1 is the refined end position obtained in the last iteration, represents the feature corresponding to the (N-1) / N frame at the end of the sign language word query video, T short represents the (N-1) / N frame at the end of the sign language word query video. represents the neighborhood of e k-1 , i.e. the range [e k-1 -R, e k-1 +R].

[0077] 3) Splice the feature F query ′ and the feature F lookup ′ to obtain the spliced feature F k-1 ′, and use the hidden variable feature h k-1 ′ of the second recurrent neural network in the last iteration to generate the hidden variable feature h k-1 ′ of the second recurrent neural network in the current iteration through the convolution gate recurrent unit, combine the hidden variable feature h k ′ of the second recurrent neural network in the current iteration to obtain the correction amount Δe k-1 of the start position in the current iteration in the current iteration, combine the refined start position e k-1 obtained in the last iteration to obtain the refined start position e k in the current iteration: e k-1 = e k-1 + Δe gt .

[0078] Similarly, calculate the mean square loss between the refined end position and the data label in each iteration, and then add them up with weights, with the weights gradually increasing from the first iteration to the last iteration to obtain the second training loss function, and use the second training loss function to optimize the second recurrent neural network.

[0079] The second training loss function is represented as:

[0080]

[0081] Among them, γ is a decimal between 0 and 1, which makes the weight gradually increase from the first iteration to the last iteration, K is the total number of iterations, e gt is the end position of the dataset annotation, e i is the final position after refinement obtained in the i-th iteration.

[0082] Figure 1 The overall framework shown is only described by taking the iterative optimization method of the starting position as an example. As previously described, the iterative optimization method of the ending position is generally the same; at the same time, Figure 1 An example of N=3 is provided. In practical applications, a corresponding proportional coefficient can be set according to circumstances or experience.

[0083] The solution provided by the present invention comprises a traditional algorithm, a lookup table creation module, and an iterative refinement module, all of which are connected in series to form a complete framework. This framework employs a recurrent neural network paradigm, continuously iteratively updating and refining the start and end positions, significantly improving positioning accuracy. This enables the sign language word localization method to be more widely applied to the annotation of sign language datasets, thereby improving the accuracy of sign language translation.

[0084] Through the description of the above embodiments, those skilled in the art will clearly understand that the above embodiments can be implemented through software or by using software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash drive, a mobile hard disk, etc.) and includes a number of instructions for causing a computer device (such as a personal computer, a server, or a network device) to execute the methods described in the various embodiments of the present invention.

[0085] Example 2

[0086] The present invention also provides a plug-and-play sign language word positioning and refinement system, which is mainly implemented based on the method provided in the above embodiment, such as Figure 2 As shown, the system mainly includes:

[0087] The preliminary positioning result acquisition module is used to calculate the cosine similarity of each frame of the sign language query video and each frame of the target video using the sign language word positioning algorithm, form a cosine similarity graph, and extract the interval with the highest confidence as the preliminary positioning result;

[0088] Multi-scale lookup table building block for

[0089] The cosine similarity graph is downsampled to different multiples in the dimension of the target video to form a multi-scale lookup table;

[0090] The boundary refinement module is used to use a recurrent neural network to iteratively optimize the start position and the end position in the preliminary positioning result. During the current iteration, the first part of the feature or the second part of the feature of the sign language query video, the multi-scale lookup table feature in the neighborhood of the refined start position or end position obtained in the previous iteration, and the latent variable feature of the recurrent neural network in the previous iteration are taken to generate the latent variable feature of the recurrent neural network in the current iteration, and thereby obtain the correction amount of the start position or end position of the current iteration, and combine the refined start position or end position obtained in the previous iteration to obtain the refined start position or end position obtained in the current iteration; after performing multiple iterative optimizations on the start position and the end position, an optimized positioning result formed by the final optimized start position and end position is obtained.

[0091] Those skilled in the art will clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0092] Example 3

[0093] The present invention also provides a processing device, such as Figure 3 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the aforementioned embodiment.

[0094] Furthermore, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0095] In the embodiment of the present invention, the specific types of the memory, input device, and output device are not limited; for example:

[0096] The input device can be a touch screen, image acquisition device, physical button or mouse;

[0097] The output device may be a display terminal;

[0098] The memory can be a random access memory (RAM) or a non-volatile memory such as a disk memory.

[0099] Embodiment Four

[0100] The application further provides a readable storage medium storing a computer program, which, when executed by a processor, implements the method provided in the foregoing embodiments.

[0101] The readable storage medium in the embodiments of the application as a computer readable storage medium can be arranged in the foregoing processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various media capable of storing program codes.

[0102] The above merely describes the preferred embodiments of the application, but the protection scope of the application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the application, which should be covered by the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A plug-and-play sign language word localization and refinement method, characterized in that: include: Given a sign language query video and a target video, the sign language word localization algorithm is used to calculate the cosine similarity between each frame of the sign language query video and each frame of the target video, forming a cosine similarity graph, and taking the interval with the highest confidence as the preliminary localization result; The cosine similarity graph is downsampled to different multiples in the dimension of the target video to form a multi-scale lookup table; The starting position and the ending position in the preliminary positioning result are iteratively optimized using a recurrent neural network. During the current iteration, the first or second part of the features of the sign language query video, the multi-scale lookup table features in the neighborhood of the refined starting position or ending position obtained in the previous iteration, and the latent variable features of the recurrent neural network in the previous iteration are taken to generate the latent variable features of the recurrent neural network in the current iteration, and thereby obtain a correction amount for the starting position or ending position of the current iteration. The refined starting position or ending position obtained in the previous iteration is combined to obtain the refined starting position or ending position obtained in the current iteration. After performing multiple iterative optimizations on the starting position and the ending position, an optimized positioning result formed by the final optimized starting position and ending position is obtained. The iterative optimization of the start position and the end position in the preliminary positioning result by using a recurrent neural network includes: Set up two recurrent neural networks, called the first recurrent neural network and the second recurrent neural network; The method of iteratively optimizing the starting position in the preliminary positioning result includes: taking out the first part of the feature F of the sign language query video query , the multi-scale lookup table feature F in the neighborhood of the starting position after refinement obtained in the previous iteration lookup , and concatenate the two to obtain the concatenated feature F k-1 , through the convolutional gated recurrent unit, the latent variable feature h of the first recurrent neural network in the previous iteration is used k-1 and the splicing feature F k-1 Generate the hidden variable feature h of the first recurrent neural network of the current iteration k , combined with the hidden variable feature h of the first recurrent neural network of the current iteration k Get the correction value ΔS of the starting position of the current iteration k-1 , combined with the refined starting position S obtained in the previous iteration k-1 , get the starting position after the current iteration refinement: S k =S k-1 +ΔS k-1 ; The method of iteratively optimizing the end position in the preliminary positioning result includes: taking the features of the (N-1) / N frames at the end of the sign language query video The multi-scale lookup table feature F in the neighborhood of the final position after refinement obtained in the previous iteration lookup ′, concatenate the two to obtain the concatenated feature F k-1 ′, through the convolutional gated recurrent unit, the latent variable feature h of the second recurrent neural network in the previous iteration is used k-1 ′ and splicing feature F k-1 'Generate the hidden variable feature h of the second recurrent neural network of the current iteration k ′, combined with the latent variable feature h of the second recurrent neural network of the current iteration k 'Get the correction value Δe of the starting position of the current iteration k-1 , combined with the refined starting position e obtained in the previous iteration k-1 , get the starting position after the current iteration refinement: e k =e k-1 +Δe k-1 .

2. The plug-and-play sign language word positioning and refinement method according to claim 1, characterized in that: The sign language word localization algorithm is used to calculate the cosine similarity between each frame of the sign language word query video and each frame of the target video, forming a cosine similarity graph. The interval with the highest confidence score is taken as the preliminary localization result, including: Calculate the cosine similarity between each frame of the sign language query video and each frame of the target video to obtain a cosine similarity graph. The cosine similarity calculation method is as follows: Among them, T short is the total number of frames of the sign language word query video, T long is the target video frame number, F t1 F represents the features corresponding to the sign language word query video frame t1, t2 represents the features corresponding to the target video frame t2, Sim[t1,t2] represents the cosine similarity between the sign language query video frame t1 and the target video frame t2; Using the dynamic programming algorithm, the formula is as follows: Among them, f(.) is the state of dynamic programming, f(t1,t2) represents the maximum sum of cosine similarity of the sign language query video from the starting position to the t1 frame and the target video from the starting position to the t2 frame; f(t1-1,t2-1) represents the maximum sum of cosine similarity of the sign language query video from the starting position to the t1-1 frame and the target video from the starting position to the t2-1 frame; f(t1,t2-1) represents the maximum sum of cosine similarity of the sign language query video from the starting position to the t1 frame and the target video from the starting position to the t2-1 frame; f(t1-1,t2) represents the maximum sum of cosine similarity of the sign language query video from the starting position to the t1-1 frame and the target video from the starting position to the t2 frame; Enumerate the starting positions of the two videos, select the largest f(t1, t2), and combine the t2 frame in the largest f(t1, t2) to determine the interval with the highest confidence as [target video starting position: t2]. That is, the starting position in the preliminary positioning result is the starting position of the target video, and the end position is the t2 frame.

3. The plug-and-play sign language word positioning and refinement method according to claim 1, characterized in that: The step of downsampling the cosine similarity graph in the dimension of the target video by different multiples to form a multi-scale lookup table includes: Let the lookup table of a single scale be L m , the value at the t1th row and t2th column is calculated as follows: Among them, 2 m is the step size of average pooling, which means downsampling the cosine similarity map by 2 m times, m is used to control whether to downsample and the multiple of downsampling, m = 0, 1, 2, 3..., the neighborhood range of average pooling is also 2 m , p is the target video frame number in the enumerated neighborhood range, t2′ represents the t2′ frame of the target video after downsampling, sim[t1,2 m t2′+p] represents the sign language query video frame t1 and the target video frame 2 m Cosine similarity of t2′+p frames; By setting the value of m, a multi-scale lookup table formed by several single-scale lookup tables is obtained.

4. The plug-and-play sign language word positioning and refinement method according to claim 1, characterized in that: When iteratively optimizing the starting position in the preliminary positioning result, the first part of the feature F of the sign language word query video query Expressed as: When iteratively optimizing the end position in the preliminary positioning result, the features of the (N-1) / N frames at the end of the sign language query video are Expressed as: Among them, F short The feature sequence representing the sign language word query video, Indicates the features corresponding to the first 1 / N frames of the sign language query video, where 1 / N is the set scale factor. Indicates the features corresponding to the (N-1) / N frames at the end of the sign language word query video.

5. The plug-and-play sign language word positioning and refinement method according to claim 1 or 4, characterized in that: When iteratively optimizing the starting position in the preliminary positioning result, the multi-scale lookup table feature F in the neighborhood of the refined starting position obtained in the previous iteration is determined by combining the multi-scale lookup table. lookup ; Record the current iteration as the kth iteration, call the lookup table of each scale one by one, and obtain the single-scale lookup table feature corresponding to a neighborhood of the starting position obtained in the previous iteration Then all the single-scale lookup table features are spliced ​​together to obtain the multi-scale lookup table feature F lookup ; Among them, the single-scale lookup table feature The way to obtain is expressed as: Among them, L m Represents a single scale lookup table, S k-1 is the refined starting position obtained in the previous iteration, Indicates S k-1 neighborhood, R is the neighborhood width, Represents the first 1 / N frames of the sign language word query video; When iteratively optimizing the end position in the preliminary positioning result, the multi-scale lookup table feature F in the neighborhood of the refined start position obtained in the previous iteration is determined by combining the multi-scale lookup table. lookup '; Call the lookup table of each scale one by one to obtain the single-scale lookup table features corresponding to a neighborhood of the starting position obtained in the previous iteration Then all the single-scale lookup table features are spliced ​​together to obtain the multi-scale lookup table feature F lookup ′; Among them, the single-scale lookup table feature The way to obtain is expressed as: Among them, e k-1 is the final position after refinement obtained in the previous iteration, Indicates the features corresponding to the (N-1) / N frames at the end of the sign language word query video, (N-1) / N frames representing the end of the sign language word query video; Indicates e k-1 neighborhood.

6. The plug-and-play sign language word positioning and refinement method according to claim 1, characterized in that: During the first recurrent neural network training, the mean square loss between the starting position and the data label after each iteration is calculated, and then weighted addition is performed. The weight gradually increases from the first iteration to the last iteration to obtain the first training loss function Using the first training loss function Optimize the first recurrent neural network; when training the second recurrent neural network, calculate the mean square loss between the end position and the data label after each iteration of refinement, and then add them weightedly. The weight gradually increases from the first iteration to the last iteration to obtain the second training loss function Using the second training loss function Optimize the second recurrent neural network; First training loss function With the second training loss function Expressed as: Among them, γ is a decimal between 0 and 1, which makes the weight gradually increase from the first iteration to the last iteration, K is the total number of iterations, s gt is the starting position of the dataset annotation, S i is the starting position after refinement obtained in the i-th iteration, e gt is the end position of the dataset annotation, e i is the final position after refinement obtained in the i-th iteration.

7. A plug-and-play sign language word positioning and refinement system, characterized by: The method according to any one of claims 1 to 6 is implemented, and the system comprises: The preliminary positioning result acquisition module is used to calculate the cosine similarity of each frame of the sign language query video and each frame of the target video using the sign language word positioning algorithm, form a cosine similarity graph, and extract the interval with the highest confidence as the preliminary positioning result; Multi-scale lookup table building block for The cosine similarity graph is downsampled to different multiples in the dimension of the target video to form a multi-scale lookup table; The boundary refinement module is used to use a recurrent neural network to iteratively optimize the start position and the end position in the preliminary positioning result. During the current iteration, the first part of the feature or the second part of the feature of the sign language query video, the multi-scale lookup table feature in the neighborhood of the refined start position or end position obtained in the previous iteration, and the latent variable feature of the recurrent neural network in the previous iteration are taken to generate the latent variable feature of the recurrent neural network in the current iteration, and thereby obtain the correction amount of the start position or end position of the current iteration. Combined with the refined start position or end position obtained in the previous iteration, the refined start position or end position obtained in the current iteration is obtained; after multiple iterative optimizations of the start position and the end position, an optimized positioning result formed by the final optimized start position and end position is obtained.

8. A processing device, characterized in that include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images