Sign language recognition method and device based on improved YOLOv12

Abstract

The application discloses a sign language recognition method and device based on an improved YOLOv12, and the method comprises the following steps: acquiring a target sign language video, and converting the target sign language video into a target image; improving a YOLOv12 model through an efficient multi-scale large-core attention mechanism and a deformable hollow convolution to obtain a sign language recognition model; inputting the target image into the sign language recognition model to obtain a sign language category; and integrating the sign language category to obtain a target sign language recognition result. The application can improve the precision and speed of sign language recognition and can be widely applied to the technical field of computer vision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a sign language recognition method and device based on improved YOLOv12. BACKGROUND

[0002] Deploying sign language recognition devices in public places can effectively improve the experience of sign language communication in public services and provide support for building an accessible communication environment. However, existing static sign language recognition researches mostly focus on the performance improvement of algorithm models, and pay less attention to the deployment problems of the system in real complex environments. On the one hand, the deployment in public places is large in number and wide in coverage. If a cloud computing solution is adopted, it will cause high transmission delay and high cost of computing resources. Under this background, deploying sign language recognition models to edge devices becomes a more feasible path selection. Edge computing has the advantages of low delay, low energy consumption and privacy protection, and can effectively support the needs of real-time sign language recognition. However, at the same time, the large parameter quantity of deep neural networks requires high operation capability of edge devices. Although existing researches have proposed various lightweight improvement methods for YOLO structure, these methods often sacrifice the computational efficiency of the model to increase the inference time in exchange for smaller parameter size, and still cannot meet the high real-time requirement on resource-limited edge platforms. On the other hand, the data sets used in existing sign language recognition researches are mostly static images with single background, prominent gesture area and less environmental interference, and lack modeling of real complex background factors such as occlusion, dynamic crowd and light change, resulting in poor generalization ability of the model in actual scenes. SUMMARY

[0003] Therefore, the embodiments of the present application mainly aim to provide a sign language recognition method and device based on improved YOLOv12, so as to solve at least one of the problems in the prior art, and improve the accuracy and efficiency of sign language recognition.

[0004] To achieve the above-mentioned purpose, one aspect of the embodiments of the present application provides a sign language recognition method based on improved YOLOv12, which comprises the following steps:

[0005] Obtaining a target sign language video, and converting the target sign language video into a target image;

[0006] Improving the YOLOv12 model through an efficient multi-scale large kernel attention mechanism and a deformable dilated convolution to obtain a sign language recognition model;

[0007] Inputting the target image into the sign language recognition model to obtain a sign language category;

[0008] Integrating the sign language category to obtain a target sign language recognition result.

[0009] In some embodiments, the target sign language video is obtained and converted into a target image, including the following steps:

[0010] An initial sign language video is collected through a camera connected to a Raspberry Pi;

[0011] The initial sign language video is preprocessed to obtain the target sign language video;

[0012] The target sign language video is converted to obtain a single-frame continuous target image.

[0013] In some embodiments, the YOLOv12 model is improved through an efficient multi-scale large kernel attention mechanism and a deformable dilated convolution to obtain a sign language recognition model, including the following steps:

[0014] Two dilated convolutions and one standard convolution are set for each branch of the multi-scale large kernel attention mechanism to obtain the efficient multi-scale large kernel attention mechanism; wherein the efficient multi-scale large kernel attention mechanism includes at least two multi-scale branches;

[0015] The efficient multi-scale large kernel attention mechanism is introduced into the YOLOv12 model.

[0016] In some embodiments, the YOLOv12 model is improved through an efficient multi-scale large kernel attention mechanism and a deformable dilated convolution to obtain a sign language recognition model, including the following steps:

[0017] The deformable dilated convolution is combined with the C3k2 module to obtain a deformable dilated convolution C3k2 module;

[0018] The deformable dilated convolution C3k2 module is introduced into the YOLOv12 model.

[0019] In some embodiments, the target image is input into the sign language recognition model to obtain a sign language category, including the following steps:

[0020] The target image is input into the backbone network of the sign language recognition model, and the backbone network features are output through a plurality of convolution modules, feature extraction units, deformable dilated convolution C3k2 modules, region attention enhanced cross-feature modules, and efficient multi-scale large kernel attention mechanisms;

[0021] The backbone network features are input into the neck network of the sign language recognition model, and the neck network features are output through a nearest neighbor up-sampling module, a splicing module, and a region attention enhanced cross-feature module;

[0022] The neck network features are input into the detection head of the sign language recognition model to obtain the sign language category.

[0023] In some embodiments, the integrating the sign language categories comprises the following steps:

[0024] The discrete sign language categories are integrated in chronological order to obtain a continuous initial sign language recognition result.

[0025] The initial sign language recognition result is error-corrected and supplemented in grammar structure by a natural language processing technology to obtain the target sign language recognition result.

[0026] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a sign language recognition device based on an improved YOLOv12, the device comprises:

[0027] An image acquisition module is configured to acquire a target sign language video and convert the target sign language video into a target image.

[0028] A model construction module is configured to improve a YOLOv12 model by using an efficient multi-scale large kernel attention mechanism and a deformable dilated convolution to obtain a sign language recognition model.

[0029] A data processing and analysis module is configured to input the target image into the sign language recognition model to obtain a sign language category.

[0030] A data integration module is configured to integrate the sign language categories to obtain a target sign language recognition result.

[0031] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method described above when executing the computer program.

[0032] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method described above.

[0033] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the method described above.

[0034] The embodiments of the present application at least have the following beneficial effects: the present application provides a sign language recognition method and device based on improved YOLOv12, the scheme obtains a target sign language video, and converts the target sign language video into a target image; the YOLOv12 model is improved through an efficient multi-scale large kernel attention mechanism and a deformable hollow convolution, and a sign language recognition model is obtained; the target image is input into the sign language recognition model, and a sign language category is obtained; the sign language category is integrated, and a target sign language recognition result is obtained, thereby improving the precision and speed of sign language recognition. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0036] Figure 1 is a sign language recognition method flowchart based on improved YOLOv12 provided by the embodiments of the present application;

[0037] Figure 2 is a schematic diagram of the overall structure of the improved YOLOv12 model provided by the embodiments of the present application;

[0038] Figure 3 is a schematic diagram of the structure of each part of the improved YOLOv12 model provided by the embodiments of the present application;

[0039] Figure 4 is a schematic diagram of the deformable hollow convolution structure provided by the embodiments of the present application;

[0040] Figure 5 is a schematic diagram of the principle of deformable hollow convolution provided by the embodiments of the present application;

[0041] Figure 6 is a schematic diagram of the sign language recognition process based on improved YOLOv12 provided by the embodiments of the present application;

[0042] Figure 7 is a schematic diagram of the hardware structure of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0043] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application, but are only examples of devices and methods consistent with some aspects of the embodiments of the present application as described in the appended claims.

[0044] It should be noted that although the functional modules are divided in the system schematic diagram, and the logical sequence is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the system or the sequence in the flowchart. The terms "first / S100", "second / S200" in the specification and claims and the above drawings can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to a determination".

[0045] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0047] As Figure 1 As shown in the figure, the embodiments of the present application provide a sign language recognition method based on improved YOLOv12, which can include but is not limited to steps S100 to S400:

[0048] Step S100, acquiring a target sign language video, and converting the target sign language video into a target image;

[0049] Step S200, improving the YOLOv12 model through an efficient multi-scale large kernel attention mechanism and a deformable dilated convolution to obtain a sign language recognition model;

[0050] Step S300, inputting the target image into the sign language recognition model to obtain a sign language category;

[0051] Step S400, the sign language categories are integrated to obtain a target sign language recognition result.

[0052] In some embodiments, step S100 can include but is not limited to steps S110-S130:

[0053] Step S110, an initial sign language video is collected through a camera connected to a Raspberry Pi;

[0054] Step S120, the initial sign language video is preprocessed to obtain a target sign language video;

[0055] Step S130, the target sign language video is converted to obtain a single-frame continuous target image.

[0056] In step S110 of some embodiments, the initial sign language video is captured in real time through a camera connected to a Raspberry Pi, thereby ensuring high-quality picture input. Through the combination of Raspberry Pi and the improved YOLOv12 sign language recognition model, lightweight deployment of the edge is realized, the hardware cost and network dependence are reduced, and it is especially suitable for scenarios without network environment or with high real-time requirements (such as instant communication).

[0057] In step S120 of some embodiments, the collected initial sign language video is preprocessed to optimize the quality of the video image and improve the efficiency of subsequent processing. Optionally, the initial sign language video is subjected to frame rate adjustment, resolution adjustment, and noise reduction processing, but is not limited thereto. Illustratively, the frame rate of the video is adjusted according to requirements, the number of frames is reduced by frame extraction, or the number of frames is increased by interpolation. The resolution of the video is adjusted to meet the needs of subsequent processing, the resolution is reduced by scaling algorithm (such as bilinear interpolation, bicubic interpolation), thereby reducing the amount of calculation and improving the processing speed, or the resolution is improved by super-resolution technology (such as deep learning model), thereby enhancing the image details. The noise in the video is reduced and the image quality is improved by averaging or median filtering several consecutive frames, or by applying Gaussian filtering, bilateral filtering, etc. to each frame, thereby enhancing the effect of subsequent processing.

[0058] In step S130 of some embodiments, the target sign language video obtained after preprocessing is converted, and by splitting the target sign language video, a single-frame continuous target image can be obtained for subsequent frame-by-frame analysis.

[0059] In some embodiments, step S200 can include but is not limited to steps S210-S220:

[0060] Step S210, two times of hole convolution and one time of standard convolution are set for each branch of the multi-scale large kernel attention mechanism, to obtain the efficient multi-scale large kernel attention mechanism;

[0061] Step S220, the efficient multi-scale large kernel attention mechanism is introduced into the YOLOv12 model.

[0062] In step S210 of some embodiments, based on the multi-scale large kernel attention mechanism (MLKA), a design combining channel splitting and standard convolution is adopted, to obtain an efficient multi-scale large kernel attention mechanism (E-MLKA) with high efficiency and light weight. Illustratively, based on the multi-scale large kernel attention mechanism, at least two multi-scale branches are retained to reduce the calculation overhead, and only two times of hole convolution and one time of standard convolution are used in each branch, so as to balance the calculation efficiency and feature extraction capability. Let the input x∈R C×H×W , each branch uses a standard convolution kernel , then:

[0063] F i = X * W i , i∈{1,…,N}

[0064] wherein X is input data with a dimension of C×H×W; k i is the convolution kernel size of the i-th branch (such as 3×3, 5×5); W i is the convolution kernel of the i-th branch; F i is the output data of the i-th branch; * is the convolution operation; and N is the number of branches.

[0065] The E-MLKA obtained by the design combining splitting and standard convolution effectively controls the number of branches and the number of convolutions, reduces the overall calculation amount compared with the MLKA, and further enhances the expression ability of the multi-scale receptive field. Optionally, a Sigmoid function is used instead of simple point multiplication, and a learnable scaling factor is added to realize adaptive adjustment of the feature intensity.

[0066] In step S220 of some embodiments, as Figure 2As shown, an efficient multi-scale large kernel attention mechanism is introduced at the end of the YOLOv12 model's backbone network. This mechanism receives input from the last 256 channels of the backbone network and adaptively assigns higher weights to the key hand space. The processed features are then fed to the neck network for multi-scale fusion. This efficient multi-scale large kernel attention mechanism allows for further optimization of the model structure while maintaining multi-scale modeling capabilities, achieving a more lightweight design.

[0067] In some embodiments, step S200 may also include, but is not limited to, steps S230 to S240:

[0068] Step S230: Combine the deformable dilated convolution with the C3k2 module to obtain the deformable dilated convolution C3k2 module;

[0069] Step S240: Introduce the deformable dilated convolution C3k2 module into the YOLOv12 model.

[0070] In step S230 of some embodiments, based on the C3k2 module, a switchable atrous convolution (SAConv) is used instead of the standard convolution to obtain the switchable atrous convolution C3k2 module, i.e., the SA_C3k2 structure. This allows the network to dynamically select between standard convolution and atrous convolution, and controls the weighting ratio of the two through a learnable weight map G, flexibly expanding the receptive field and finding a suitable balance between local details and global context. For example, during the convolution fusion process, SAConv introduces a position-sensitive switching function to output a learnable weight map G. This weight map controls the weighting ratio of the standard convolution output and the atrous convolution output at each spatial location, as expressed below:

[0071] y = G·y s +(1-G)·y l

[0072] Where G is an adjustable weight; y s This is the output of a standard convolution; y l This is the output of dilated convolution.

[0073] like Figure 4As shown, the structure consists of pre-global context enhancement and post-global context enhancement, and a core deformable dilated convolution. Here, S represents adjustable gating weights; Global AvgPool represents a global average pooling layer; Conv(1x1) represents a pointwise convolution; and Conv(3x3), Atrous=3 represents a dilated convolution with an inflation rate of 3. Through an efficient multi-scale large-kernel attention mechanism, the network can flexibly adjust between hand detail capture and global context modeling, dynamically expanding the receptive field to flexibly detect sign language at different scales, such as… Figure 5 As shown, after inputting the image, it undergoes standard convolution and atrous convolution respectively, and the ratio is dynamically selected through a gating mechanism (Switch) to finally generate a fused output feature map.

[0074] In step S240 of some embodiments, such as Figure 2 As shown, an efficient multi-scale large-kernel attention mechanism is introduced into layers 2 and 4 of the YOLOv12 model, corresponding to stages P2 and P3 respectively, to enhance the modeling ability of shallow structures for hand edges and long-distance hand movements. Furthermore, the efficient multi-scale large-kernel attention mechanism is applied to the P5 branch (layer 20) of head detection to enhance the spatial modeling ability of deep semantic features before output, thereby improving the contextual understanding of multi-scale target sign language.

[0075] In step S200 of some embodiments, based on the multi-scale attention concept of the MLKA module, a highly efficient and lightweight multi-scale large-kernel attention mechanism is proposed, balancing computational efficiency and feature extraction capabilities. While maintaining multi-scale deep semantic modeling capabilities, the model structure is further optimized, achieving a better lightweight effect. Furthermore, the shallow and mid-layer structures of YOLOv12 are adjusted, and an SA_C3k2 structure is proposed, which can flexibly expand the receptive field and find a suitable balance between local details and global context. When processing sign language in scenarios with different scales, blurred edges, or complex backgrounds, it can cover the complete semantic region, enhancing the model's ability to recognize multi-scale features. The improved YOLOv12 sign language recognition model can automatically adapt to different gesture shapes, sizes, and positional changes through a structural attention mechanism, effectively suppressing background interference and strengthening hand edge detection capabilities, thereby improving the overall accuracy and stability of recognition.

[0076] In some embodiments, a static sign language image dataset covering various real-world conditions is constructed, encompassing multiple variables such as different clothing styles, different background environments, different lighting conditions, and different shooting distances and angles. This significantly enhances the diversity and complexity of the data. The image dataset is then used to train the sign language recognition model, providing strong support for the training and evaluation of the model under real-world conditions.

[0077] Optionally, in the deployment of the sign language recognition model, the LAMP (Layer-wise Adaptive Magnitude-based Pruning) method is used to prune the structure of the sign language recognition model, and it is deployed on the edge computing platform of Raspberry Pi 5 fusion AI Kit. Without significantly affecting the detection accuracy, the computational load of the sign language recognition model is reduced by about 50% compared with the original YOLOv12, and the recognition speed reaches 23 FPS. This verifies the deployability and practicality of the sign language recognition model obtained after improving YOLOv12 in this embodiment of the invention on the edge computing platform.

[0078] In some embodiments, step S300 may include, but is not limited to, steps S310 to S330:

[0079] Step S310: Input the target image into the backbone network of the sign language recognition model, and output the backbone network features through multiple convolutional modules, feature extraction units, deformable dilated convolution C3k2 modules, region attention enhancement cross-feature modules, and efficient multi-scale large kernel attention mechanisms.

[0080] Step S320: Input the backbone network features into the neck network of the sign language recognition model, and output the neck network features through the nearest neighbor upsampling module, the splicing module, and the region attention enhancement cross-feature module;

[0081] Step S330: Input the neck network features into the detection head of the sign language recognition model to obtain the sign language category.

[0082] In steps S310 to S330 of some embodiments, a sign language recognition model is used to efficiently detect and classify the target image in a single frame, accurately identifying the category of sign language actions. The improved YOLOv12 sign language recognition model in this embodiment is as follows: Figure 2 As shown, the structure of each part of this sign language recognition model is as follows: Figure 3As shown, SA_BottleNeck is a component of SA_C3k, SA_C3k is a component of SA_C3k2, and ABlock is a component of A2C2f. For example, the target image first enters the backbone network, undergoing multiple convolutional operations (Conv, including Conv2D, BN, and SiLU) through multiple convolutional modules to extract basic features. During these multiple convolutional operations, a deformable dilated convolution C3k2 module (SA_C3k2) is interspersed to enhance feature extraction. A region attention enhancement cross-feature module (A2C2f) is also introduced to focus on key region features. Then, an efficient multi-scale large kernel attention mechanism (E-MLKA) is used to mine multi-scale features, outputting the backbone network features. Subsequently, the backbone network features flow into the neck network, where they are improved in resolution through a nearest neighbor upsampling module. Combined with features from different levels, these features are fused through a concatenation module (Concat), and then optimized by the region attention enhancement cross-feature module (A2C2f). After multiple iterations, the multi-scale effective features are integrated. Finally, the features output by the neck network are fed into the detector of the head. Based on the fused features, the detector predicts the category and location of gestures in the sign language image, thus completing the recognition and localization of sign language content. The detector also includes sub-components such as depthwise separable convolution (DSC), probability distribution + distribution focal loss (Problou+DFL loss), and binary cross-entropy loss (BCE loss).

[0083] The sign language recognition model optimizes recognition speed and accuracy through deep learning algorithms, adapting to different lighting conditions and complex backgrounds to ensure real-time processing capabilities. Sign language categories are output as structured data, providing a reliable basis for subsequent data analysis.

[0084] In some embodiments, step S400 may include, but is not limited to, steps S410 to S420:

[0085] Step S410: Integrate the discrete sign language categories in chronological order to obtain continuous initial sign language recognition results;

[0086] Step S420: The initial sign language recognition result is corrected for errors and supplemented with grammatical structure using natural language processing technology to obtain the target sign language recognition result.

[0087] In steps S410 to S420 of some embodiments, discrete sign language categories are integrated in chronological order and combined with contextual semantic analysis to form coherent sentences or expressions, resulting in continuous initial sign language recognition results. Then, natural language processing (NLP) technology is used to correct possible recognition errors and supplement grammatical structures, ultimately generating fluent and accurate text information to obtain the target sign language recognition result.

[0088] In some embodiments, the target sign language recognition results are fed back to the user in real time via a front-end interface in text or voice form. Users can view the recognition results through a visual interface or obtain information through voice broadcast. Interactive operations are also supported, such as correcting recognition errors or adjusting the output method, ensuring a flexible and user-friendly service to meet the needs of different users.

[0089] refer to Figure 6 The process of sign language recognition using the improved YOLOv12 sign language recognition model includes:

[0090] Step 1, Image Acquisition Layer

[0091] Sign language video is captured by a camera connected to a Raspberry Pi and then processed into single-frame continuous images.

[0092] Step 2, Data Processing and Analysis Layer

[0093] The improved YOLOv12 sign language recognition model was applied to the acquired images to identify the sign language categories.

[0094] Step 3, Data Integration Layer

[0095] Isolated sign language categories are integrated into continuous sentences and transmitted to the user interaction layer.

[0096] Step 4, User Interaction Layer

[0097] It provides real-time sign language recognition to offer thoughtful services to users.

[0098] This invention also provides a sign language recognition device based on an improved YOLOv12, which can implement the above-described method. The device includes:

[0099] The image acquisition module is used to acquire the target sign language video and convert the target sign language video into a target image;

[0100] The model building module is used to improve the YOLOv12 model through an efficient multi-scale large kernel attention mechanism and deformable dilated convolution to obtain a sign language recognition model.

[0101] The data processing and analysis module is used to input the target image into the sign language recognition model to obtain the sign language category;

[0102] The data integration module is used to integrate the sign language categories to obtain the target sign language recognition result.

[0103] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0104] This invention also provides an electronic device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including a tablet computer, an in-vehicle computer, or similar device.

[0105] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0106] refer to Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0107] The processor 501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0108] The memory 502 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 502 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501.

[0109] The input / output interface 503 is used to implement information input and output;

[0110] The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0111] Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504);

[0112] The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0113] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0114] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0115] This invention also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions to cause the computer device to perform the aforementioned method.

[0116] In summary, the sign language recognition method and apparatus based on the improved YOLOv12 according to the embodiments of the present invention have the following advantages:

[0117] To address the issue of decreased sign language recognition accuracy in complex public scenes due to factors such as background interference, target occlusion, and lighting variations, this invention reconstructs the YOLOv12 network structure using atrous convolution as its core concept. It proposes an efficient E-MLKA attention mechanism, which guides attention to salient features in the gesture region through channel splitting and multi-scale atrous convolution branches. Furthermore, it employs deformable atrous convolution fused with a C3k2 module to adaptively adjust the feature response path. This enhances the generalization ability of the sign language recognition model in complex scenes without sacrificing inference speed or introducing additional computation.

[0118] For example, in order to further improve the YOLOv12 model’s ability to express target regions and model context, a multi-scale large kernel attention mechanism (MLKA) can be introduced into the original YOLOv12 model to replace the static processing of high-level features by traditional convolutional layers. This can be used for image super-resolution tasks, while maintaining convolutional efficiency, and introducing a large receptive field and multi-scale modeling capabilities.

[0119] MLKA extracts information from different receptive fields through multi-scale feature partitioning and large-kernel convolution branches. Specifically, it involves dividing the input feature map X∈R... C×H×W Divide into n sub-features {X1, X2, ..., X} along the channel dimension n Each set of features is processed through attention paths configured at different scales. Each path computes the attention response using regular depthwise convolutions (e.g., 3×3), dilated depthwise convolutions (e.g., 5×5, dilation=2), and pointwise convolutions (1×1), i.e.:

[0120]

[0121] To further enhance dynamic control capabilities and suppress block effects caused by expansion operations, MLKA introduces a gating mechanism in each branch to modulate the attention map element-wise. The output of the i-th branch can be expressed as:

[0122] MLKA i (X i ) = G i (X i )⊙LKA i (X i )

[0123] Among them, G i (X i LKA is a gated graph generated through depthwise convolution. i (X i ) represents the big-kernel attention response of the i-th branch; ⊙ represents element-wise multiplication. All branch outputs are concatenated along the channel dimension to form the final attention output:

[0124] MLKA(X)=Concat[MLKA1(X1),…,MLKA n (X n )]

[0125] In a further alternative embodiment, an efficient and lightweight attention mechanism, E-MLKA, is proposed based on the multi-scale attention concept of MLKA. The original MLKA module achieves lightweighting through multiple Group Convolutions and Depthwise Convolutions, but this is essentially a trade-off between time cost and space efficiency. Although the computational cost is reduced, the inference time increases significantly. Furthermore, the use of DWConv and PWConv inevitably leads to a decrease in accuracy. This embodiment of the invention reduces computational overhead through E-MLKA while balancing computational efficiency and feature extraction capabilities.

[0126] In addition to enhancing deep semantic modeling capabilities through E-MLKA, this embodiment of the invention also adjusts the shallow and mid-level structures of YOLOv12. The C3k2 module in the original YOLOv12 is a highly efficient feature extraction unit that optimizes the traditional CSP Bottleneck structure. It divides the input feature map along the channel dimension into two parts: one part passes directly to retain shallow features, while the other part passes through multiple Bottleneck modules to process deep features, and finally, the two feature maps are fused and stitched together. However, when dealing with sign language in scenarios with different scales, blurred edges, or complex backgrounds, the static receptive field of C3k2 fails to cover the complete semantic region, potentially leading to missed target detections or inaccurate boundary judgments. Although dilated convolutions can expand the receptive field without increasing computation, fixed dilation settings often introduce blocking artifacts, which are detrimental to the expression of continuous features.

[0127] This invention proposes an SA_C3k2 structure that replaces standard convolution with SAConv, allowing the network to dynamically select between standard convolution and dilated convolution, flexibly expanding the receptive field, and finding a suitable balance between local details and global context. This solves the problems of missed detection and false detection in dynamic gesture continuous recognition in traditional schemes.

[0128] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0129] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0130] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0132] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0133] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0134] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0135] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0136] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A sign language recognition method based on an improved YOLOv12, characterized in that, Includes the following steps: Acquire the target sign language video and convert the target sign language video into a target image; By improving the YOLOv12 model through an efficient multi-scale large kernel attention mechanism and deformable dilated convolution, a sign language recognition model is obtained, including the following steps: The efficient multi-scale large kernel attention mechanism is obtained by setting two dilated convolutions and one standard convolution for each branch of the multi-scale large kernel attention mechanism; wherein the efficient multi-scale large kernel attention mechanism includes at least two multi-scale branches; the efficient multi-scale large kernel attention mechanism is introduced into the YOLOv12 model; The target image is input into the sign language recognition model to obtain the sign language category; The sign language categories are integrated to obtain the target sign language recognition result.

2. The method according to claim 1, characterized in that, The process of acquiring the target sign language video and converting it into a target image includes the following steps: The initial sign language video was captured using a camera connected to the Raspberry Pi. The initial sign language video is preprocessed to obtain the target sign language video; The target sign language video is converted to obtain a single-frame continuous target image.

3. The method according to claim 1, characterized in that, The improvement of the YOLOv12 model through an efficient multi-scale large kernel attention mechanism and deformable dilated convolution to obtain a sign language recognition model also includes the following steps: Combining deformable dilated convolution with the C3k2 module yields the deformable dilated convolution C3k2 module; The deformable dilated convolution C3k2 module is introduced into the YOLOv12 model.

4. The method according to claim 1, characterized in that, The step of inputting the target image into the sign language recognition model to obtain the sign language category includes the following steps: The target image is input into the backbone network of the sign language recognition model, and the backbone network features are output through multiple convolutional modules, feature extraction units, deformable dilated convolution C3k2 modules, region attention enhancement cross-feature modules, and efficient multi-scale large kernel attention mechanisms. The backbone network features are input into the neck network of the sign language recognition model, and the neck network features are output through the nearest neighbor upsampling module, the splicing module, and the region attention enhancement cross-feature module. The neck network features are input into the detection head of the sign language recognition model to obtain the sign language category.

5. The method according to claim 1, characterized in that, The process of integrating the sign language categories to obtain the target sign language recognition result includes the following steps: The discrete sign language categories are integrated in chronological order to obtain continuous initial sign language recognition results; The initial sign language recognition result is corrected for errors and supplemented with grammatical structure using natural language processing technology to obtain the target sign language recognition result.

6. A sign language recognition device based on an improved YOLOv12, characterized in that, include: The image acquisition module is used to acquire the target sign language video and convert the target sign language video into a target image; The model building module is used to improve the YOLOv12 model through an efficient multi-scale large-kernel attention mechanism and deformable dilated convolution to obtain a sign language recognition model; the model building module is specifically used for: The efficient multi-scale large kernel attention mechanism is obtained by setting two dilated convolutions and one standard convolution for each branch of the multi-scale large kernel attention mechanism; wherein the efficient multi-scale large kernel attention mechanism includes at least two multi-scale branches; the efficient multi-scale large kernel attention mechanism is introduced into the YOLOv12 model; The data processing and analysis module is used to input the target image into the sign language recognition model to obtain the sign language category; The data integration module is used to integrate the sign language categories to obtain the target sign language recognition result.

7. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images