Event-based lip-reading methods, apparatuses, devices, and storage media

Abstract

This application relates to an event-based lip reading method, apparatus, device, and storage medium. The method includes: acquiring a raw event stream of lip sequence images using an event camera; thereby asynchronously recording the brightness changes of each pixel at a microsecond-level temporal resolution, achieving ultra-low latency, high dynamic range, and sparse data representation. A voxel-based representation method converts the raw event stream into frame-like event tensors, obtaining voxelized event volumes; spatial features are extracted from the voxelized event volumes using a front-end network in the event-based lip reading model, obtaining multi-scale spatial features; temporal dependency modeling of the multi-scale spatial features is performed using a back-end sequence model in the event-based lip reading model, obtaining sequence encoding; thus, spatial and temporal features can be fused, improving the accuracy and stability of sequence encoding. The content of the lip reading is determined based on the sequence encoding. This enables stable recognition of lip movements and improves lip reading accuracy.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an event-based lip-reading method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] Event cameras, with their microsecond-level temporal resolution and sparse visual coding, offer a revolutionary paradigm for Automatic Lipreading (ALR). As an important complement to auditory speech recognition, ALR demonstrates enormous potential in complex scenarios where speech signals are unclear or severely degraded.

[0003] Most existing event-based lip-reading methods are primarily based on 3D convolutional or recurrent neural networks, focusing on temporal dynamic modeling. These methods typically follow a video-based lip-reading design paradigm, implicitly assuming that event streams provide structurally coherent and semantically complete spatiotemporal representations. However, event data differs fundamentally from traditional frame-based images; it lacks explicit spatial texture and continuous visual structure, failing to capture crucial lip structure information, leading to inaccurate lip-reading results. Summary of the Invention

[0004] Therefore, it is necessary to provide an event-based lip-reading method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve the accuracy of lip reading, addressing the aforementioned technical problems.

[0005] Firstly, this application provides an event-based lip-reading method, the method comprising:

[0006] The raw event stream of lip sequence images is acquired using an event camera;

[0007] The voxel-based representation method converts the original event stream into a frame-like event tensor, resulting in a voxelized event body.

[0008] Spatial features are extracted from the voxelized event body by the front-end network in the event-based lip reading model to obtain multi-scale spatial features;

[0009] The multi-scale spatial features are modeled temporally using a back-end sequence model in an event-based lip-reading model to obtain sequence encoding.

[0010] The content of lip reading recognition is determined based on the sequence encoding.

[0011] In one embodiment, the raw event stream of lip sequence images acquired via an event camera includes:

[0012] The event camera asynchronously records the brightness change of each pixel in the lip sequence image, and triggers an event when the logarithmic intensity change of a pixel exceeds a preset contrast threshold.

[0013] The events that occur within a preset time window are obtained to obtain the original event stream.

[0014] In one embodiment, the voxel-based representation method converts the original event stream into a frame-like event tensor to obtain a voxelized event body, including:

[0015] Based on the events in the original event stream and the given time interval, the event timestamps are scaled to the given time interval using a voxel grid method to obtain a voxelized event body.

[0016] In one embodiment, the front-end network includes a 3D convolutional layer and a deep neural network. The front-end network in the event-based lip-reading model extracts spatial features from the voxelized event volume to obtain multi-scale spatial features, including:

[0017] The initial spatiotemporal features are obtained by performing preliminary spatiotemporal feature extraction on the voxelized event body through the 3D convolutional layer.

[0018] The initial spatiotemporal features are sequentially input into the convolutional layer and multiple residual blocks of the deep neural network. The outputs of the multiple residual blocks are processed by global average pooling to obtain a high-level spatial feature map. The outputs of the multiple residual blocks are processed by shallow adaptive pooling to obtain multiple spatial feature maps.

[0019] Using the intermediate layer feature map of multiple spatial feature maps as the alignment reference resolution, the spatial dimension of multi-level features is reshaped into a sequence of node representations for graph construction;

[0020] The node representation sequence is input in parallel into a frequency-aware modulation module for frequency enhancement processing to obtain enhanced features; the frequency enhancement processing includes: injecting noise into the first frequency band, and using adaptive frequency filtering to enhance the discriminative edge dynamics of the second frequency band; the lower limit frequency of the second frequency band is higher than the upper limit frequency of the first frequency band;

[0021] The enhanced features are input into the fully connected layer of the deep neural network to obtain multi-level spatial node features;

[0022] The multi-level spatial node features are input into the spatial region hypermap to extract the high-order intra-frame dependencies between lip regions.

[0023] Multi-scale spatial features are obtained based on the high-order intra-frame dependencies between lip regions and the high-level spatial feature map.

[0024] In one embodiment, the multi-scale spatial features are modeled temporally using a back-end sequence model in an event-based lip-reading model to obtain sequence encoding, including:

[0025] The multi-scale spatial features are transformed into temporally structured features using a temporal semantic hypergraph.

[0026] Sequence encoding is obtained by modeling based on the time-structured features using a recurrent neural network.

[0027] In one embodiment, determining the content of lip reading recognition based on the sequence encoding includes:

[0028] The content of the lip reading is determined by the target lip reading model based on the sequence encoding; wherein the target lip reading model adopts a spectrogram-based supervision strategy, which is used to encode visual similarity into a label space.

[0029] In one embodiment, the view-based supervision strategy includes:

[0030] The normalized word tags are associated with phoneme sequences, and each phoneme sequence is converted into a corresponding visual sequence according to the preset phoneme-to-visual mapping rules.

[0031] Edit distance is used to quantify the similarity between the visual sequences of different words; where the minimum visual edit distance between any two different words is 1, and the smaller the visual edit distance, the more similar the words are in pronunciation.

[0032] A distance matrix is ​​constructed by calculating the editing distance between pairs of pixels;

[0033] The distance matrix is ​​converted into a probability distribution by row-wise normalization.

[0034] Soft labels are created for each word based on the probability distribution.

[0035] Secondly, this application also provides an event-based lip-reading device, the device comprising:

[0036] The acquisition module is used to acquire the raw event stream of lip sequence images via the event camera;

[0037] The conversion module is used to convert the original event stream into a frame-like event tensor based on a voxel-based representation method, thereby obtaining a voxelized event body;

[0038] The feature extraction module is used to extract spatial features from the voxelized event body through the front-end network in the event-based lip reading model to obtain multi-scale spatial features.

[0039] The time modeling module is used to perform time-dependent modeling of the multi-scale spatial features through the back-end sequence model in the event-based lip-reading model to obtain sequence encoding;

[0040] The recognition module is used to determine the content of lip reading recognition based on the sequence encoding.

[0041] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0042] The raw event stream of lip sequence images is acquired using an event camera;

[0043] The voxel-based representation method converts the original event stream into a frame-like event tensor, resulting in a voxelized event body.

[0044] Spatial features are extracted from the voxelized event body by the front-end network in the event-based lip reading model to obtain multi-scale spatial features;

[0045] The multi-scale spatial features are modeled temporally using a back-end sequence model in an event-based lip-reading model to obtain sequence encoding.

[0046] The content of lip reading recognition is determined based on the sequence encoding.

[0047] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0048] The raw event stream of lip sequence images is acquired using an event camera;

[0049] The voxel-based representation method converts the original event stream into a frame-like event tensor, resulting in a voxelized event body.

[0050] Spatial features are extracted from the voxelized event body by the front-end network in the event-based lip reading model to obtain multi-scale spatial features;

[0051] The multi-scale spatial features are modeled temporally using a back-end sequence model in an event-based lip-reading model to obtain sequence encoding.

[0052] The content of lip reading recognition is determined based on the sequence encoding.

[0053] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0054] The raw event stream of lip sequence images is acquired using an event camera;

[0055] The voxel-based representation method converts the original event stream into a frame-like event tensor, resulting in a voxelized event body.

[0056] Spatial features are extracted from the voxelized event body by the front-end network in the event-based lip reading model to obtain multi-scale spatial features;

[0057] The multi-scale spatial features are modeled temporally using a back-end sequence model in an event-based lip-reading model to obtain sequence encoding.

[0058] The content of lip reading recognition is determined based on the sequence encoding.

[0059] The aforementioned event-based lip reading method, apparatus, computer device, computer-readable storage medium, and computer program product acquire the raw event stream of lip movement sequence images through an event camera; thus, it can asynchronously record the brightness changes of each pixel with microsecond-level temporal resolution, achieving ultra-low latency, high dynamic range, and sparse data representation. The voxel-based representation method converts the raw event stream into frame-like event tensors, obtaining voxelized event volumes; this allows the event data to be converted into a video-like representation, facilitating subsequent feature extraction processing. The front-end network in the event-based lip reading model extracts spatial features from the voxelized event volumes, obtaining multi-scale spatial features; the back-end sequence model in the event-based lip reading model models the temporal dependencies of these multi-scale spatial features, obtaining sequence encoding; this allows for the fusion of spatial and temporal features, improving the accuracy and stability of sequence encoding. The content of the lip reading is determined based on the sequence encoding. This enables stable recognition of lip movements, improving lip reading accuracy. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart illustrating an event-based lip-reading method in one embodiment;

[0062] Figure 2 for Figure 1 A flowchart illustrating step 103 in the embodiment;

[0063] Figure 3(a) is a standard diagram in one embodiment;

[0064] Figure 3(b) shows a hypergraph in one embodiment;

[0065] Figure 4 This is a schematic diagram of the architecture of an event-based lip-reading model in one embodiment;

[0066] Figure 5 This is a schematic diagram comparing the pixel-based supervision strategy with the traditional one-heat supervision strategy.

[0067] Figure 6 This is a structural block diagram of an event-based lip-reading device in one embodiment;

[0068] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0070] It should be noted that the terms "comprising" and "having," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusion. The term "multiple" as used in this application refers to two or more. The term "and / or" as used in this application refers to one of the solutions, or any combination of multiple solutions.

[0071] To facilitate understanding of the technical solutions in the various embodiments of this application, the following explanations are provided for the technical terms that may appear in the embodiments of this application:

[0072] An event-based camera (or simply event camera) is a new type of sensor used to detect changes in the brightness of a pixel. When the cumulative brightness change of a pixel reaches a certain threshold, it outputs an event. The output of an event camera is related to the brightness change, but not to the absolute value of the brightness. An event indicates at what time and which pixel experienced an increase or decrease in brightness.

[0073] Voxel representation is a 3D modeling technique in computer graphics that decomposes an object into a series of voxels, or three-dimensional pixels, similar to pixels in a 2D image. Each voxel represents a small cube in space, and the set of these voxels represents the shape and structure of a 3D object.

[0074] Multi-scale spatial features: First, the scale of an image does not refer to its size, but rather to its degree of blur. For example, the degree of blurriness differs when viewing an object up close versus from a distance; the process of an image becoming increasingly blurry from near to far is also a process of the image scale increasing. Multi-scale spatial features refer to features extracted at different spatial scales. For example, by analyzing signals using filters of different scales, feature information at different scales can be obtained to comprehensively describe the image or signal.

[0075] Lip reading model: refers to a model used to recognize lip movements. It can process video and audio information and identify the sounds made by the speaker by fusing the two modalities, thereby improving the accuracy of lip reading recognition.

[0076] Sequence encoding: This is an important step in lip reading models for lip recognition. For example, an image sequence encoding containing spatiotemporal features can be input into a lip reading model, and the model can output the content corresponding to the image sequence encoding.

[0077] Event cameras offer a revolutionary paradigm for automatic lip reading due to their microsecond-level temporal resolution and sparse visual coding. However, event data itself lacks a clear spatial structure and exhibits significant frequency domain bias. Low-frequency components fail to capture crucial lip structure information, fundamentally hindering the modeling of intra-frame topological dependencies and inter-frame semantic evolution, both of which are essential for robust lip reading.

[0078] To address the problems existing in the prior art, this application aims to provide an event-based lip-reading method. This method employs an event-based lip-reading model (a frequency-aware spatiotemporal hypergraph framework), which can improve the robustness of the model in capturing discriminative features and integrate adaptive high-frequency filtering to enhance edge-aware representation. Optionally, this application also constructs a spatial region hypergraph and a temporal semantic hypergraph. The former is used to capture intra-frame topological dependencies between lip regions, while the latter explicitly models inter-frame structural associations throughout the entire lip movement process, enabling the model to capture discriminative patterns in lip dynamics. Furthermore, this application also provides a pixel-based label smoothing strategy, which can use pixel-level edit distance to quantify visual similarity between categories and guide the construction of soft labels.

[0079] In one exemplary embodiment, such as Figure 1 As shown, an event-based lip-reading method is provided, which may include the following steps 101 to 105. Wherein:

[0080] Step 101: Acquire the raw event stream of lip sequence images using the event camera.

[0081] It should be understood that the method in this embodiment can be applied to automatic lip reading scenarios, such as hearing aids, smart cockpits, human-computer interaction, and security monitoring.

[0082] For example, the brightness change of each pixel in the lip sequence image can be recorded asynchronously by an event camera, and an event can be triggered when the logarithmic intensity change of the pixel exceeds a preset contrast threshold; the events that occur within a preset time window are obtained to get the original event stream.

[0083] The formula for calculating the preset contrast threshold is as follows:

[0084]

[0085] In the formula: Indicates scene brightness intensity. This indicates the polarity of the brightness change (p=1 indicates an increase, p=-1 indicates a decrease). It is a predefined contrast threshold. This represents the time elapsed since the last event was triggered at position (x, y). Therefore, the output of the event camera can be represented as a stream of events occurring within the time window [T0, T1], calculated as follows:

[0086]

[0087] In the formula: Represents a set of event streams. This represents the k-th event. This represents the x-coordinate of the event in pixels in space. Indicates the timestamp of the event. This represents the pixel ordinate of the event in space. The polarity of the event (i.e., the direction of brightness change) is represented by N, which represents the total number of events in the event stream, i.e., the number of events collected within the time period [T0, T1].

[0088] Step 102: The original event stream is converted into a frame-like event tensor using a voxel-based representation method to obtain a voxelized event body.

[0089] For example, based on the events in the original event stream and a given time interval, the voxel grid method can be used to scale the event timestamps to the given time interval range to obtain a voxelized event body.

[0090] In this embodiment, a voxel-based representation method is used to convert the original event stream into a frame-like event tensor to preserve as much motion information as possible. Optionally, given a set of N input events:

[0091] Given a time interval count T, the event timestamps are first scaled to the range [0, T−1] using a voxel grid method, and then a T × H × W voxelized event body is generated. :

[0092]

[0093]

[0094] In the formula: This represents the normalized time index, where T × H × W represent the height and width in the time and spatial dimensions of the event data, respectively. Let t1 represent the Nth timestamp, and t1 represent the 1st timestamp. This represents the voxel intensity at position (x, y) and time t in the voxel grid.

[0095] After voxelization, the event data is transformed into a video-like representation X. , where T is the number of event frames, and the channel dimension C=1 indicates that events of both polarities are accumulated in each voxel frame.

[0096] Step 103: Extract spatial features from the voxelized event body using the front-end network in the event-based lip reading model to obtain multi-scale spatial features.

[0097] The front-end network can include 3D convolutional layers and deep neural networks.

[0098] Optionally, preliminary spatiotemporal feature extraction is performed on the input event frame X, and then deeper semantic features are further extracted using ResNet-18.

[0099] For example, the front-end network can employ 3D convolutional layers with a kernel size of 5 × 7 × 7 and connect to the ResNet-18 backbone network to extract spatial features from voxelized event frames.

[0100] Step 104: The multi-scale spatial features are modeled for temporal dependence using the back-end sequence model in the event-based lip reading model to obtain the sequence code.

[0101] For example, the multi-scale spatial features can be converted into temporally structured features using a temporal semantic hypergraph; and a sequence encoding can be obtained by modeling based on the temporally structured features using a recurrent neural network.

[0102] In this embodiment, during the time modeling stage, a time-structured feature representation is obtained through a time semantic hypergraph. :

[0103]

[0104] In the formula: This indicates the use of the temporal semantic hypergraph network HGNN. TSH Extracted temporal features.

[0105] Optionally, a bidirectional gated recurrent unit (BiGRU) is used to model the time dependency and generate the final sequence encoding representation. :

[0106]

[0107] In the formula: This indicates that a bidirectional gated loop unit is used to model time dependencies.

[0108] In this embodiment, the temporal semantic hypergraph can explicitly capture the semantic evolution between frames, thereby enabling overall spatiotemporal representation learning.

[0109] Step 105: Determine the content of lip reading recognition based on sequence encoding.

[0110] In this embodiment, the content of the lip reading can be determined by the target lip reading model based on the sequence encoding; wherein, the target lip reading model adopts a spectrogram-based supervision strategy, which is used to encode visual similarity into a label space.

[0111] Optionally, the visual-based supervision strategy includes: associating normalized word tags with phoneme sequences, and converting each phoneme sequence into a corresponding visual sequence according to a preset phoneme-to-visual mapping rule; quantifying the similarity between visual sequences of different words using edit distance; wherein the minimum visual edit distance between any two different words is 1, and the smaller the visual edit distance, the more similar the words are in pronunciation; constructing a distance matrix by calculating pairwise visual edit distances; converting the distance matrix into a probability distribution by row-wise normalization; and establishing soft tags for each word based on the probability distribution.

[0112] For example, such as Figure 5 As shown, a comparison is made between the pixel-based supervision strategy and the traditional one-heat supervision strategy. The pixel-based supervision strategy preserves the inter-class pixel similarity and achieves a smoother label transition compared to the traditional one-heat supervision strategy.

[0113] In this embodiment, word stress marks (e.g., "AH0" → and "ah") are removed, retaining only basic phoneme units to obtain normalized word labels. Each word label is associated with a clean phoneme sequence. Each phoneme sequence is converted into its corresponding visual sequence. Assume the visual sequences of two given words are represented as a = [a1, a2, ..., a...]. p ] and b = [b1, b2, ..., b q ], where p and q represent the number of visual elements in each sequence. Levenshtein distance is introduced to quantify the similarity between visual element sequences of different words. This metric is widely used in string matching to calculate the minimum number of editing operations (insertion, deletion, and replacement) required to transform one sequence into another. Given two visual element sequences a = [a1, a2, ..., a...], ... p ] and b = [b1, b2, ..., b q The edit distance at the pixel level is defined as:

[0114]

[0115] Where p = |a| and q = |b|.

[0116] In the formula: This represents the minimum number of edit operations required to transform the first p terms of sequence a into the first q terms of sequence b. Indicates an indicator function, if If true, return 1; otherwise, return 0. To ensure the model can effectively distinguish different lip pronunciation patterns, the minimum visual edit distance between any two different words is required to be 1. The smaller the value, the more similar the two words are phonetically, and therefore the easier they are to confuse in lip-reading tasks. After calculating the pairwise view-edit distances, a distance matrix can be obtained. Each element Dlev(i,j) represents the minimum number of edit operations required to transform the view sequence of word i into the view sequence of word j. These distances are then converted into a probability distribution using row-wise Softmax normalization. Specifically, the soft label for the i-th word is determined by... Given that the smaller the distance, the higher the probability.

[0117] In the formula: This represents the soft label of the i-th category, and softmax() represents converting the edit distance matrix into a probability distribution. This refers to the temperature parameter that controls the sharpness of the soft label distribution. When... When the value is smaller, the resulting probability distribution becomes sharper, causing the model to focus more on categories highly similar to the target word. Conversely, when... When the size is large, the distribution becomes smoother, which helps the model capture a wider range of inter-class visual similarities.

[0118] In this embodiment, the pixel-based supervision strategy utilizes pixel-level edit distance to encode visual similarity into the label space, thereby constructing soft labels. The pixel-based supervision strategy does not treat all non-target categories as equally negative; instead, it introduces structured supervision signals to encourage the model to learn visually coherent feature representations, thus achieving clearer inter-category separation.

[0119] In the aforementioned event-based lip reading method, the raw event stream of lip sequence images is acquired through an event camera. This allows for asynchronous recording of brightness changes for each pixel at a microsecond-level temporal resolution, achieving ultra-low latency, high dynamic range, and sparse data representation. The voxel-based representation method converts the raw event stream into frame-like event tensors, resulting in voxelized event volumes. This allows the event data to be converted into a video-like representation, facilitating subsequent feature extraction. The front-end network in the event-based lip reading model extracts spatial features from the voxelized event volumes, obtaining multi-scale spatial features. The back-end sequence model in the event-based lip reading model models the temporal dependencies of these multi-scale spatial features, resulting in sequence encoding. This allows for the fusion of spatial and temporal features, improving the accuracy and stability of the sequence encoding. The content to be recognized by lip reading is determined based on the sequence encoding. This enables stable recognition of lip movements and improves lip reading accuracy.

[0120] In one exemplary embodiment, such as Figure 2 As shown, step 103 includes steps 1031 to 1037. Wherein:

[0121] Step 1031: Perform preliminary spatiotemporal feature extraction processing on the voxelized event body through the 3D convolutional layer to obtain initial spatiotemporal features.

[0122] In this embodiment, a 3D convolutional layer with a kernel size of 5×7×7 can be used and connected to the ResNet-18 backbone network.

[0123] Step 1032: The initial spatiotemporal features are sequentially input into the convolutional layer and multiple residual blocks of the deep neural network. The outputs of the multiple residual blocks are processed by global average pooling to obtain a high-level spatial feature map. The outputs of the multiple residual blocks are processed by shallow adaptive pooling to obtain multiple spatial feature maps.

[0124] In this embodiment, we take the initial spatiotemporal feature extraction of the input event frame X as an example, and then further extract deeper semantic features using ResNet-18. The deeper semantic representation captured by the ResNet-18 backbone network is as follows:

[0125]

[0126] in, Indicates the first Each residual block The output, This represents multi-level spatiotemporal features. The final residual block output undergoes global average pooling to obtain spatial feature maps from the ResNet. The calculation formula is as follows:

[0127]

[0128] These four residual blocks encode hierarchical semantic information with progressively decreasing spatial resolution. To align them spatially, adaptive pooling is applied to the shallow features, and bilinear interpolation is used to upsample the deeper features, ultimately resulting in a uniform spatial resolution. :

[0129]

[0130] In the formula: This represents the alignment result of the spatial feature map of the i-th layer. This indicates that average pooling is performed on the shallow feature map. This indicates that bilinear interpolation is performed on deep features.

[0131] Step 1033: Using the intermediate layer feature map of multiple spatial feature maps as the alignment reference resolution, the spatial dimension of the multi-level features is reshaped into a sequence of node representations for graph construction.

[0132] For example, the third-layer feature map can be chosen as the reference resolution for alignment because it achieves a good balance between spatial granularity and computational efficiency. The spatial dimensions of the fourth-level features are reshaped into a sequence of node representations suitable for graph construction, denoted as... ,in It represents the total number of spatial nodes.

[0133] Step 1034: The node representation sequence is input in parallel into the frequency sensing modulation module for frequency enhancement processing to obtain the enhanced features.

[0134] The frequency enhancement process includes: injecting noise into the first frequency band and using adaptive frequency filtering to enhance the discriminative edge dynamics of the second frequency band; the lower limit frequency of the second frequency band is higher than the upper limit frequency of the first frequency band.

[0135] For example, spatial node feature sequences from four semantic levels are input in parallel to the frequency-aware modulation module. In the perturbation branch, noise is selectively injected into specific frequency bands to improve feature robustness, while the adaptive filtering branch emphasizes information-rich frequency components.

[0136] In this embodiment, the frequency-aware modulation module compensates for the missing spatial structure in the event representation by injecting low-frequency disturbances, and integrates an adaptive frequency filtering mechanism to enhance the dynamic encoding of lip contours and motion trajectories.

[0137]

[0138] In the formula: Indicates low-frequency characteristics. Indicates the filtered features. This indicates the low-frequency characteristics after adding noise perturbation. This represents the features after adaptive filtering.

[0139] Step 1035: Input the enhanced features into the fully connected layer of the deep neural network to obtain multi-level spatial node features.

[0140] For example, the enhanced features of all four layers are connected to form a multi-level spatial node representation, which is represented as follows: and The specific calculation formula is as follows:

[0141]

[0142] in, , Indicates the combined channel dimension.

[0143] Step 1036: Input the multi-level spatial node features into the spatial region hypergraph respectively, and extract the high-order intra-frame dependencies between the lip regions.

[0144] and The data are input into the spatial region hypermap to extract high-order intra-frame dependencies between lip regions. :

[0145]

[0146] In the formula: This indicates the use of a spatial region hypergraph neural network for input features. The output result after processing.

[0147] In this embodiment, a spatial region hypergraph can be used to model higher-order topological dependencies between intra-frame lip regions.

[0148] For example, Figures 3(a) and 3(b) illustrate a comparison between standard graph representations and hypergraph representations. Specifically, at the speckle level, the standard graph representation and hypergraph representation of the word "methodology" are compared. In Figure 3(a), the neural network captures pairwise relationships, while in Figure 3(b), the hypergraph neural network can simulate higher-order group interactions between multiple speckle subwords.

[0149] It should be understood that hypergraphs extend traditional graphs by allowing hyperedges to connect more than two nodes, making them well-suited for modeling higher-order relationships. As shown in Figure 3(b), hypergraph neural networks can model group interactions compared to traditional neural networks that only capture pairwise relationships. In recommender systems, hypergraph structures are used to overcome oversmoothness and sparsity. Applying coarse-to-fine hypergraph modeling to action detection can capture multi-object temporal relationships.

[0150] Optionally, by constructing a spatial region hypergraph within each frame and a temporal semantic hypergraph between frames, it is possible to explicitly model the higher-order structural dependencies of the lip region in both spatial and temporal dimensions.

[0151] It should be understood that robust lip movement modeling can be achieved in event-based lip reading models by combining frequency-aware feature refinement with hypergraph-based dependency reasoning.

[0152] Step 1037: Based on the high-order intra-frame dependencies between lip regions and the high-level spatial feature map, obtain multi-scale spatial features.

[0153] In this embodiment, With advanced spatial features Connecting them together yields a comprehensive multi-scale spatial representation. :

[0154]

[0155] For example, Figure 4 This is a schematic diagram of the architecture of an event-based lip-reading model in one embodiment, such as... Figure 4As shown, the input event frame first undergoes preliminary spatiotemporal feature extraction through a 3D convolutional layer. The extracted spatiotemporal features are then sequentially input into four residual blocks. The output of the first residual block is split into two branches: one branch is transmitted to the second residual block, and the other to the first frequency-aware multi-branch module. The output of the second residual block is also split into two branches: one branch is transmitted to the third residual block, and the other to the second frequency-aware multi-branch module. The output of the third residual block is split into two branches: one branch is transmitted to the fourth residual block, and the other to the third frequency-aware multi-branch module. The output of the fourth residual block is split into two branches: one branch connects to the output of the temporal semantic hypergraph and is then transmitted to a Bidirectional Gated Recurrent Unit (BiGRU) before being sent to the classifier; the other branch is transmitted to the fourth frequency-aware multi-branch module.

[0156] like Figure 4 As shown, the outputs of the four frequency-aware multi-branch modules are each divided into two branches, denoted as the first branch and the second branch, respectively. The first branches of the first, second, third, and fourth frequency-aware multi-branch modules are concatenated and input into the first spatial region hypergraph. The second branches of the first, second, third, and fourth frequency-aware multi-branch modules are concatenated and input into the second spatial region hypergraph. The outputs of the two spatial region hypergraphs are then concatenated and transmitted to the temporal semantic hypergraph.

[0157] like Figure 4 As shown, the frequency-aware multi-branch module can include: selective frequency routing and a low-frequency disturbance branch. Data entering the selective frequency routing branch undergoes sequential processing via Fast Fourier Transform (FFT), a learnable filter, and an Inverse Fast Fourier Transform (IFT). Data entering the low-frequency disturbance branch undergoes sequential processing via FFT, event-driven low-frequency disturbance, and an IFT.

[0158] Optionally, the low-frequency perturbation branch injects controlled perturbations into low-frequency components to compensate for missing structural semantics, particularly lip topology. Selective frequency routing adaptively emphasizes information-rich high-frequency features, such as lip contours and motion boundaries, while suppressing noise response.

[0159] For example, for an event sequence containing T frames, each frame contains The spatial node will be the first The node features of a layer are represented as follows:

[0160]

[0161] In the formula: This represents the channel feature vector of the nth spatial node at time t.

[0162] To obtain the frequency domain representation, along each Apply one-dimensional fast Fourier transform to the channel dimension of the node:

[0163]

[0164] In the formula: This represents the input features of node i in the nth branch of frame t. The discrete Fourier transform result for the c-th channel, This represents the value of the k-th channel in the time-domain signal, where k represents the time-domain channel index and c represents the frequency-domain channel index.

[0165] In the frequency enhancement branch, low-frequency components are extracted by applying a centrosymmetric binary mask on the spectrum:

[0166]

[0167] in, This represents a low-pass filter mask used to preserve low-frequency components. This indicates the proportion of the spectrum allocated to the low-frequency region. A binary mask is applied to the frequency representation to retain only the low-frequency components:

[0168]

[0169] in, This represents the low-frequency portion after masking. This indicates element-wise multiplication.

[0170] Next, the low-frequency spectrum of each node is modeled as a Gaussian distribution, and a controllable perturbation is introduced on this basis:

[0171]

[0172] in, This represents the low-frequency component after the disturbance. This represents the intensity of the disturbance sampled from a uniform distribution. Representing the corresponding frequency standard deviation Indicates uniform distribution. Hyperparameters that represent the range of control disturbances.

[0173] The perturbed frequency domain features are then transformed back to the original frequency domain using an inverse fast Fourier transform, and then node-level features are extracted using a multilayer perceptron (MLP).

[0174]

[0175] Wherein: IFFT represents Inverse Fast Fourier Transform.

[0176] Unlike the low-frequency band, the selective frequency routing branch directly performs adaptive filtering on the frequency domain features obtained through the Fast Fourier Transform. A learnable filter W is introduced to modulate the spectral representation of each node in the frequency space:

[0177]

[0178] in: This represents the channel characteristics (high-frequency selection) after weighted filtering in the frequency domain. This represents the learnable frequency channel weights. This represents the result of performing a Fourier transform on the feature z of node i, time frame t, and frequency branch n, i.e., entering the frequency domain space.

[0179] Finally, the filtered frequency domain features are also transformed back to the spatial domain using IFFT, and then processed by MLP to obtain node representations from the selective frequency routing branch:

[0180]

[0181] in: This represents the node characteristics of node i in frame t, on the nth branch, after frequency channel selection and returning to the time domain.

[0182] After passing through the frequency-aware multi-branch module, each frame obtains two different spatial node representations:

[0183]

[0184]

[0185] It should be understood that Emphasizing stable low-frequency structural patterns in node characteristics, After suppressing background interference, the model focuses on significant frequency domain information, thereby enhancing its sensitivity to key motion regions. These two representations are input in parallel to the spatiotemporal hypergraph module to capture richer spatiotemporal dependencies and higher-order semantic relationships between nodes.

[0186] It should be understood that, in order to explicitly model the spatiotemporal semantic structure in event data, we construct two types of hypergraphs: spatial region hypergraphs and temporal semantic hypergraphs. Spatial region hypergraphs capture high-order topological relationships between inner lip regions within each frame, while temporal semantic hypergraphs encode semantic evolution across frames.

[0187] For example, a spatial region hypergraph is defined as ,in This represents the set of all spatial nodes within the framework. Let represent the set of hyperedges used to model the higher-order structural relationships between these nodes. For each node, the Euclidean distance to all other nodes is calculated, and its k nearest neighbors are selected. Then, by connecting the node to its k neighbors to form hyperedges, local spatial dependencies are effectively captured. The resulting spatial region hypergraph can be encoded as a matrix of k nearest neighbors:

[0188]

[0189] in, This represents a hypergraph adjacency matrix based on a constructed hypergraph, which applies hypergraph convolution to capture higher-order structural dependencies between spatial nodes:

[0190]

[0191] in: The node feature representation of the l-th layer of the temporal semantic hypergraph. This represents the activation function. The degree matrix representing the nodes. Represents the transpose adjacency matrix of a hypergraph. Represents the hyperedge weight matrix. The degree matrix represents the hyperedge. This represents the node-hyperedge incidence matrix. This represents the spatial feature representation of the input at the l-th layer. This represents the learnable spatial hypergraph convolution parameter matrix.

[0192] Average pooling is applied to the spatial region hypergraph node features from the low-frequency perturbation branch and the selective frequency routing branch. After T layers of spatial hypergraph convolution, the outputs of all layers are concatenated to obtain the spatial feature representation of the entire event sequence, as follows:

[0193]

[0194]

[0195] in, This represents the global spatial feature representation of the low-frequency branch after spatial hypergraph modeling. This represents the spatial feature representation of the frequency-selective branch after spatial hypergraph modeling. This represents the output channel dimension of the spatial hypergraph module. Then, these two representations are compared with baseline convolutional features. Connecting them along the time axis T yields a multi-scale spatial feature representation that integrates convolutional and hypergraph-enhanced features.

[0196] After extracting spatial features through a spatial region hypergraph, a temporal semantic hypergraph was constructed as a backend temporal modeling module.

[0197] The temporal semantic hypergraph is defined as follows: .

[0198] in, This represents the set of nodes in the temporal hypergraph, where each node represents a frame-level spatial representation. This represents the hyperframe set in the temporal hypergraph, where each hyperedge connects a frame node to its temporal neighbor. The hyperedges in the model are constructed by calculating the Euclidean distance between each frame and all other frames, and connecting each frame to its k nearest temporal neighbors.

[0199] The temporal hypergraph convolution formula is as follows:

[0200]

[0201] in, This represents the feature representation of the frame nodes after the (l+1)th layer temporal hypergraph convolution. This represents the normalization term of the node degree matrix. This represents the normalization term of the hypermarginality matrix. This represents the time node features of the input at the l-th layer.

[0202] To further preserve the original temporal order between frames, residual connections are introduced, adding the output of the temporal hypergraph convolution to its input. The obtained feature representation is then input into the novel Conditional Image Generation Model (BiGR) module for global temporal modeling.

[0203] After testing on multiple challenging datasets, the event-based lip-reading model proposed in this application demonstrates significant advantages in identifying confused words. This model outperforms existing technologies in both overall accuracy and the ability to discriminate confused subsets, effectively improving the accuracy and robustness of lip-reading systems when processing visually similar word pairs, thereby enhancing the model's practicality and reliability.

[0204] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0205] Based on the same inventive concept, this application also provides an event-based lip-reading device for implementing the event-based lip-reading method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more event-based lip-reading device embodiments provided below can be found in the limitations of the event-based lip-reading method described above, and will not be repeated here.

[0206] In one exemplary embodiment, such as Figure 6 As shown, an event-based lip-reading device is provided, comprising: a data acquisition module 601, a conversion module 602, a feature extraction module 603, a time modeling module 604, and a recognition module 605, wherein:

[0207] Acquisition module 601 is used to acquire the raw event stream of lip sequence images through an event camera;

[0208] The conversion module 602 is used to convert the original event stream into a frame-like event tensor based on a voxel-based representation method to obtain a voxelized event body;

[0209] The feature extraction module 603 is used to extract spatial features from the voxelized event body through the front-end network in the event-based lip reading model to obtain multi-scale spatial features.

[0210] The time modeling module 604 is used to perform time-dependent modeling of the multi-scale spatial features through the back-end sequence model in the event-based lip reading model to obtain sequence encoding;

[0211] The recognition module 605 is used to determine the content of lip reading recognition based on the sequence encoding.

[0212] For example, the acquisition module 601 is specifically used to: asynchronously record the brightness change of each pixel of the lip sequence image through the event camera, and trigger an event when the logarithmic intensity change of the pixel exceeds a preset contrast threshold; acquire the events that occur within a preset time window to obtain the original event stream.

[0213] For example, the conversion module 602 is specifically used to scale the event timestamps to the given time interval range using a voxel grid method based on the events in the original event stream and the given time interval, so as to obtain a voxelized event body.

[0214] For example, the front-end network includes: a 3D convolutional layer and a deep neural network. The feature extraction module 603 is specifically used for: performing preliminary spatiotemporal feature extraction processing on the voxelized event body through the 3D convolutional layer to obtain initial spatiotemporal features; sequentially inputting the initial spatiotemporal features into the convolutional layer and multiple residual blocks of the deep neural network; obtaining a high-level spatial feature map after global average pooling of the outputs of the multiple residual blocks; obtaining multiple spatial feature maps after shallow adaptive pooling of the outputs of the multiple residual blocks; and reshaping the spatial dimension of the multi-level features into a node representation of a graph using the intermediate layer feature map of the multiple spatial feature maps as the alignment reference resolution. The sequence of nodes is input in parallel into a frequency-aware modulation module for frequency enhancement processing to obtain enhanced features. The frequency enhancement processing includes: injecting noise into the first frequency band and using adaptive frequency filtering to enhance the discriminative edge dynamics of the second frequency band; the lower limit frequency of the second frequency band is higher than the upper limit frequency of the first frequency band; the enhanced features are input into the fully connected layer of the deep neural network to obtain multi-level spatial node features; the multi-level spatial node features are input into the spatial region hypermap to extract the high-order intra-frame dependencies between lip regions; based on the high-order intra-frame dependencies between lip regions and the high-level spatial feature map, multi-scale spatial features are obtained.

[0215] For example, the time modeling module 604 is specifically used to: convert the multi-scale spatial features into time-structured features through a time semantic hypergraph; and to model the sequence encoding based on the time-structured features through a recurrent neural network.

[0216] For example, the recognition module 605 is specifically used to: determine the content of the lip reading based on the sequence encoding by the target lip reading model; wherein the target lip reading model adopts a spectrogram-based supervision strategy, and the spectrogram-based supervision strategy is used to encode visual similarity into a label space.

[0217] For example, the visual-based supervision strategy includes: associating normalized word tags with phoneme sequences, and converting each phoneme sequence into a corresponding visual sequence according to a preset phoneme-to-visual mapping rule; quantifying the similarity between visual sequences of different words by using edit distance; wherein the minimum visual edit distance between any two different words is 1, and the smaller the visual edit distance, the more similar the words are in pronunciation; constructing a distance matrix by calculating pairwise visual edit distances; converting the distance matrix into a probability distribution by row-wise normalization; and establishing soft tags for each word based on the probability distribution.

[0218] The modules in the aforementioned event-based lip-reading device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the corresponding operations of each module.

[0219] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements an event-based lip-reading method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0220] Those skilled in the art will understand that Figure 7The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0221] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0222] The event stream is acquired by capturing lip sequence images using an event camera; the original event stream is converted into frame-like event tensors using a voxel-based representation method to obtain voxelized event volumes; spatial features are extracted from the voxelized event volumes using a front-end network in an event-based lip reading model to obtain multi-scale spatial features; temporal dependency modeling is performed on the multi-scale spatial features using a back-end sequence model in the event-based lip reading model to obtain sequence encoding; and the content of lip reading recognition is determined based on the sequence encoding.

[0223] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0224] The brightness changes of each pixel in the lip sequence image are asynchronously recorded by an event camera, and an event is triggered when the logarithmic intensity change of a pixel exceeds a preset contrast threshold; the events that occur within a preset time window are obtained to obtain the original event stream.

[0225] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0226] Based on the events in the original event stream and the given time interval, the event timestamps are scaled to the given time interval using a voxel grid method to obtain a voxelized event body.

[0227] In one embodiment, the front-end network includes: 3D convolutional layers and a deep neural network, and the processor, when executing a computer program, further implements the following steps:

[0228] The voxelized event body is subjected to preliminary spatiotemporal feature extraction processing through the 3D convolutional layer to obtain initial spatiotemporal features. These initial spatiotemporal features are then sequentially input into the convolutional layer and multiple residual blocks of the deep neural network. The outputs of these residual blocks undergo global average pooling to obtain high-level spatial feature maps. The outputs of these residual blocks then undergo shallow adaptive pooling to obtain multiple spatial feature maps. Using the intermediate feature maps of these spatial feature maps as an alignment reference resolution, the spatial dimensions of the multi-level features are reshaped into a sequence of node representations for graph construction. This sequence of node representations is then input in parallel into a frequency-aware system. Frequency enhancement processing is performed in the modulation module to obtain enhanced features. The frequency enhancement processing includes: injecting noise into the first frequency band and using adaptive frequency filtering to enhance the discriminative edge dynamics of the second frequency band; the lower limit frequency of the second frequency band is higher than the upper limit frequency of the first frequency band; the enhanced features are input into the fully connected layer of the deep neural network to obtain multi-level spatial node features; the multi-level spatial node features are input into the spatial region hypermap to extract the high-order intra-frame dependencies between lip regions; based on the high-order intra-frame dependencies between lip regions and the high-level spatial feature map, multi-scale spatial features are obtained.

[0229] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0230] The multi-scale spatial features are transformed into temporally structured features using a temporal semantic hypergraph.

[0231] Sequence encoding is obtained by modeling based on the time-structured features using a recurrent neural network.

[0232] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0233] The content of the lip reading is determined by the target lip reading model based on the sequence encoding; wherein the target lip reading model adopts a spectrogram-based supervision strategy, which is used to encode visual similarity into a label space.

[0234] In one embodiment, the visual-based supervision strategy includes: associating normalized word tags with phoneme sequences, and converting each phoneme sequence into a corresponding visual sequence according to a preset phoneme-to-visual mapping rule; quantifying the similarity between visual sequences of different words using edit distance; wherein the minimum visual edit distance between any two different words is 1, and the smaller the visual edit distance, the more similar the words are in pronunciation; constructing a distance matrix by calculating pairwise visual edit distances; converting the distance matrix into a probability distribution by row-wise normalization; and establishing soft tags for each word based on the probability distribution.

[0235] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method steps of the various embodiments described above.

[0236] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the method steps of the various embodiments described above.

[0237] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0238] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0239] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0240] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An event-based lip-reading method, characterized in that, The method includes: The raw event stream of lip sequence images is acquired using an event camera; The voxel-based representation method converts the original event stream into a frame-like event tensor, resulting in a voxelized event body. Spatial features are extracted from the voxelized event body using a front-end network in an event-based lip-reading model to obtain multi-scale spatial features. The front-end network includes a 3D convolutional layer and a deep neural network. The 3D convolutional layer performs preliminary spatiotemporal feature extraction on the voxelized event body to obtain initial spatiotemporal features. These initial spatiotemporal features are then sequentially input into the convolutional layer and multiple residual blocks of the deep neural network. The outputs of these residual blocks undergo global average pooling to obtain high-level spatial feature maps. The outputs of these residual blocks also undergo shallow adaptive pooling to obtain multiple spatial feature maps. Using the intermediate feature map of these multiple spatial feature maps as an alignment reference resolution, the spatial features of the multi-level features are... Dimensional reshaping is performed on a graph-constructed sequence of node representations. This sequence is then input in parallel into a frequency-aware modulation module for frequency enhancement processing to obtain enhanced features. The frequency enhancement processing includes injecting noise into a first frequency band and employing adaptive frequency filtering to enhance the discriminative edge dynamics of a second frequency band. The lower limit frequency of the second frequency band is higher than the upper limit frequency of the first frequency band. The enhanced features are input into a fully connected layer of the deep neural network to obtain multi-level spatial node features. These multi-level spatial node features are then input into a spatial region hypergraph to extract high-order intra-frame dependencies between lip regions. Based on these high-order intra-frame dependencies and the high-level spatial feature map, multi-scale spatial features are obtained. The multi-scale spatial features are modeled temporally using a back-end sequence model in an event-based lip-reading model to obtain sequence encoding; the multi-scale spatial features are then converted into temporally structured features using a temporal semantic hypergraph; and finally, a recurrent neural network is used to model the sequence encoding based on these temporally structured features. The content of lip reading recognition is determined according to the sequence encoding; the content of lip reading is determined by the target lip reading model according to the sequence encoding; wherein, the target lip reading model adopts a visual-based supervision strategy, which is used to encode visual similarity into a label space; the visual-based supervision strategy includes: associating normalized word labels with phoneme sequences, and converting each phoneme sequence into a corresponding visual sequence according to a preset phoneme-to-visual mapping rule; quantifying the similarity between visual sequences of different words by editing distance; wherein, the minimum visual edit distance between any two different words is 1, and the smaller the visual edit distance, the more similar the words are in pronunciation; constructing a distance matrix by calculating paired visual edit distances; converting the distance matrix into a probability distribution by row-wise normalization; and establishing soft labels for each word based on the probability distribution.

2. The method according to claim 1, characterized in that, The raw event stream acquired via the event camera for the lip sequence images includes: The event camera asynchronously records the brightness change of each pixel in the lip sequence image, and triggers an event when the logarithmic intensity change of a pixel exceeds a preset contrast threshold. The events that occur within a preset time window are obtained to obtain the original event stream.

3. The method according to claim 1, characterized in that, The voxel-based representation method converts the original event stream into a frame-like event tensor, resulting in a voxelized event body, including: Based on the events in the original event stream and the given time interval, the event timestamps are scaled to the given time interval using a voxel grid method to obtain a voxelized event body.

4. An event-based lip-reading device, characterized in that, The apparatus, applied to the event-based lip-reading method as described in any one of claims 1 to 3, comprises: The acquisition module is used to acquire the raw event stream of lip sequence images via the event camera; The conversion module is used to convert the original event stream into a frame-like event tensor based on a voxel-based representation method, thereby obtaining a voxelized event body; The feature extraction module is used to extract spatial features from the voxelized event body through the front-end network in the event-based lip reading model to obtain multi-scale spatial features. The time modeling module is used to perform time-dependent modeling of the multi-scale spatial features through the back-end sequence model in the event-based lip-reading model to obtain sequence encoding; The recognition module is used to determine the content of lip reading recognition based on the sequence encoding.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images