Virtual event detection method based on multi-modal data, computer equipment and medium

Abstract

The invention relates to the technical field of computers and provides a virtual event detection method based on multi-modal data, computer equipment and a medium. Performing first modal feature extraction on the first modal data stream to obtain first modal input, and performing second modal feature extraction on the second modal data stream to obtain second modal input; the method comprises the following steps of: performing feature fusion to obtain multi-modal input, then performing time sequence modeling on the multi-modal input by using an LSTM branch to obtain LSTM modeling features, and performing time sequence modeling by using a Transform branch to obtain Transform modeling features; performing time sequence feature fusion to obtain time sequence features; performing feature mapping on the time sequence features by using the plurality of classification heads so as to obtain a plurality of prediction results; based on the plurality of prediction results, the classification probability of each of the plurality of categories is calculated, and then the classification probability of the virtual event category is output.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a virtual event detection method, computer device and medium based on multimodal data. Background Technology

[0002] In the fields of virtual reality, augmented reality, game interaction, and driving simulation, it is often necessary to accurately and quickly identify ongoing virtual events and then provide users with corresponding interactive experiences, such as providing prompts through haptic feedback, or through the vibration function of the seat, or the force feedback of the steering wheel or pedals. However, various possible interference factors may lead to the inability to correctly and timely identify virtual events. For example, when relying on sound to identify the sound of a horse's hooves, the sound of a person walking may be confused with the sound of a horse's hooves moving at a slower speed, because these two may appear similar in audio analysis, leading to system misjudgment. Similarly, when relying on image information to identify the identity or action information of virtual objects, the influence of ambient light or occlusion may also lead to system misjudgment. Although some software provides built-in interfaces to output telemetry data, the information source of telemetry data depends on the built-in interface and is limited by the external hardware interface standards. Therefore, the telemetry data may not provide sufficient correlation information for accurately and quickly identifying ongoing virtual events. For example, telemetry data might include the player's movement speed and direction, but it's difficult to distinguish whether the player is walking or moving slowly while riding a horse based solely on this information.

[0003] To address this, this application proposes a virtual event detection method, computer device, and medium based on multimodal data. This overcomes the shortcomings of single-modal data, helps to accurately and quickly identify virtual events, and improves the accuracy of multimodal signal recognition through algorithm and model optimization, thereby enhancing the user's interactive experience. Summary of the Invention

[0004] In a first aspect, this application provides a virtual event detection method based on multimodal data. The method includes: acquiring at least two modal data for representing virtual content, wherein the modal data includes at least one of video data and audio data; extracting first modal features from a first modal data stream using a first modal feature extraction network to obtain a first modal input; and extracting second modal features from a second modal data stream using a second modal feature extraction network to obtain a second modal input, wherein the first modal data stream and the second modal data stream are temporally synchronized; fusing features from the first modal input and the second modal input to obtain a multimodal input; and then using a first temporal modeling branch to perform temporal modeling on the multimodal input. The system obtains a first temporal modeling feature and a second temporal modeling feature by performing temporal modeling on the multimodal input using a second temporal modeling branch. It then performs temporal feature fusion on the first and second temporal modeling features to obtain a temporal feature corresponding to the multimodal input. Using one or more classification heads, it performs feature mapping on the temporal feature to obtain one or more prediction results corresponding one-to-one with the one or more classification heads. Based on the one or more prediction results, it calculates the classification probability of one or more categories corresponding one-to-one with the one or more classification heads and outputs the category information and / or classification probability of the virtual event category.

[0005] The first aspect of this application overcomes the shortcomings of single-modal data, which helps to accurately and quickly determine virtual events. Furthermore, through the optimization of algorithms and models, it realizes multimodal temporal feature recognition based on deep learning, taking into account both local and global temporal patterns, improving the accuracy of multimodal signal recognition, and helping to improve the user's interactive experience.

[0006] In one possible implementation of the first aspect of this application, the plurality of classification heads satisfy a constraint condition that limits the sum of the classification probabilities of the plurality of categories to 1.

[0007] In one possible implementation of the first aspect of this application, the plurality of categories includes the virtual event category, scene category, and behavior category.

[0008] In one possible implementation of the first aspect of this application, the method further includes: when the classification probability of the virtual event category is greater than a preset threshold, selecting the prediction result with the highest probability among the prediction results corresponding to the virtual event category as the event result; and when the classification probabilities of each of the multiple categories are less than the preset threshold, determining that there is no result.

[0009] In one possible implementation of the first aspect of this application, the plurality of classification heads consist of fully connected layers, and the plurality of classification heads are used to perform feature mapping to the category space according to task requirements.

[0010] In one possible implementation of the first aspect of this application, the first temporal modeling feature is an LSTM modeling feature, and the second temporal modeling feature is a Transformer modeling feature. The temporal feature is obtained by temporal feature fusion of the LSTM modeling feature and the Transformer modeling feature, which includes: using a direct concatenation algorithm, a weighted fusion algorithm, or an attention mechanism fusion algorithm to perform temporal feature fusion of the LSTM modeling feature and the Transformer modeling feature to obtain the temporal feature.

[0011] In one possible implementation of the first aspect of this application, the first temporal modeling feature is an LSTM modeling feature, and the second temporal modeling feature is a Transformer modeling feature, wherein the LSTM branch is used to simultaneously perform temporal data forward-backward processing and temporal data backward-forward processing on the multimodal input, thereby simultaneously utilizing the past temporal information and future temporal information included in the multimodal input.

[0012] In one possible implementation of the first aspect of this application, the first temporal modeling feature is an LSTM modeling feature, and the second temporal modeling feature is a Transformer modeling feature, wherein the Transformer branch is used to encode the multimodal input into a temporal feature representation including global information, thereby obtaining the temporal patterns and dependencies included in the multimodal input.

[0013] In one possible implementation of the first aspect of this application, feature fusion of the first modal input and the second modal input to obtain the multimodal input includes: using a direct concatenation algorithm, a weighted fusion algorithm, or an attention mechanism fusion algorithm to perform feature fusion of the first modal input and the second modal input to obtain the multimodal input.

[0014] In one possible implementation of the first aspect of this application, the first modal data stream is a video data stream continuously and in real time acquired by a video sensor, the first modal feature extraction network is a video feature extraction network, the video data stream is composed of game screen frame images, the game screen frame images include virtual characters, virtual items, virtual vehicles and virtual scenes, and the video feature extraction network includes a spatial feature extraction branch for single frame feature extraction and a temporal feature extraction branch for motion change feature extraction between different frames.

[0015] In one possible implementation of the first aspect of this application, the second modal data stream is an audio data stream acquired by an audio sensor, the second modal feature extraction network is an audio feature extraction network, the audio data stream is composed of the sounds of triggering events, the sounds of triggering events include weapon sounds, animal sounds, human voices and virtual scene sound effects, and the audio feature extraction network is used for raw audio waveform feature extraction and audio data to spectrogram conversion.

[0016] In one possible implementation of the first aspect of this application, the second modal data stream is a telemetry data stream acquired in real time by a built-in tool, the second modal feature extraction network is a telemetry feature extraction network, the telemetry data stream includes motion data, the motion data includes pitch angle, roll angle, yaw angle and acceleration, and the telemetry feature extraction network includes a multilayer perceptron and a convolutional neural network with a one-dimensional convolutional kernel.

[0017] In one possible implementation of the first aspect of this application, the combination of the first modal data stream and the second modal data stream is a combination of a video data stream and an audio data stream, or a combination of a video data stream and a telemetry data stream.

[0018] In one possible implementation of the first aspect of this application, the method further includes: extracting third modal features from the third modal data stream using a third modal feature extraction network to obtain a third modal input; and fusing features from the first modal input, the second modal input, and the third modal input to obtain the multimodal input, wherein the combination of the first modal data stream, the second modal data stream, and the third modal data stream is a combination of a video data stream, an audio data stream, and a telemetry data stream.

[0019] Secondly, this application provides a computer device. The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following operations: acquiring at least two modal data for characterizing virtual content, the modal data including at least one of video data and audio data; extracting first modal features from a first modal data stream using a first modal feature extraction network to obtain a first modal input; and extracting second modal features from a second modal data stream using a second modal feature extraction network to obtain a second modal input, wherein the first modal data stream and the second modal data stream are time-synchronized; and performing feature fusion on the first modal input and the second modal input to obtain... The system takes a multimodal input, then performs temporal modeling on the multimodal input using a first temporal modeling branch to obtain a first temporal modeling feature, and performs temporal modeling on the multimodal input using a second temporal modeling branch to obtain a second temporal modeling feature; it then performs temporal feature fusion on the first and second temporal modeling features to obtain a temporal feature corresponding to the multimodal input; it then performs feature mapping on the temporal feature using one or more classification heads to obtain one or more prediction results corresponding one-to-one with the one or more classification heads; based on the one or more prediction results, it calculates the classification probability of one or more categories corresponding one-to-one with the one or more classification heads, and outputs the category information and / or classification probability of the virtual event category.

[0020] The second aspect of this application overcomes the shortcomings of single-modal data, which helps to accurately and quickly determine virtual events. Furthermore, through the optimization of algorithms and models, it realizes multimodal temporal feature recognition based on deep learning, taking into account both local and global temporal patterns, improving the accuracy of multimodal signal recognition, and helping to improve the user's interactive experience.

[0021] In one possible implementation of the second aspect of this application, the plurality of classification heads satisfy a constraint condition that limits the sum of the classification probabilities of the plurality of categories to 1, and the plurality of categories include the virtual event category, the scene category, and the behavior category.

[0022] In one possible implementation of the second aspect of this application, the combination of the first modal data stream and the second modal data stream is a combination of a video data stream and an audio data stream, or a combination of a video data stream and a telemetry data stream.

[0023] Thirdly, this application provides a computer-readable storage medium. The computer-readable storage medium stores computer instructions that, when executed on a computer device, cause the computer device to perform the following operations: acquiring at least two modal data representing virtual content, the modal data including at least one of video data and audio data; extracting first modal features from a first modal data stream using a first modal feature extraction network to obtain a first modal input; and extracting second modal features from a second modal data stream using a second modal feature extraction network to obtain a second modal input, wherein the first modal data stream and the second modal data stream are time-synchronized; and fusing features from the first modal input and the second modal input to obtain a multimodal input. Then, a first temporal modeling feature is obtained by temporally modeling the multimodal input using a first temporal modeling branch, and a second temporal modeling feature is obtained by temporally modeling the multimodal input using a second temporal modeling branch. Temporal feature fusion is performed on the first and second temporal modeling features to obtain temporal features corresponding to the multimodal input. Feature mapping is performed on the temporal features using one or more classification heads to obtain one or more prediction results corresponding one-to-one with the one or more classification heads. Based on the one or more prediction results, the classification probability of one or more categories corresponding one-to-one with the one or more classification heads is calculated, and the category information and / or classification probability of the virtual event category are output.

[0024] The third aspect of this application overcomes the shortcomings of single-modal data, which helps to accurately and quickly determine virtual events. Furthermore, through the optimization of algorithms and models, it realizes multimodal temporal feature recognition based on deep learning, taking into account both local and global temporal patterns, improving the accuracy of multimodal signal recognition, and helping to improve the user's interactive experience.

[0025] In one possible implementation of the third aspect of this application, the plurality of classification heads satisfy a constraint condition that limits the sum of the classification probabilities of the plurality of categories to 1, and the plurality of categories include the virtual event category, the scene category, and the behavior category.

[0026] In one possible implementation of the third aspect of this application, the combination of the first modal data stream and the second modal data stream is a combination of a video data stream and an audio data stream, or a combination of a video data stream and a telemetry data stream. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A flowchart illustrating a virtual event detection method based on multimodal data provided in this application embodiment; Figure 2 A schematic diagram of multimodal input for a first embodiment of this application; Figure 3 A schematic diagram of multimodal input according to a second embodiment of this application; Figure 4 A schematic diagram of multimodal input according to a third embodiment of this application; Figure 5 A schematic diagram of a multimodal deep learning temporal recognition model provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0029] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0030] It should be understood that in the description of this application, "at least one" means one or more, and "multiple" means two or more. In addition, the words "first," "second," etc., unless otherwise stated, are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance or order.

[0031] Figure 1 This is a flowchart illustrating a virtual event detection method based on multimodal data, provided as an embodiment of this application. Figure 1 As shown, the method includes the following steps.

[0032] Step S101: Obtain at least two modal data for characterizing virtual content, wherein the modal data includes at least one of video data and audio data; Step S103: Extract first modal features from the first modal data stream using a first modal feature extraction network to obtain a first modal input; and extract second modal features from the second modal data stream using a second modal feature extraction network to obtain a second modal input, wherein the first modal data stream and the second modal data stream are synchronized in time. Step S105: Perform feature fusion on the first modal input and the second modal input to obtain a multimodal input. Then, use the first temporal modeling branch to perform temporal modeling on the multimodal input to obtain a first temporal modeling feature. And use the second temporal modeling branch to perform temporal modeling on the multimodal input to obtain a second temporal modeling feature. Step S107: Perform temporal feature fusion on the first temporal modeling feature and the second temporal modeling feature to obtain the temporal feature corresponding to the multimodal input; Step S109: Using one or more classification heads, perform feature mapping on the temporal features to obtain one or more prediction results that correspond one-to-one with the one or more classification heads; Step S111: Based on the one or more prediction results, calculate the classification probability of one or more categories that correspond one-to-one with the one or more classification heads, and output the category information and / or classification probability of the virtual event category.

[0033] Figure 1 The virtual event detection method based on multimodal data shown improves the accuracy of tactile feedback by collecting various types of data. Furthermore, the fusion of different data types enhances the realism of the tactile feedback. In some embodiments, multimodal signal recognition includes combinations of visual data plus auditory data, visual data plus telemetry data, visual data plus auditory data plus telemetry data, or a combination thereof. Utilizing different signal recognition methods helps improve the accuracy of signal recognition, and superposition achieves even better results. Figure 1The virtual event detection method based on multimodal data shown can be applied in fields such as virtual reality, augmented reality, game interaction, and driving simulation. It supports accurate and rapid identification of ongoing virtual events and provides users with corresponding interactive experiences. Examples include providing haptic feedback, seat vibration, steering wheel or pedal force feedback, and providing multimodal cues such as visual and auditory cues. Relying on unimodal input can be affected by various interference factors, leading to incorrect and untimely identification of virtual events. For example, when relying on sound to identify the hoofbeats of a horse, the system may confuse the footsteps of a person walking with the hoofbeats of a horse moving at a slower speed, as these two sounds may appear similar in audio analysis, potentially leading to misjudgment. Similarly, when relying on image information to identify the identity or action information of virtual objects, ambient light or occlusions can also affect the system, potentially causing misjudgment. For example, when relying on built-in interfaces to output telemetry data, the telemetry data may not provide enough correlation information to accurately and quickly identify the virtual events that are happening, due to limitations in the correlation of the telemetry data provided by the built-in interfaces and the external hardware interface standards.

[0034] See Figure 1In step S101, at least two modal data for representing virtual content are acquired, wherein the modal data includes at least one of video data and audio data. In step S103, a first modal input is obtained by extracting first modal features from the first modal data stream through a first modal feature extraction network, and a second modal input is obtained by extracting second modal features from the second modal data stream through a second modal feature extraction network, wherein the first modal data stream and the second modal data stream are synchronized in time. Thus, a multimodal deep learning temporal recognition model is proposed, which can impose restrictions on the input data type through input constraints, thereby unifying the input specifications, such as uniformly limiting the size of video input images. Here, the first modal data stream and the second modal data stream belong to different modalities. The combination of the first modal data stream and the second modal data stream can be a combination of video data stream and audio data stream, or a combination of video data stream and telemetry data stream. Generally, the first modal data stream and the second modal data stream should include video data stream, such as real-time acquired video data stream. For example, the video data stream can be a real-time video stream of game footage acquired through screen recording, in-game recording, or a High Definition Multimedia Interface (HDMI) capture card. Another example is a replay of a pre-recorded game, or real-time footage of vehicles or other transportation. One of the first and second modal data streams is a video data stream, and the other is an audio data stream or a telemetry data stream. The audio data stream can be, for example, sounds triggered by audio devices, such as weapon sounds or game sound effects. Telemetry data can be data collected in real-time through in-game tools, such as motion data including pitch, roll, yaw, and acceleration. Here, the first and second modal data streams are synchronized in time, meaning that the data from different modalities originates from the same virtual scene or is confined to a certain range. For example, in a scenario where a player-controlled virtual character moves on foot or horseback, the video data stream over a certain time period can identify whether the character is on horseback or not, while the audio data stream over the same time period can detect the sound of hooves. Furthermore, because multimodal temporal feature extraction based on deep learning techniques is performed on both the video and audio data streams within the same time period, the multimodal data input can distinguish between the sound of footsteps and hooves, thus reducing the possibility of the system mistakenly determining the character is on horseback in a non-horseback scenario, and consequently reducing the possibility of triggering incorrect haptic feedback. Therefore, using temporally synchronized multimodal data streams as raw data is beneficial for subsequent deep learning-based multimodal temporal feature recognition.

[0035] Continue reading Figure 1In step S105, feature fusion is performed on the first modal input and the second modal input to obtain a multimodal input. Then, a first temporal modeling branch is used to perform temporal modeling on the multimodal input to obtain a first temporal modeling feature, and a second temporal modeling branch is used to perform temporal modeling on the multimodal input to obtain a second temporal modeling feature. It should be noted that the first modal input and the second modal input are the corresponding modal feature extraction results. For example, video feature extraction includes obtaining feature representations from a video frame sequence, which may include extracting spatial features (features within a single frame) and temporal features (motion, changes, etc. between different frames). Video feature extraction can use convolutional neural networks (3D CNNs) with 3D convolutional kernels, where the kernels can slide convolution over a certain time, thereby capturing both temporal and spatial information simultaneously. Video feature extraction can also use residual-connected convolutional neural networks (ResNets) for feature extraction. For example, audio feature extraction involves extracting feature representations from the original audio waveform. A Spectrogram-CNN architecture can be used to convert audio data into a spectrogram, which is then classified and recognized using a convolutional neural network. Similarly, telemetry feature extraction involves extracting feature representations from telemetry data. This can be achieved using a Multi-layer Perceptron (MLP), a neural network composed of multiple fully connected layers, or a 1DCNN with one-dimensional convolutional kernels to capture local temporal patterns. Thus, after obtaining inputs from different modalities through the corresponding modality feature extraction network, feature fusion is performed on these inputs to obtain multimodal inputs, essentially merging features from different modalities. The feature fusion algorithm can be a direct concatenation algorithm (i.e., concatenating into a longer vector), a weighted fusion algorithm (i.e., distinguishing the importance of different modalities by setting different weights), or an attention mechanism fusion algorithm. It can be seen that when performing feature fusion on inputs from different modalities and using different branches to perform temporal modeling of multimodal inputs, the category information contained within the various modalities is not distinguished. For example, the input to the video modality obtained by extracting video features from a video data stream may include features corresponding to people, objects, vehicles, game scenes, etc., included in the image information of the video data stream. Information related to the recognition of virtual events may only be a portion of these features. In some embodiments, the first temporal modeling feature is an LSTM modeling feature, and the second temporal modeling feature is a Transformer modeling feature. The first temporal modeling branch is an LSTM branch, and the second temporal modeling branch is a Transformer branch.For example, the audio modal input obtained by extracting audio features from an audio data stream may include various sounds acquired by audio devices, such as weapon sounds, voices, and game sound effects. Information relevant to virtual event recognition may only be a portion of this. Here, without classifying or filtering the first or second modal input, feature fusion is directly performed on the first and second modal inputs to obtain a multimodal input. The same multimodal input is then output separately to the LSTM branch and the Transformer branch for time-series modeling. This helps to fully utilize various information to improve the accuracy of virtual event recognition. Furthermore, the LSTM branch and the Transformer branch each have their own model and algorithmic focuses. The LSTM branch uses a Long Short-Term Memory (LSTM) network to model time-series data and includes a bidirectional LSTM (Bi-LSTM), thus processing time-series data both forward and backward, achieving simultaneous utilization of past and future information. The Transformer branch processes the input sequence using a Transformer, encoding it into a feature representation containing global information through an encoder, effectively capturing patterns and dependencies within the sequence. Thus, by leveraging the respective model and algorithmic focuses of the LSTM and Transformer branches, temporal data modeling is performed on the same multimodal input. The LSTM branch captures long-term temporal dependencies, while the Transformer branch handles global context and long-range dependencies. The outputs of these two branches are then fused, achieving a balance between local and global temporal patterns. Furthermore, the temporal modeling of the LSTM and Transformer branches can be performed simultaneously or sequentially, as long as the outputs of the last two branches are fused.

[0036] Continue reading Figure 1In step S107, the first temporal modeling feature and the second temporal modeling feature are fused to obtain the temporal feature corresponding to the multimodal input. Thus, temporal feature fusion is performed by fusing the modeling features output by the LSTM branch and the Transformer branch, which can be achieved using a direct concatenation algorithm, a weighted fusion algorithm, or an attention mechanism fusion algorithm. Specifically, in step S103, the first modal input and the second modal input are fused to obtain the multimodal input, and in step S105, the LSTM modeling feature and the Transformer modeling feature are fused to obtain the temporal feature corresponding to the multimodal input. This requires two feature fusions, but the processes and meanings are different. The first feature fusion occurs after multimodal feature extraction, fusing features from different modal inputs; the second feature fusion occurs after temporal modeling of the multimodal input using different branches, fusing the modeling features output by the LSTM branch and the Transformer branch. This two-step feature fusion design helps to fully utilize various information to improve the accuracy of virtual event recognition. It also facilitates the use of temporal feature representation based on correlated information, enabling the complementary use of data from different modalities within the same virtual scene or a certain time frame to achieve deep learning-based multimodal temporal feature recognition. For example, in a scene where a player-controlled virtual character moves on foot or horseback, the video data stream over a certain time period can identify whether the character is on horseback or not, while the audio data stream over that time period can detect the sound of hooves. Furthermore, because multimodal deep learning-based temporal feature extraction is performed on both video and audio data streams within the same time frame, multimodal inputs (or combinations of other modal data streams) can be provided using both streams. First, feature fusion is performed on the inputs from different modalities to obtain the multimodal input. Finally, feature fusion is performed on the modeling features of different branches to obtain the temporal features corresponding to the multimodal input. Thus, the output of deep learning-based multimodal temporal feature recognition can be used to distinguish between footsteps and hoof sounds. For example, it can detect whether the sound is from a person riding a horse or walking without riding a horse. This reduces the possibility of the system mistakenly judging a person as riding a horse in a non-riding scenario, thereby reducing the likelihood of triggering incorrect tactile feedback. Therefore, using temporally synchronized multimodal data streams as raw data is beneficial for subsequent deep learning-based multimodal temporal feature recognition.

[0037] Continue reading Figure 1In step S109, one or more classification heads are used to perform feature mapping on the temporal features, thereby obtaining one or more prediction results corresponding one-to-one with the one or more classification heads. In step S111, based on the one or more prediction results, the classification probability of one or more categories corresponding one-to-one with the one or more classification heads is calculated, and the category information and / or classification probability of the virtual event category are output. Here, the classification head is used to map features to a category space. According to different tasks, the extracted features are processed through different classification heads to predict game events, scenes, actions, etc. The classification head uses fully connected layers (dense layers) as the basic construction of the classification head. The output layer means that the final prediction result is obtained through the output layer. The output layer may include a Softmax function to calculate the probability distribution of different categories. According to the design constraints, the sum of the probabilities of all categories is 1. In addition, the output layer may also include a sigmoid function to independently predict the probability of each category (the overall constraint is that the sum is 100%). Thus, after passing through the output layer, classification probabilities are obtained, i.e., the probabilities of different event categories. Event categories are selected based on these probabilities, and different force feedback parameters are set for different events. Finally, based on the classification results, corresponding force feedback strategies are provided. For example, one force feedback strategy targets event categories, and another targets behavior categories. Generally, a preset threshold can be set; if the classification probability is greater than this threshold, the corresponding event category is considered to have occurred. Conversely, if the classification probabilities of all categories are not higher than the preset threshold, it is considered that no category has occurred. Furthermore, in some embodiments, the categories that the classification head can identify may include action prediction categories. This allows it to be combined with information such as player posture and arm position collected by real-world sensors as the basis for the force feedback strategy. In this way, the classification head integrates features through fully connected layers, and the output layer finally provides the classification results, supporting detection tasks with multiple categories or multiple labels.

[0038] In short, Figure 1 The virtual event detection method based on multimodal data presented here overcomes the shortcomings of single-modal data, facilitating accurate and rapid identification of virtual events. Furthermore, through algorithm and model optimization, it achieves deep learning-based multimodal temporal feature recognition, taking into account both local and global temporal patterns, thus improving the accuracy of multimodal signal recognition and enhancing the user's interactive experience. Moreover, this virtual event detection method can complete virtual event recognition without relying on the telemetry interface provided by the game engine.

[0039] Figure 2 This is a schematic diagram of multimodal input according to a first embodiment of this application. (See attached diagram.) Figure 2As shown, the multimodal input A210 is a combination of the video data stream A220 and the audio data stream A230. See also... Figure 1 and Figure 2 In some embodiments, the first modal data stream is a video data stream A220 continuously and in real-time acquired by a video sensor. The first modal feature extraction network is a video feature extraction network. The video data stream A220 consists of game screen frame images, including virtual characters, virtual items, virtual vehicles, and virtual scenes. The video feature extraction network includes a spatial feature extraction branch for single-frame feature extraction and a temporal feature extraction branch for motion change feature extraction between different frames. The second modal data stream is an audio data stream A230 acquired by an audio sensor. The second modal feature extraction network is an audio feature extraction network. The audio data stream A230 consists of sounds triggered by events, including weapon sounds, animal sounds, human voices, and virtual scene sound effects. The audio feature extraction network is used for raw audio waveform feature extraction and audio data to spectrogram conversion. Thus, utilizing different signal recognition methods helps improve the accuracy of signal recognition, and better results can be achieved through superposition.

[0040] Figure 3 This is a schematic diagram illustrating multimodal input according to a second embodiment of this application. (See attached diagram.) Figure 3 As shown, the multimodal input B310 is a combination of the video data stream B320 and the telemetry data stream B340. See also... Figure 1 and Figure 3 In some embodiments, the first modal data stream is a video data stream B320 continuously and in real-time acquired by a video sensor. The first modal feature extraction network is a video feature extraction network. The video data stream B320 consists of game frame images, including virtual characters, virtual items, virtual vehicles, and virtual scenes. Furthermore, the video feature extraction network includes a spatial feature extraction branch for single-frame feature extraction and a temporal feature extraction branch for motion change feature extraction between different frames. The second modal data stream is a telemetry data stream B340 acquired in real-time by built-in tools. The second modal feature extraction network is a telemetry feature extraction network. The telemetry data stream B340 includes motion data, including pitch angle, roll angle, yaw angle, and acceleration. Furthermore, the telemetry feature extraction network includes a multilayer perceptron and a convolutional neural network with one-dimensional convolutional kernels. Thus, utilizing different signal recognition methods helps improve the accuracy of signal recognition, and better results can be achieved through superposition.

[0041] See Figure 1 , Figure 2 and Figure 3 The combination of the first and second modal data streams is either a combination of video and audio data streams, or a combination of video and telemetry data streams. Thus, the first and second modal data streams are temporally synchronized, meaning that data from different modalities originates from the same virtual scene or is confined to a certain range. For example, in a scene where a player-controlled virtual character moves on foot or horseback, the video data stream over a certain time period can identify whether the character is on horseback or not, while the audio data stream over that time period can detect the sound of hooves. Furthermore, because multimodal temporal feature extraction based on deep learning technology is performed on the video and audio data streams within the same time period, the multimodal data input can be used to distinguish between the sound of footsteps and the sound of hooves, reducing the possibility of the system mistakenly determining the character's riding status in a non-riding scene, and thus reducing the possibility of triggering incorrect haptic feedback. Therefore, using temporally synchronized multimodal data streams as raw data is beneficial for subsequent deep learning-based multimodal temporal feature recognition.

[0042] Figure 4 This is a schematic diagram illustrating multimodal input according to a third embodiment of this application. (See attached diagram.) Figure 4 As shown, the multimodal input C410 is a combination of video data stream C420, audio data stream C430, and telemetry data stream C440. See also... Figure 1 and Figure 4 In some embodiments, the method further includes: extracting third-modal features from the third-modal data stream using a third-modal feature extraction network to obtain a third-modal input; and fusing features from the first-modal input, the second-modal input, and the third-modal input to obtain the multimodal input C410, wherein the combination of the first-modal data stream, the second-modal data stream, and the third-modal data stream is a combination of a video data stream C420, an audio data stream C430, and a telemetry data stream C440. This overcomes the limitations of single-modal data and, by combining data streams from multiple modalities, achieves deep learning-based multimodal temporal feature recognition, which helps to correctly and timely identify virtual events.

[0043] See Figure 1 , Figure 2 , Figure 3 and Figure 4The multimodal data stream can be a combination of video and audio data streams, or a combination of video and telemetry data streams, or a combination of video, audio, and telemetry data streams. In one possible implementation, the multiple classification heads satisfy a constraint condition that limits the sum of the classification probabilities of each of the multiple categories to 1. This achieves the prediction and output of classification probabilities for multiple different categories, where all category probabilities include not only virtual event categories but also other categories such as scenes and actions. When performing feature fusion on inputs from different modalities and using different branches to perform temporal modeling of multimodal inputs, the category information contained in the inputs of various modalities is not distinguished. For example, the video modal input obtained by extracting video features from a video data stream may include features corresponding to people, objects, vehicles, game scenes, etc., included in the image information of the video data stream; the information related to the recognition of virtual events may only be a part of this. For example, the audio modal input obtained by extracting audio features from an audio data stream may include various sounds acquired by the audio device, such as weapon sounds, voices, and game sound effects. Information relevant to virtual event recognition may only be a portion of this. Here, without classifying or filtering the first or second modal input, feature fusion is directly performed on the first and second modal inputs to obtain a multimodal input. The same multimodal input is then output separately to the LSTM branch and the Transformer branch for temporal modeling. This helps to fully utilize various information to improve the accuracy of virtual event recognition.

[0044] In one possible implementation, the multiple categories include virtual event categories, scene categories, and behavior categories. Thus, all category probabilities include not only virtual event categories but also other categories such as scenes and actions. When performing feature fusion on inputs from different modalities and using different branches to perform temporal modeling of multimodal inputs, the category information contained within the inputs of various modalities is not distinguished. After passing through the output layer, classification probabilities, i.e., the probabilities of different event categories, are obtained. Event categories are selected based on these probabilities, and different force feedback parameters are set for different events. Finally, based on the classification results, a corresponding force feedback strategy is provided.

[0045] In one possible implementation, the method further includes: when the classification probability of the virtual event category is greater than a preset threshold, selecting the prediction result with the highest probability among the prediction results corresponding to the virtual event category as the event result; and when the classification probabilities of each of the multiple categories are less than the preset threshold, determining that there is no result. Thus, feature fusion is first performed on the inputs of different modalities to obtain multimodal input, and finally feature fusion is performed on the modeling features of different branches to obtain temporal features corresponding to the multimodal input. In this way, the output results of deep learning-based multimodal temporal feature recognition can be used to distinguish between the sound of footsteps and the sound of hooves, for example, detecting whether the sound of hooves is made while the person is riding a horse or the sound of footsteps is made while the person is not riding a horse. This reduces the possibility of the system mistakenly triggering the judgment of the person being on horseback in a non-riding scenario, thus reducing the possibility of initiating incorrect tactile feedback. Therefore, limiting the multimodal data stream synchronized in time as the raw data is beneficial for subsequent deep learning-based multimodal temporal feature recognition.

[0046] In one possible implementation, the multiple classification heads consist of fully connected layers, and these classification heads are used to perform feature mapping to the category space according to task requirements. Thus, the classification heads integrate features through fully connected layers, and the final classification result is provided by the output layer, supporting detection tasks with multiple categories or multiple labels.

[0047] In one possible implementation, the temporal features are obtained by temporal feature fusion of the LSTM modeling features and the Transformer modeling features. This includes using a direct concatenation algorithm, a weighted fusion algorithm, or an attention mechanism fusion algorithm to fuse the LSTM modeling features and the Transformer modeling features. This requires two feature fusions, but the processes and meanings differ. The first feature fusion occurs after multimodal feature extraction, fusing features from different modal inputs. The second feature fusion occurs after temporal modeling of the multimodal inputs using different branches, fusing the modeling features output by the LSTM and Transformer branches. This two-step feature fusion design helps to fully utilize various information to improve the accuracy of virtual event recognition. It also facilitates the use of temporal feature representation based on correlation information, enabling the complementary use of data from different modalities within the same virtual scene or a certain time range to achieve deep learning-based multimodal temporal feature recognition.

[0048] In one possible implementation, the LSTM branch is used to simultaneously perform forward and backward processing of the multimodal input, thereby utilizing both past and future temporal information included in the multimodal input. In another possible implementation, the Transformer branch is used to encode the multimodal input into a temporal feature representation including global information, thereby obtaining the temporal patterns and dependencies included in the multimodal input. Thus, the LSTM and Transformer branches each have their own model and algorithmic focuses. The LSTM branch utilizes a Long Short-Term Memory network to model time-series data and includes a bidirectional LSTM, thus processing temporal data both forward and backward, achieving simultaneous utilization of past and future information. The Transformer branch uses a Transformer to process the input sequence, encoding it into a feature representation containing global information through an encoder, effectively capturing patterns and dependencies in the sequence. Therefore, by utilizing the respective model and algorithmic focuses of the LSTM and Transformer branches, temporal data modeling is performed on the same multimodal input. This approach captures long-term temporal dependencies through the LSTM branch and handles global context and remote dependencies through the Transformer branch. The outputs of these two branches are then merged to achieve a balance between local and global temporal patterns.

[0049] In one possible implementation, feature fusion of the first modal input and the second modal input is performed to obtain the multimodal input. This includes using a direct concatenation algorithm, a weighted fusion algorithm, or an attention mechanism fusion algorithm to perform feature fusion on the first modal input and the second modal input to obtain the multimodal input. Thus, two feature fusions are required, but the processes and meanings differ. The first feature fusion occurs after multimodal feature extraction, fusing features from inputs of different modalities. The second feature fusion occurs after temporal modeling of the multimodal input using different branches, fusing the modeling features output by the LSTM branch and the Transformer branch. This two-step feature fusion design helps to fully utilize various information to improve the accuracy of virtual event recognition and also helps to utilize temporal feature representation of correlated information. It facilitates the complementary use of data from different modalities within the same virtual scene or a certain time range to achieve deep learning-based multimodal temporal feature recognition.

[0050] See Figure 1 , Figure 2 , Figure 3 and Figure 4In some embodiments, the first modality feature extraction network and / or the second modality feature extraction network are trained using machine learning. Thus, by optimizing the training method and collecting multiple types of data, the accuracy of haptic feedback is improved. The fusion of multiple data types further enhances the realism of the haptic feedback. This overcomes the limitations of single-modality data, facilitating accurate and rapid identification of virtual events. Furthermore, through algorithm and model optimization, deep learning-based multimodal temporal feature recognition is achieved, considering both local and global temporal patterns, improving the accuracy of multimodal signal recognition and enhancing the user's interactive experience. Moreover, the virtual event detection can be completed without relying on the telemetry interface provided by the game engine. See Figure 1 , Figure 2 , Figure 3 and Figure 4 In some embodiments, the first modal data stream and the second modal data stream are time-aligned before or after feature extraction. Thus, the optimized training method, by collecting multiple types of data, helps improve the accuracy of haptic feedback. The fusion of multiple data types further enhances the realism of the haptic feedback. This overcomes the limitations of single-modal data, facilitating accurate and rapid identification of virtual events. Furthermore, through algorithm and model optimization, deep learning-based multimodal temporal feature recognition is achieved, considering both local and global temporal patterns, improving the accuracy of multimodal signal recognition and enhancing the user's interactive experience. Moreover, the virtual event detection can be completed without relying on the telemetry interface provided by the game engine.

[0051] See Figure 1 , Figure 2 , Figure 3 and Figure 4 In some embodiments, at least one of the first and second temporal modeling branches employs a recursive structure, an attention structure, a convolutional structure, or a combination thereof. Thus, by optimizing the training method and collecting multiple types of data, the accuracy of haptic feedback is improved. The fusion of multiple data types further enhances the realism of the haptic feedback. This overcomes the limitations of single-modal data, facilitating accurate and rapid identification of virtual events. Furthermore, through algorithm and model optimization, deep learning-based multimodal temporal feature recognition is achieved, considering both local and global temporal patterns, improving the accuracy of multimodal signal recognition and enhancing the user's interactive experience. Moreover, the virtual event detection can be completed without relying on the telemetry interface provided by the game engine.

[0052] See Figure 1, Figure 2 , Figure 3 and Figure 4 In some embodiments, the receptive fields corresponding to the first and second temporal modeling branches are different. Thus, by optimizing the training method and collecting multiple types of data, the accuracy of haptic feedback can be improved. The fusion of multiple data types further enhances the realism of the haptic feedback. This overcomes the limitations of single-modal data, facilitating accurate and rapid identification of virtual events. Furthermore, through algorithm and model optimization, deep learning-based multimodal temporal feature recognition is achieved, considering both local and global temporal patterns, improving the accuracy of multimodal signal recognition and enhancing the user's interactive experience. Moreover, the virtual event detection can be completed without relying on the telemetry interface provided by the game engine.

[0053] See Figure 1 , Figure 2 , Figure 3 and Figure 4 In some embodiments, the one or more classification heads are used to output prediction results for mutually exclusive categories or non-mutually exclusive categories, respectively. Thus, by optimizing the training method and collecting multiple types of data, the accuracy of haptic feedback is improved. The fusion of multiple data types further enhances the realism of the haptic feedback. This overcomes the limitations of single-modal data, facilitating accurate and rapid identification of virtual events. Furthermore, through algorithm and model optimization, deep learning-based multimodal temporal feature recognition is achieved, taking into account both local and global temporal patterns, improving the accuracy of multimodal signal recognition and enhancing the user's interactive experience. Moreover, the virtual event detection can be completed without relying on the telemetry interface provided by the game engine.

[0054] See Figure 1 , Figure 2 , Figure 3 and Figure 4 In some embodiments, the classification probabilities of the virtual event categories are smoothed or thresholded before output. Thus, the optimized training method, by collecting multiple types of data, helps improve the accuracy of haptic feedback. The fusion of multiple data types further enhances the realism of the haptic feedback. It overcomes the shortcomings of single-modal data, facilitating accurate and rapid identification of virtual events. Furthermore, through algorithm and model optimization, it achieves deep learning-based multimodal temporal feature recognition, considering both local and global temporal patterns, improving the accuracy of multimodal signal recognition and enhancing the user's interactive experience. Moreover, the virtual event detection can be completed without relying on the telemetry interface provided by the game engine.

[0055] Figure 5 This is a schematic diagram of a multimodal deep learning temporal recognition model provided in an embodiment of this application. Figure 5 As shown, the input of the multimodal deep learning temporal recognition model 500 is a first modality data stream 501 and a second modality data stream 503, and the output of the multimodal deep learning temporal recognition model 500 is the classification probability 572 of the virtual event category. The multimodal deep learning temporal recognition model 500 includes two parts: the first part is a multimodal feature extraction and fusion part 510, and the second part is a temporal modeling and classification prediction part 512. The multimodal feature extraction and fusion part 510 extracts corresponding modal features from the first modality data stream 501 and the second modality data stream 503 respectively, and then performs feature fusion to obtain the multimodal input 514. The multimodal feature extraction and fusion part 510 includes a first modality feature extraction network 520, a second modality feature extraction network 522, and a feature fusion network 524. Specifically, a first modal input 505 is obtained by extracting first modal features from the first modal data stream 501 through a first modal feature extraction network 520, and a second modal input 507 is obtained by extracting second modal features from the second modal data stream 503 through a second modal feature extraction network 522. A feature fusion network 524 is used to fuse the features of the first modal input 505 and the second modal input 507 to obtain a multimodal input 514.

[0056] See Figure 5 The temporal modeling and classification prediction part 512 is used to perform temporal modeling on the multimodal input 514 through two branches respectively, thereby obtaining the corresponding modeling features, then performing temporal feature fusion, and then mapping the features to different categories. The temporal modeling and classification prediction part 512 includes an LSTM branch 530 and a Transformer branch 532. The LSTM branch 530 is used to perform temporal modeling on the multimodal input 514 to obtain LSTM modeling features 540, and the Transformer branch 532 is used to perform temporal modeling on the multimodal input to obtain Transformer modeling features 542. The temporal modeling and classification prediction part 512 also includes a temporal feature fusion network 550. The temporal feature fusion network 550 is used to perform temporal feature fusion on the LSTM modeling features 540 and the Transformer modeling features 542 to obtain temporal features 552 corresponding to the multimodal input 514. The temporal modeling and classification prediction part 512 also includes a classification head model 560. The classification head model 560 is used to perform feature mapping on the temporal features 552 using multiple classification heads, thereby obtaining multiple prediction results that correspond one-to-one with the multiple classification heads. Figure 5The prediction result 562 is shown in the figure. The temporal modeling and classification prediction part 512 also includes an output layer 570. The output layer 570 is used to calculate the classification probability of each of the multiple categories corresponding one-to-one with the multiple classification heads based on the multiple prediction results, and then output the classification probability 572 of the virtual event category among the multiple categories.

[0057] See Figure 5 This requires two feature fusion processes, but their processes and implications differ. The first feature fusion occurs after multimodal feature extraction, where feature fusion network 524 fuses features from different modalities. The second feature fusion occurs after temporal modeling of the multimodal inputs using different branches, where temporal feature fusion network 550 fuses the modeling features output from LSTM branch 530 and Transformer branch 532. This two-step feature fusion design helps to fully utilize various information to improve the accuracy of virtual event recognition. It also facilitates the use of temporal feature representation based on correlation information, enabling the complementary use of data from different modalities within the same virtual scene or a certain time frame to achieve deep learning-based multimodal temporal feature recognition. For example, in a scenario where a player-controlled virtual character moves on foot or horseback, the video data stream within a certain time period can identify whether the character is on horseback or not, while the audio data stream within that time period can detect the sound of hooves. Furthermore, since this method involves multimodal temporal feature extraction based on deep learning from video and audio data streams within the same time period, it utilizes both streams (or combinations of other modal data streams) to provide multimodal input. First, features from different modal inputs are fused to obtain the multimodal input. Finally, features from different branches are fused to obtain the temporal features corresponding to the multimodal input. In this way, the output of deep learning-based multimodal temporal feature recognition can be used to distinguish between footsteps and hoof sounds. For example, it can detect whether the sound is from a person riding a horse or walking without riding a horse, thus reducing the possibility of the system mistakenly judging a person as riding a horse in a non-riding scenario and reducing the likelihood of erroneous tactile feedback being triggered. Therefore, limiting the raw data to temporally synchronized multimodal data streams is beneficial for subsequent deep learning-based multimodal temporal feature recognition.

[0058] In short, Figure 5The multimodal deep learning time series recognition model 500 shown overcomes the shortcomings of single-modal data, helps to accurately and quickly identify virtual events, and through algorithm and model optimization, realizes multimodal time series feature recognition based on deep learning, taking into account both local and global time series patterns, improving the accuracy of multimodal signal recognition, and helping to improve the user's interactive experience.

[0059] Figure 6 This is a schematic diagram of the structure of a computer device 600 provided in an embodiment of this application. The computer device 600 includes one or more processors 610, a communication interface 620, and a memory 630. The processors 610, the communication interface 620, and the memory 630 are interconnected via a bus 640. Optionally, the computer device 600 may further include an input / output interface 650, which is connected to input / output devices for receiving user-set parameters, etc. The computer device 600 can be used to implement some or all of the functions of the device embodiment or system embodiment in the above-described embodiments of this application; the processor 610 can also be used to implement some or all of the operation steps of the method embodiment in the above-described embodiments of this application. For example, the specific implementation of various operations performed by the computer device 600 can be referred to the specific details in the above embodiments, such as the processor 610 being used to execute some or all of the steps or operations in the above-described method embodiments. For example, in the embodiments of this application, the computer device 600 can be used to implement some or all of the functions of one or more components in the above-described device embodiments. In addition, the communication interface 620 can be used specifically for communication functions necessary to implement the functions of these devices and components, and the processor 610 can be used specifically for processing functions necessary to implement the functions of these devices and components.

[0060] It should be understood that, Figure 6 The computer device 600 may include one or more processors 610, and the multiple processors 610 may cooperate to provide processing power in a parallel connection mode, a serial connection mode, a serial-parallel connection mode, or an arbitrary connection mode; or the multiple processors 610 may form a processor sequence or a processor array; or the multiple processors 610 may be divided into a main processor and an auxiliary processor; or the multiple processors 610 may have different architectures, such as adopting a heterogeneous computing architecture. Furthermore, Figure 6 The structural and functional descriptions of the computer device 600 shown are exemplary and non-limiting. In some exemplary embodiments, the computer device 600 may include... Figure 6 The diagram shows more or fewer components, or combinations of some components, or splitting of some components, or different arrangements of components.

[0061] The processor 610 can have various specific implementations. For example, the processor 610 may include one or more combinations of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or a data processing unit (DPU), etc. This application embodiment does not impose specific limitations. The processor 610 can also be a single-core processor or a multi-core processor. The processor 610 can be a combination of a CPU and hardware chips. The aforementioned hardware chips can be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The aforementioned PLDs can be complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs), or any combination thereof. The processor 610 can also be implemented solely using logic devices with built-in processing logic, such as FPGAs or digital signal processors (DSPs). The communication interface 620 can be a wired interface or a wireless interface, used to communicate with other modules or devices. The wired interface can be an Ethernet interface, a local interconnect network (LIN), etc., and the wireless interface can be a cellular network interface or a wireless LAN interface, etc.

[0062] Memory 630 may be non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Memory 630 may also be volatile memory, which may be random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM). The memory 630 can also be used to store program code and data, so that the processor 610 can call the program code stored in the memory 630 to execute some or all of the operation steps in the above method embodiments, or to execute the corresponding functions in the above device embodiments. Furthermore, the computer device 600 may include, compared to... Figure 6 The number of components displayed may be more or less, or there may be different component configurations.

[0063] Bus 640 can be a Peripheral Component Interconnect Express (PCIe) bus, or an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. Bus 640 can be divided into address bus, data bus, control bus, etc. In addition to the data bus, bus 640 can also include a power bus, control bus, and status signal bus. However, for clarity, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0064] The methods and devices provided in this application are based on the same inventive concept. Since the principles by which the methods and devices solve problems are similar, the embodiments, implementation methods, examples, or methods of implementation of the methods and devices can be referred to each other, and repeated details will not be repeated. This application also provides a system comprising multiple computer devices, the structure of each computer device of which can refer to the structure of the computer devices described above. The functions or operations achievable by this system can refer to the specific implementation steps in the above method embodiments and / or the specific functions described in the above device embodiments, and will not be repeated here.

[0065] This application also provides a computer-readable storage medium storing computer instructions. When these computer instructions are executed on a computer device (such as one or more processors), they can implement the method steps described in the above method embodiments. The specific implementation of the above method steps by the processor of the computer-readable storage medium can refer to the specific operations described in the above method embodiments and / or the specific functions described in the above device embodiments, and will not be repeated here.

[0066] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. This application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Embodiments of this application can be implemented wholly or partially by software, hardware, firmware, or any other combination. When implemented in software, the above embodiments can be implemented wholly or partially as a computer program product. This application can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless network communication, microwave, etc.) means. Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that contains one or more sets of available media. Available media can be magnetic media (such as floppy disks, hard disks, and magnetic tapes), optical media, or semiconductor media. Semiconductor media can be solid-state drives, random access memory, flash memory, read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, or any other suitable form of storage medium.

[0067] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. Each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0068] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. The steps in the methods of the embodiments of this application can be adjusted in order, combined, or deleted according to actual needs; the modules in the systems of the embodiments of this application can be divided, combined, or deleted according to actual needs. If these modifications and variations of the embodiments of this application fall within the scope of the claims of this application and their equivalents, then this application also intends to include these modifications and variations.

Claims

1. A virtual event detection method based on multimodal data, characterized in that, The method includes: Acquire at least two modal data for characterizing virtual content, wherein the modal data includes at least one of video data and audio data; A first modal input is obtained by extracting first modal features from a first modal data stream using a first modal feature extraction network, and a second modal input is obtained by extracting second modal features from a second modal data stream using a second modal feature extraction network, wherein the first modal data stream and the second modal data stream are synchronized in time. The first modal input and the second modal input are fused to obtain a multimodal input. Then, the multimodal input is temporally modeled using a first temporal modeling branch to obtain a first temporal modeling feature. The multimodal input is temporally modeled using a second temporal modeling branch to obtain a second temporal modeling feature. Temporal feature fusion is performed on the first temporal modeling feature and the second temporal modeling feature to obtain the temporal feature corresponding to the multimodal input; Using one or more classification heads, feature mapping is performed on the time-series features to obtain one or more prediction results that correspond one-to-one with the one or more classification heads; Based on the one or more prediction results, calculate the classification probability of one or more categories that correspond one-to-one with the one or more classification heads, and output the category information and / or classification probability of the virtual event category.

2. The method according to claim 1, characterized in that, The multiple classification heads satisfy a constraint condition, which limits the sum of the classification probabilities of the multiple categories to 1.

3. The method according to claim 1, characterized in that, The multiple categories include the virtual event category, scene category, and behavior category.

4. The method according to claim 1, characterized in that, The method further includes: When the classification probability of the virtual event category is greater than a preset threshold, the prediction result with the highest probability among the prediction results corresponding to the virtual event category is selected as the event result. When the classification probabilities of each of the multiple categories are less than the preset threshold, it is determined that there is no result.

5. The method according to claim 1, characterized in that, The multiple classification heads are composed of fully connected layers, and the multiple classification heads are used to perform feature mapping to the category space according to task requirements.

6. The method according to claim 1, characterized in that, The first temporal modeling feature is an LSTM modeling feature, and the second temporal modeling feature is a Transformer modeling feature. The temporal feature is obtained by fusing the LSTM modeling feature and the Transformer modeling feature with temporal features, including: using a direct concatenation algorithm, a weighted fusion algorithm, or an attention mechanism fusion algorithm to fuse the LSTM modeling feature and the Transformer modeling feature with temporal features to obtain the temporal feature.

7. The method according to claim 1, characterized in that, The first temporal modeling feature is an LSTM modeling feature, and the second temporal modeling feature is a Transformer modeling feature. The LSTM branch is used to simultaneously perform forward and backward temporal data processing on the multimodal input, thereby utilizing both past and future temporal information included in the multimodal input.

8. The method according to claim 1, characterized in that, The first temporal modeling feature is an LSTM modeling feature, and the second temporal modeling feature is a Transformer modeling feature. The Transformer branch is used to encode the multimodal input into a temporal feature representation that includes global information, thereby obtaining the temporal patterns and dependencies included in the multimodal input.

9. The method according to claim 1, characterized in that, The multimodal input is obtained by fusing features of the first modal input and the second modal input, including: using a direct concatenation algorithm, a weighted fusion algorithm, or an attention mechanism fusion algorithm to fuse features of the first modal input and the second modal input.

10. The method according to claim 1, characterized in that, The first modal data stream is a video data stream continuously and in real time acquired by a video sensor. The first modal feature extraction network is a video feature extraction network. The video data stream is composed of game screen frame images, which include virtual characters, virtual items, virtual vehicles, and virtual scenes. Furthermore, the video feature extraction network includes a spatial feature extraction branch for single-frame feature extraction and a temporal feature extraction branch for motion change feature extraction between different frames.

11. The method according to claim 1, characterized in that, The second modal data stream is an audio data stream acquired by an audio sensor. The second modal feature extraction network is an audio feature extraction network. The audio data stream consists of the sounds of triggering events, including weapon sounds, animal sounds, human voices, and virtual scene sound effects. Furthermore, the audio feature extraction network is used for extracting features from the original audio waveform and converting audio data to a spectrogram.

12. The method according to claim 1, characterized in that, The second modality data stream is a telemetry data stream acquired in real time through built-in tools. The second modality feature extraction network is a telemetry feature extraction network. The telemetry data stream includes motion data, which includes pitch angle, roll angle, yaw angle and acceleration. Furthermore, the telemetry feature extraction network includes a multilayer perceptron and a convolutional neural network with a one-dimensional convolutional kernel.

13. The method according to claim 1, characterized in that, The combination of the first modal data stream and the second modal data stream is a combination of video data stream and audio data stream, or a combination of video data stream and telemetry data stream.

14. The method according to claim 1, characterized in that, The method further includes: The third modality input is obtained by extracting third modality features from the third modality data stream through a third modality feature extraction network. The multimodal input is obtained by fusing features from the first modal input, the second modal input, and the third modal input. The combination of the first modal data stream, the second modal data stream, and the third modal data stream is a combination of video data stream, audio data stream, and telemetry data stream.

15. The method according to claim 1, characterized in that, The first modality feature extraction network and / or the second modality feature extraction network are trained using machine learning.

16. The method according to claim 1, characterized in that, The first modal data stream and the second modal data stream are time-aligned before or after feature extraction.

17. The method according to claim 1, characterized in that, At least one of the first temporal modeling branch and the second temporal modeling branch is implemented using a recursive structure, an attention structure, a convolutional structure, or a combination thereof.

18. The method according to claim 1, characterized in that, The first time series modeling branch and the second time series modeling branch correspond to different temporal receptive fields.

19. The method according to claim 1, characterized in that, The one or more classification heads are used to output prediction results for mutually exclusive categories or non-mutually exclusive categories, respectively.

20. The method according to claim 1, characterized in that, The classification probabilities of the virtual event categories are output after smoothing or thresholding.

21. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The operations performed by the processor when executing the computer program include: Acquire at least two modal data for characterizing virtual content, wherein the modal data includes at least one of video data and audio data; A first modal input is obtained by extracting first modal features from a first modal data stream using a first modal feature extraction network, and a second modal input is obtained by extracting second modal features from a second modal data stream using a second modal feature extraction network, wherein the first modal data stream and the second modal data stream are synchronized in time. The first modal input and the second modal input are fused to obtain a multimodal input. Then, the multimodal input is temporally modeled using a first temporal modeling branch to obtain a first temporal modeling feature. The multimodal input is temporally modeled using a second temporal modeling branch to obtain a second temporal modeling feature. Temporal feature fusion is performed on the first temporal modeling feature and the second temporal modeling feature to obtain the temporal feature corresponding to the multimodal input; Using one or more classification heads, feature mapping is performed on the time-series features to obtain one or more prediction results that correspond one-to-one with the one or more classification heads; Based on the one or more prediction results, calculate the classification probability of one or more categories that correspond one-to-one with the one or more classification heads, and output the category information and / or classification probability of the virtual event category.

22. The computer device according to claim 21, characterized in that, The multiple classification heads satisfy the constraint condition that the sum of the classification probabilities of each of the multiple categories is 1, and the multiple categories include the virtual event category, the scene category, and the behavior category.

23. The computer device according to claim 21, characterized in that, The combination of the first modal data stream and the second modal data stream is a combination of video data stream and audio data stream, or a combination of video data stream and telemetry data stream.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on a computer device, cause the computer device to perform the following operations: Acquire at least two modal data for characterizing virtual content, wherein the modal data includes at least one of video data and audio data; A first modal input is obtained by extracting first modal features from a first modal data stream using a first modal feature extraction network, and a second modal input is obtained by extracting second modal features from a second modal data stream using a second modal feature extraction network, wherein the first modal data stream and the second modal data stream are synchronized in time. The first modal input and the second modal input are fused to obtain a multimodal input. Then, the multimodal input is temporally modeled using a first temporal modeling branch to obtain a first temporal modeling feature. The multimodal input is temporally modeled using a second temporal modeling branch to obtain a second temporal modeling feature. Temporal feature fusion is performed on the first temporal modeling feature and the second temporal modeling feature to obtain the temporal feature corresponding to the multimodal input; Using one or more classification heads, feature mapping is performed on the time-series features to obtain one or more prediction results that correspond one-to-one with the one or more classification heads; Based on the one or more prediction results, calculate the classification probability of one or more categories that correspond one-to-one with the one or more classification heads, and output the category information and / or classification probability of the virtual event category.

25. The computer-readable storage medium according to claim 24, characterized in that, The multiple classification heads satisfy the constraint condition that the sum of the classification probabilities of each of the multiple categories is 1, and the multiple categories include the virtual event category, the scene category, and the behavior category.

26. The computer-readable storage medium according to claim 24, characterized in that, The combination of the first modal data stream and the second modal data stream is a combination of video data stream and audio data stream, or a combination of video data stream and telemetry data stream.

Images