Multimodal data processing method, server and smart pen

Abstract

This invention relates to the field of data processing technology, and particularly to a multimodal data processing method, a server, and a smart pen. The method includes receiving first modal data and second modal data sent by the smart pen; extracting first feature information from the first modal data and second feature information from the second modal data; fusing the first and second feature information to obtain multimodal joint feature information; and sending the processing result information corresponding to the multimodal joint feature information to the smart pen. The main objective of this invention is to provide a multimodal data processing method that aims to reduce the power consumption of the smart pen by having the server receive and process the data collected by the smart pen.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multimodal data processing method, a server, and a smart pen. Background Technology

[0002] With the rapid development of artificial intelligence technology, smart pens have gradually evolved from traditional writing trajectory recording tools into intelligent terminal devices with human-computer interaction capabilities. To achieve intelligent understanding of written content and voice commands, smart pens typically need to run multimodal data processing models such as image recognition, speech recognition, and semantic understanding, which places extremely high demands on the computing power of the processing unit.

[0003] If the high-performance processing unit, along with its associated memory and heat dissipation structure, which are essential for data analysis, parsing, and computation, are all integrated inside the pen body, the instantaneous and average power consumption of the processing unit inside the pen body will increase significantly. On the one hand, this will cause the battery to be consumed rapidly, severely affecting the smart pen's battery life, requiring users to charge it frequently, and greatly reducing its convenience. On the other hand, the heat generated by high power consumption is difficult to dissipate effectively in the small space of the pen body, causing the pen body to overheat, affecting the user's grip comfort and the long-term stability of the device. Summary of the Invention

[0004] The main objective of this invention is to provide a multimodal data processing method that aims to reduce the power consumption of a smart pen by having a server receive and process the data collected by the smart pen.

[0005] To achieve the above objectives, the present invention provides a multimodal data processing method applied to a server in a data processing system, wherein the data processing system further includes a smart pen, and the method includes: Receive first-modal data and second-modal data sent by the smart pen; Extract the first feature information from the first modality data, and extract the second feature information from the second modality data; The first feature information and the second feature information are fused to obtain multimodal joint feature information; The processing result information corresponding to the multimodal joint feature information is sent to the smart pen.

[0006] In one embodiment of the present invention, the steps of extracting first feature information from the first modal data and extracting second feature information from the second modal data include: The first modality data is input into the first feature extraction network, and the first feature information of the first modality data is extracted through the first feature extraction network; The second modality data is input into the second feature extraction network, and the second feature information of the second modality data is extracted through the second feature extraction network.

[0007] In one embodiment of the present invention, the step of fusing the first feature information and the second feature information includes: Obtain the first timestamp information corresponding to the first modal data and the second timestamp information corresponding to the second modal data; Based on the first timestamp information and the second timestamp information, determine the time overlap interval between the first feature information and the second feature information; Based on the overlapping interval, a first feature subset corresponding to the overlapping interval is extracted from the first feature information, and a second feature subset corresponding to the overlapping interval is extracted from the second feature information.

[0008] In one embodiment of the present invention, the step of fusing the first feature information and the second feature information further includes: Calculate the similarity between the first feature subset and the second feature subset to obtain a similarity matrix; Based on the similarity matrix, the correlation information between the first feature subset and the second feature subset is determined.

[0009] In one embodiment of the present invention, the step of fusing the first feature information and the second feature information further includes: Evaluate the first confidence information of the first feature subset and the second confidence information of the second feature subset; Based on the correlation information, the first confidence information, and the second confidence information, determine the fusion weights of the first feature subset and the second feature subset respectively; Based on the fusion weights, the first feature subset and the second feature subset are weighted and fused to obtain the multimodal joint feature information.

[0010] In one embodiment of the present invention, the step of sending the processing result information corresponding to the multimodal joint feature information to the smart pen includes: The multimodal joint feature information is input into a preset semantic model to obtain the processing result information output by the preset semantic model; The processing result information is sent to the smart pen.

[0011] The multimodal data processing method proposed in this invention is applied to a smart pen in a data processing system, wherein the data processing system further includes a server, and the method includes: Generate first modal data and second modal data; The first modal data and the second modal data are sent to the server, so that the server extracts the first feature information of the first modal data and the second feature information of the second modal data, performs fusion processing on the first feature information and the second feature information to obtain multimodal joint feature information, and sends the processing result information corresponding to the multimodal joint feature information to the smart pen.

[0012] In one embodiment of the present invention, the step of generating the first modal data and the second modal data includes: Obtain the raw data of the first mode and the raw data of the second mode; Timestamps are added to the original data of the first modality and the original data of the second modality respectively to generate the first modality data and the second modality data.

[0013] The present invention also proposes a server, the server comprising: The communication module, which communicates with the smart pen, is configured to receive first-mode data and second-mode data sent by the smart pen; The feature extraction module is configured to extract first feature information from the first modality data and second feature information from the second modality data; The fusion processing module is configured to perform fusion processing on the first feature information and the second feature information to obtain multimodal joint feature information; The communication module is further configured to send the processing result information corresponding to the multimodal joint feature information to the smart pen.

[0014] The present invention also proposes a smart pen, the smart pen comprising: An information acquisition module is configured to acquire raw data from a first modality and raw data from a second modality. A processing module, electrically connected to the information acquisition module, is configured to add timestamps to the first modality raw data and the second modality raw data, respectively, to generate timestamped first modality data and timestamped second modality data; and A communication module is electrically connected to the processing module. The communication module is configured to send the timestamped first modal data and the timestamped second modal data to the server, and to receive the processing result information sent by the server.

[0015] In this technical solution, the multimodal data processing method provided by the present invention adopts an end-cloud collaborative architecture in which "the server receives and processes multimodal data, and the smart pen is only responsible for collection and transmission". This can effectively solve the technical problems of high power consumption, short battery life and serious heat generation caused by integrating high-performance processing units into the pen body. Specifically, this method uses a server in the data processing system as the main processing entity to perform all data processing tasks. The smart pen does not need to perform any computationally intensive processing such as recognition, analysis, or understanding on the collected data. That is, after the smart pen collects the first modality data and the second modality data, it directly sends them to the server. After receiving the two modal data, the server extracts their respective feature information in the cloud and performs deep fusion processing at the feature level to generate multimodal joint feature information carrying cross-modal semantic associations. Then, the processing result information corresponding to the joint feature information is sent back to the smart pen. In this way, all high-performance computing tasks such as image recognition, speech recognition, semantic understanding, and multimodal fusion are completely transferred from the pen body to the cloud server. This means that the smart pen itself does not need to be equipped with any high-performance processing unit, matching heat dissipation structure, or large memory. The pen tip only needs to perform extremely low-power operations such as data collection, sending, and receiving. The power consumption of the pen tip is significantly reduced, and the battery life is greatly extended. At the same time, since the smart pen does not need to perform any high-power calculations inside, the pen body will not generate significant heat accumulation in a small space, completely avoiding the problem of overheating and ensuring user grip comfort and long-term stability of the device. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0017] Figure 1 A first flowchart of an embodiment of the multimodal data processing method provided by the present invention; Figure 2 This is a second flowchart of an embodiment of the multimodal data processing method provided by the present invention; Figure 3 This is a third flowchart of an embodiment of the multimodal data processing method provided by the present invention; Figure 4 This is a fourth flowchart of an embodiment of the multimodal data processing method provided by the present invention; Figure 5 This is a fifth flowchart of an embodiment of the multimodal data processing method provided by the present invention; Figure 6A sixth flowchart of an embodiment of the multimodal data processing method provided by the present invention; Figure 7 A seventh flowchart of an embodiment of the multimodal data processing method provided by the present invention; Figure 8 This is the eighth flowchart of an embodiment of the multimodal data processing method provided by the present invention.

[0018] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0021] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the meaning of "and / or" throughout the text includes three parallel solutions; for example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of a person skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0022] The main objective of this invention is to provide a multimodal data processing method that aims to reduce the power consumption of a smart pen by having a server receive and process the data collected by the smart pen.

[0023] To achieve the above objectives, the multimodal data processing method provided by this invention is applied to a server in a data processing system. The data processing system also includes a smart pen. (See attached image.) Figure 1 The methods include: S10: Receive first-mode data and second-mode data sent by the smart pen; S20: Extract the first feature information of the first modality data and extract the second feature information of the second modality data; S30: The first feature information and the second feature information are fused to obtain multimodal joint feature information; S40: Send the processing result information corresponding to the multimodal joint feature information to the smart pen.

[0024] In step S10, the first modal data and the second modal data sent by the smart pen are received. Specifically, the first modal data and the second modal data are different types of external information collected by the smart pen through its information acquisition module. For example, the first modal data can be image data of the writing medium surface (including visual content such as the graphics drawn by the user and the text written by the user) collected by the smart pen's visual acquisition module (such as a camera), and the second modal data can be user voice signal data (including voice content such as instructions, questions or descriptions issued by the user) collected by the smart pen's acoustic acquisition module (such as a microphone).

[0025] The data processing system includes a server and a smart pen. The smart pen establishes a communication connection with the server via a wireless network (such as Wi-Fi, Bluetooth, or mobile communication networks). The smart pen sends the collected multimodal data to the server, which then performs subsequent data processing steps. The smart pen itself does not participate in any computationally intensive data analysis. It should be noted that the specific types of the first and second modal data (such as image data, audio data, text data, etc.) can be selected according to the type of acquisition module configured in the smart pen; this invention does not limit the specific data format.

[0026] In step S20, first feature information of the first modal data and second feature information of the second modal data are extracted. Specifically, the feature information is the transformation of the original modal data (such as a pixel matrix or audio waveform) into a high-dimensional numerical vector representation that can characterize its core semantic content.

[0027] The first feature information is a vectorized representation obtained after feature extraction of the first modality data (such as image data). It numerically represents the key visual semantic elements in the image, including the basic shape of the graphic, the position of the geometric boundary, the spatial relationship between different regions, and other visual semantic information.

[0028] The second feature information is a vectorized representation obtained after feature extraction from the second modality data (such as speech signals). It numerically represents the key semantic information in the speech—including the user's expressed instructions, key entities, referential words, and other speech semantic information. Through feature extraction, the original data is transformed into a unified representation that can be processed by subsequent fusion processing steps.

[0029] In step S30, the first feature information and the second feature information are fused to obtain multimodal joint feature information. Specifically, the fusion process refers to integrating and associating feature information from different modalities at the feature level, rather than simply splicing the recognition results of the two modalities together.

[0030] Since visual and acoustic features reflect the user's input intent from different perspectives—visual information provides spatial and shape information of "what the user drew," while acoustic information provides intent and referential information of "what the user wants to express"—the two are semantically complementary.

[0031] The purpose of fusion processing is to make full use of this complementarity, semantically align and integrate visual and acoustic features at the feature level, and generate a multimodal joint feature information that simultaneously integrates the semantic information of the two modalities, so that the joint feature information can be understood and used by subsequent processing steps.

[0032] In step S40, the processing result information corresponding to the multimodal joint feature information is sent to the smart pen. Specifically, the processing result information is the response content generated after semantic understanding or reasoning based on the multimodal joint feature information. After generating the processing result information, the server transmits it back to the smart pen via a wireless network. After receiving the processing result information, the smart pen can present it to the user through its output module (such as a display module or speaker).

[0033] With the cooperation of steps S10 to S40 above, all data processing is completed on the server side. The smart pen only performs data collection and transmission and result reception and presentation, without undertaking any data analysis and processing tasks.

[0034] In one application scenario, a user holds a smart pen and draws a shaded geometric shape on paper while verbally stating, "Calculate the area of ​​the shaded part of this shape." The smart pen's camera captures image data of the writing area in real time, and its microphone simultaneously records the user's voice command. The smart pen then transmits the image and voice data to a server via a wireless network.

[0035] After receiving these two modalities of data, the server first extracts feature vectors from the image data that can represent visual information such as the shape, boundary position, and filled area of ​​the graphic. At the same time, it extracts feature vectors from the speech data that can represent the intention of the instruction and entity information such as "calculate", "shaded part", and "area". Subsequently, the server fuses these two sets of feature vectors at the feature level - identifying the correspondence between the "shaded part" in the speech and the specific filled area in the image - generating a multimodal joint feature that carries both semantic information of "what graphic the user drew" and "what operation the user wants to perform on which area of ​​the graphic".

[0036] Based on the joint features, the server generates corresponding processing results, such as calculation steps and area results, and transmits these results back to the smart pen via a wireless network. The smart pen can then present the calculation steps and answers to the user through its screen or inform the user of the results through its speaker. Throughout the process, all computationally intensive processing, such as image recognition, speech recognition, semantic understanding, and multimodal fusion, is completed on the server. The smart pen only performs lightweight operations such as acquisition, transmission, and reception, resulting in extremely low power consumption and long battery life.

[0037] In this technical solution, the multimodal data processing method provided by the present invention adopts an end-to-cloud collaborative architecture in which "the server receives and processes multimodal data, and the smart pen is only responsible for collection and transmission". This can effectively solve the technical problems of high power consumption, short battery life and serious heat generation caused by integrating high-performance processing units into the pen body. Specifically, this method uses a server in the data processing system as the main processing entity to perform all data processing tasks. The smart pen does not need to perform any computationally intensive processing such as recognition, analysis, or understanding on the collected data. That is, after the smart pen collects the first modality data and the second modality data, it directly sends them to the server. After receiving the two modal data, the server extracts their respective feature information in the cloud and performs deep fusion processing at the feature level to generate multimodal joint feature information carrying cross-modal semantic associations. Then, the processing result information corresponding to the joint feature information is sent back to the smart pen. In this way, all high-performance computing tasks such as image recognition, speech recognition, semantic understanding, and multimodal fusion are completely transferred from the pen body to the cloud server. This means that the smart pen itself does not need to be equipped with any high-performance processing unit, matching heat dissipation structure, or large memory. The pen tip only needs to perform extremely low-power operations such as data collection, sending, and receiving. The power consumption of the pen tip is significantly reduced, and the battery life is greatly extended. At the same time, since the smart pen does not need to perform any high-power calculations inside, the pen body will not generate significant heat accumulation in a small space, completely avoiding the problem of overheating and ensuring user grip comfort and long-term stability of the device.

[0038] In one embodiment of the present invention, please refer to Figure 2 Step 20 includes: S21: Input the first modality data into the first feature extraction network, and extract the first feature information of the first modality data through the first feature extraction network; S22: Input the second modality data into the second feature extraction network, and extract the second feature information of the second modality data through the second feature extraction network.

[0039] In step S21, the first modality data is input into the first feature extraction network, and the first feature information of the first modality data is extracted through the first feature extraction network. Specifically, the first feature extraction network is a pre-trained deep learning model (such as a convolutional neural network or a visual Transformer), which is deployed on the server side and used to perform feature encoding on the input raw image data.

[0040] When the server receives the first modal data (such as image data) sent by the smart pen, it inputs the image data into the first feature extraction network. The network extracts features from the image layer by layer through its internal convolutional layers, pooling layers and other structures, and transforms the original pixel matrix into a high-dimensional feature vector form, which is the first feature information.

[0041] The first feature information numerically represents the key visual semantic elements in the image—including the basic shape of the graphic, the position of the geometric boundary, the spatial relationship between different regions, and the distribution of fill or shadow within the region, enabling subsequent fusion processing steps to understand and manipulate visual semantics at the feature level.

[0042] In step S22, the second modality data is input into the second feature extraction network, which extracts the second feature information of the second modality data. Specifically, the second feature extraction network is a pre-trained deep learning model (such as a recurrent neural network, Transformer, or Conformer), which is also deployed on the server side and is used to perform feature encoding on the input raw speech data.

[0043] When the server receives the second modal data (such as a voice signal) sent by the smart pen, it inputs the voice data into the second feature extraction network. The network extracts the temporal features of the voice signal through its internal coding structure, and transforms the original audio waveform or spectrum data into a high-dimensional feature vector form, that is, the second feature information.

[0044] The second feature information represents the key semantic information in speech in a numerical way—including the user's expressed intention (such as asking, requesting calculation, requesting explanation, etc.), key entities in the instruction (such as "area", "perimeter", "color", etc.), and semantic reference information of referential words (such as "this" and "that"), so that subsequent fusion processing steps can understand and manipulate speech semantics at the feature level.

[0045] In one application scenario, after receiving image and voice data from a smart pen, the server inputs the image data into a first feature extraction network. This network outputs a high-dimensional visual feature vector, where the activation states of different dimensions represent visual semantic information such as "circular outline," "internal filling area," and "shadow distribution," encoding the complete visual semantics of the user's drawn shape into a numerical representation. Simultaneously, the server inputs the voice data into a second feature extraction network, which outputs a high-dimensional acoustic feature vector. The activation states of different dimensions in this vector represent the intention of the command "calculate," key entities such as "shaded area" and "area," and the pronoun "this shape," encoding the complete semantics of the user's voice command into a numerical representation.

[0046] At this point, the raw data of both modalities have been transformed into a unified feature vector form that can be processed in subsequent fusion processing steps, and the two feature vectors are located in the same semantic space. This allows the visual object referred to by the pronoun "this graphic" in speech to be determined in subsequent steps by calculating the correlation between the two – that is, the feature components corresponding to the "circular outline" and its "shadow area" in the image features.

[0047] In one embodiment of the present invention, please refer to Figure 3 Step 30 includes: S31: Obtain the first timestamp information corresponding to the first modal data and the second timestamp information corresponding to the second modal data; S32: Determine the time overlap interval between the first feature information and the second feature information based on the first timestamp information and the second timestamp information; S33: Based on the overlapping interval, extract the first feature subset corresponding to the overlapping interval from the first feature information, and extract the second feature subset corresponding to the overlapping interval from the second feature information.

[0048] First, it should be noted that steps S31 to S33 achieve precise time alignment and feature subset extraction through the "Dynamic Time Warping Algorithm". The Dynamic Time Warping Algorithm is an algorithm used to measure the similarity between two time series. It can perform "flexible alignment" on sequences of different lengths, speech rates, or action rhythms. That is, by finding the optimal matching path between two sequences, each point in the sequence can establish a mapping relationship with one or more semantically corresponding points in the other sequence.

[0049] In this scheme, the visual feature sequence and the acoustic feature sequence may be out of sync on the time axis—the user may speak while drawing, or add explanations after drawing. The acquisition duration, sampling frequency, and semantic correspondence of the two are not completely consistent. The dynamic time warping algorithm can automatically "compress" or "stretch" the time axis to find the optimal correspondence between the two sequences.

[0050] Based on this, in step S31, the first timestamp information corresponding to the first modal data and the second timestamp information corresponding to the second modal data are obtained. Specifically, after the server receives the first modal data and the second modal data sent by the smart pen, it parses the timestamp information carried by each data packet, including the acquisition time corresponding to each frame of image data (such as 0.00 second, 0.03 second, 0.06 second, etc.) and the acquisition time corresponding to each segment of voice data (such as 0.01 second, 0.02 second, etc.). These timestamps provide the basic time anchor points for subsequent dynamic time warping algorithms to perform time alignment, enabling the data of the two modalities to be processed under a unified time reference system.

[0051] In step S32, the overlapping interval between the first feature information and the second feature information in time is determined based on the first timestamp information and the second timestamp information. Specifically, the server determines the interval that the two modal data cover together on the time axis, i.e., the overlapping interval, based on the timestamp ranges of the two modal data. For example, if the timestamp range of image data is from second 0 to second 8, and the timestamp range of speech data is from second 3 to second 7, then the overlapping interval is from second 3 to second 7. The significance of the overlapping interval is that visual features and acoustic features coexist within this time period, providing a temporal basis for establishing cross-modal semantic association. In contrast, time periods with only a single modality (such as periods with only images and no speech) do not have the conditions for cross-modal association.

[0052] In step S33, based on the overlapping interval, a first feature subset corresponding to the overlapping interval in the first feature information is extracted, and a second feature subset corresponding to the overlapping interval in the second feature information is extracted.

[0053] Specifically, after determining the overlapping interval, the system does not simply extract all features within that interval. Instead, it first extracts the initial feature segments corresponding to the overlapping interval, and then inputs the two sets of initial feature segments into the dynamic time warping algorithm. The dynamic time warping algorithm performs elastic matching on the two feature sequences. The dynamic time warping algorithm calculates the cumulative distance matrix between the two feature sequences through dynamic programming to find an optimal matching path from the start point to the end point of the sequence.

[0054] Taking the voice phrase "this shape" as an example, the time window corresponding to this voice segment is from 3.5 seconds to 4.2 seconds. The dynamic time warping algorithm will perform "non-linear alignment" between the acoustic feature sequence within this time window and the image frames within the same time period in the visual feature sequence. If the user's drawing speed is uneven between 3.5 seconds and 4.2 seconds, resulting in different distribution densities of image frames, the dynamic time warping algorithm can adaptively establish a mapping between each frame of voice features and the corresponding frame or multiple frames of visual features, thereby accurately determining the visual object pointed to by the acoustic feature "this shape" (i.e., the circle and its shadow area being drawn around the 4th second).

[0055] Through the flexible alignment of the dynamic time warping algorithm, the final "first feature subset corresponding to the overlapping interval" and "second feature subset" are feature sets that have been accurately matched in the time dimension. This makes the establishment of subsequent cross-modal associations no longer limited by the inconsistency of the two modal data in terms of duration, sampling rate and time axis distribution, and provides accurately aligned feature inputs for the fusion processing of the attention mechanism.

[0056] In one application scenario, after the server determines the overlap interval to be from the 3rd to the 7th second, it inputs the visual feature sequence (approximately 120 frames of image features) and the acoustic feature sequence (approximately 4 seconds of speech features) within this interval into a dynamic time warping algorithm. The dynamic time warping algorithm finds an optimal matching path by performing frame-by-frame elastic matching on the two sequences. In this path, the speech "this shape" is emitted at the 4th second, and the dynamic time warping algorithm maps it to the visual features from the 3.8th to the 4.2th second—the user begins drawing the circular outline at the 3.8th second, completes the main body of the circle at the 4.2nd second, and begins drawing the shadow. Therefore, "this shape" is precisely associated with the circle and its shadow area before and after the 4th second, rather than the auxiliary lines at the beginning of the drawing or other shapes after the drawing is completed. Through the elastic alignment mechanism of the dynamic time warping algorithm, the system successfully overcomes the alignment difficulties caused by the inconsistency between the speech and action rhythms, ensuring the accuracy of subsequent fusion processing.

[0057] In one embodiment of the present invention, please refer to Figure 4 Step 30 also includes: S34: Calculate the similarity between the first feature subset and the second feature subset to obtain the similarity matrix; S35: Based on the similarity matrix, determine the correlation information between the first feature subset and the second feature subset.

[0058] In step S34, the similarity between the first feature subset and the second feature subset is calculated to obtain a similarity matrix. Specifically, the similarity matrix is ​​a two-dimensional numerical matrix formed by calculating the "cosine similarity" between each feature component in the first feature subset and each feature component in the second feature subset.

[0059] Cosine similarity measures the semantic similarity between two feature vectors by calculating the cosine of the angle between them in a high-dimensional space. For example, the closer the cosine value is to 1, the more consistent the directions of the two features in the semantic space are, meaning that the semantic relevance is higher. The closer the cosine value is to 0, the more orthogonal the directions of the two features are in the semantic space, meaning that the semantics are unrelated or weakly related.

[0060] The first feature subset contains visual feature vectors at each time step within the overlapping interval (each vector represents the visual semantics of the image at that time step), and the second feature subset contains acoustic feature vectors at each time step within the overlapping interval (each vector represents the acoustic semantics of the speech at that time step). By calculating the cosine similarity between each feature vector in the first feature subset and each feature vector in the second feature subset, a two-dimensional similarity matrix is ​​formed. This matrix fully reflects the degree of similarity between visual features and acoustic features at each time step and in each semantic dimension.

[0061] In step S35, the correlation information between the first feature subset and the second feature subset is determined based on the similarity matrix. Specifically, the correlation information is a set of information that characterizes the semantic correspondence between visual and acoustic features, determined by the numerical distribution in the similarity matrix.

[0062] The correlation information specifically includes: high similarity feature pairs selected from the similarity matrix with similarity higher than a preset threshold. These high similarity feature pairs represent which semantic unit in speech corresponds to which region or object in vision.

[0063] For example, when the phrase "this graphic" appears in speech, the cosine similarity between the acoustic feature vector corresponding to the speech segment and the visual feature component corresponding to the "graphic outline" in the visual features is significantly higher than the similarity between the vector and other visual feature components (such as the background region). This high similarity feature is retained as part of the correlation information to indicate that "the 'graphic' in the acoustics corresponds semantically to the graphic outline in the vision".

[0064] Meanwhile, the correlation information also includes the relative strength relationship between each pair of highly similar features, that is, which pair of features is more strongly correlated and which pair of features is relatively weakly correlated, providing a quantitative basis for dynamic weight calculation in subsequent steps.

[0065] In one application scenario, the system performs pairwise cosine similarity calculations on the first feature subset (visual features) and the second feature subset (acoustic features) within the overlapping interval, forming a similarity matrix. In this matrix, the cosine similarity between the component of the acoustic feature vector corresponding to the "shaded area" and the component of the visual feature vector corresponding to the "filled area of ​​the shape" is 0.92, indicating that they are highly consistent in direction in the semantic space. The cosine similarity between the component of the acoustic feature vector corresponding to "this shape" and the component of the visual feature vector corresponding to the "circular outline" is 0.88. However, the cosine similarity between the component of the acoustic feature vector corresponding to "calculation" and the component of the visual feature vector corresponding to any visual region is relatively low (approximately 0.1 to 0.3), because "calculation" itself is an operation command and does not point to any specific visual object.

[0066] Subsequently, the system determines the correlation information based on the similarity matrix and outputs the high similarity feature pairs ("shaded area → filled area" cosine similarity 0.92, "this shape → circular outline" cosine similarity 0.88) as the correlation information.

[0067] These high-similarity feature pairs indicate the semantic correspondence between pronouns in speech and corresponding regions in vision—that is, "shaded area" corresponds to the gray-filled area in the graphic, and "this graphic" corresponds to the overall area containing the circular outline. In the conflict scenario of drawing a circle and saying it's a square, because the cosine similarity between the acoustic features of "square" and the visual features of "circle" is low (e.g., 0.25), this feature pair will not be included in the correlation information, thus ensuring that "square" and "circle" are not incorrectly bound in subsequent fusion.

[0068] Through the above steps, the system completes the entire process from "semantic relevance quantification" to "cross-modal correspondence screening", providing a semantic correspondence basis for calculating dynamic weights in subsequent steps.

[0069] In one embodiment of the present invention, please refer to Figure 5 Step 30 also includes: S36: Evaluate the first confidence information of the first feature subset and the second confidence information of the second feature subset; S37: Determine the fusion weights of the first feature subset and the second feature subset based on the correlation information, the first confidence information, and the second confidence information; S38: Based on the fusion weight, the first feature subset and the second feature subset are weighted and fused to obtain multimodal joint feature information.

[0070] In step S36, the first confidence information of the first feature subset and the second confidence information of the second feature subset are evaluated. Specifically, the confidence information is used to characterize the quality reliability of each modality data itself—visual confidence is evaluated based on factors such as image clarity, integrity, and lighting conditions (e.g., whether the image is blurry, whether the image is missing, whether there is occlusion, etc.), and acoustic confidence is evaluated based on factors such as speech clarity, environmental noise level, and whether there is dropout or noise interference (e.g., whether the speech is clear, whether the background is noisy, whether the instruction is complete, etc.).

[0071] The confidence assessment is independent of cross-modal correlation and focuses only on the signal quality of the single modality data itself. It does not involve the semantic relationship between vision and acoustics, i.e., "whether the vision itself is good or not" and "whether the speech itself is good or not".

[0072] For confidence assessment of visual modalities, image quality can be comprehensively judged from multiple dimensions such as image sharpness, brightness and contrast, noise level, integrity and texture information. Sharpness is assessed by calculating the image's gradient information, edge strength, or local changes. The sharper the image and the sharper the edges, the higher the sharpness index; a blurry image (such as out-of-focus or motion blur) has a low sharpness index. Brightness and contrast are assessed by analyzing the image's grayscale distribution and dynamic range. Images with moderate brightness and good contrast can present the written content more clearly, while images that are too dark, too bright, or have too low contrast make the content difficult to read. Noise level is assessed by analyzing whether there is interference such as salt-and-pepper noise or Gaussian noise in the image. Images with high signal-to-noise ratio and low noise levels are of good quality, while images with severe noise will obscure the writing details. Integrity is assessed by detecting whether there are missing areas, obstructions, or edge truncation in the image. Images with complete and missing parts have high confidence; if the captured writing area is incomplete (such as part of the graphic being obscured or exceeding the image boundary), the visual confidence is correspondingly reduced. Color and texture information are assessed by analyzing the image's color distribution and texture complexity. Images with rich colors and clear textures can provide more complete visual information, while images with single grayscale or blurry textures provide limited information.

[0073] The visual confidence level is obtained by comprehensively evaluating the above parameters. The higher the quality of each parameter, the higher the visual confidence level. If any parameter deteriorates, the visual confidence level will decrease accordingly.

[0074] For example, when a user draws a black circle but the captured image is blurry, both the sharpness index and edge strength are low, resulting in a low overall visual confidence score, indicating that the image data quality is low.

[0075] For acoustic modalities, confidence assessment can comprehensively judge speech quality from multiple dimensions, including signal-to-noise ratio (SNR), signal amplitude and dynamic range, clipping, speech activity clarity, and coding quality. SNR is assessed by calculating the energy ratio of the speech signal to background noise. A high SNR indicates clear speech and a quiet background, resulting in high acoustic confidence; a low SNR indicates a noisy background, with speech being masked by noise, resulting in low acoustic confidence. Signal amplitude and dynamic range are assessed by analyzing the maximum and minimum amplitudes of the waveform. A moderate signal amplitude and full utilization of the dynamic range indicate sufficient speech signal strength, resulting in high acoustic confidence; a signal amplitude that is too small (too quiet) will lead to increased quantization error, resulting in low acoustic confidence. Clipping is determined by detecting whether a portion of the waveform exceeding a preset threshold is truncated. No clipping... A complete speech signal with high acoustic confidence indicates high acoustic confidence. Clipping indicates that the original speech volume is too high, causing signal distortion, and results in low acoustic confidence. Speech activity clarity is assessed by analyzing the distribution of speech segments and silence segments, and the presence of dropouts or noise. Speech signals that are continuous, without dropouts or noise interference have high acoustic confidence, while speech signals with excessively long silence segments or sudden noise have low acoustic confidence. Coding quality is assessed by analyzing the coding format and bit depth of the speech signal. Speech signals with high bit depth have high fidelity and high acoustic confidence, while speech signals with low bit depth have significant quantization noise and low acoustic confidence.

[0076] The acoustic confidence score is obtained by comprehensively evaluating the above parameters. The higher the quality of each parameter, the higher the acoustic confidence score; if any parameter degrades, the acoustic confidence score will decrease accordingly. For example, when a user says "What color is this circle?", the speech is clear, there is no noise interference, and the signal-to-noise ratio is 20dB (far higher than the good standard of 15dB). The comprehensive evaluation yields a high acoustic confidence score (e.g., 0.92), indicating that the speech data is of reliable quality.

[0077] In step S37, the fusion weights of the first feature subset and the second feature subset are determined based on the correlation information, the first confidence information, and the second confidence information.

[0078] Specifically, the fusion weight is the weight value assigned to each feature component in the first feature subset and the second feature subset after combining confidence information and correlation information. This weight value determines the degree of influence of each feature component in the multimodal joint feature information.

[0079] When a modality has high confidence and its feature has a high cosine correlation with another modality, that feature component is assigned a high fusion weight, indicating that the feature is both reliable and semantically consistent, and should dominate the fusion result. When a modality has low confidence, even if its correlation is high, its fusion weight is reduced to avoid low-quality data contaminating the fusion result. When the cosine correlation between two modalities is low, even if both have high confidence, their respective fusion weights are reduced accordingly to suppress the influence of semantically mismatched features on the fusion result. By combining confidence and correlation to determine dynamic weights, the system can automatically adjust the "discourse power" of each modality feature in different scenarios.

[0080] In one embodiment, an attention mechanism is used to determine the fusion weights of the first feature subset and the second feature subset based on relevance information, first confidence information, and second confidence information. Specifically, the attention mechanism determines the fusion weights in the following way: First, the first feature subset is linearly transformed into a query vector Q, and the second feature subset is linearly transformed into a key vector K and a value vector V. The linear transformation is achieved through a learnable weight matrix, so that Q, K, and V are located in the same feature space for similarity calculation.

[0081] In this context, the query vector Q represents the semantic representation of each part of the visual features, and the key vector K represents the semantic representation of each part of the acoustic features. The one-to-one correspondence between the two is used to establish cross-modal semantic correspondence. The value vector V carries the original semantic information of the acoustic features and is used to fuse them according to the calculated weights in the subsequent weighted summation process.

[0082] Secondly, the similarity between the query vector Q and the key vector K is calculated. This similarity calculation integrates relevance information and confidence information: based on the relevance information, the cosine similarity between each component in Q and K is determined. A higher cosine similarity indicates that the visual feature components and acoustic feature components tend to be consistent in direction in the semantic space, that is, they correspond to each other semantically. Based on the first confidence information, the components in Q are weighted and adjusted. When the visual confidence is low, the effective values ​​of each component in Q are reduced accordingly to suppress the influence of low-quality visual features in the similarity calculation. Based on the second confidence information, the components in K are weighted and adjusted. When the acoustic confidence is low, the effective values ​​of each component in K are reduced accordingly to suppress the influence of low-quality acoustic features in the similarity calculation.

[0083] The similarity between Q and K, adjusted for confidence, is used as the attention score. This score reflects both the semantic relevance between modalities and the data reliability of each modality. Next, the attention score is input into the Softmax function for normalization, resulting in a set of fusion weights that sum to 1. In this fusion weight set, feature components with high similarity receive a higher weight proportion, while feature components with low similarity have their weights compressed.

[0084] Finally, the multimodal joint feature information is obtained by weighted summation of the fusion weights and the value vector V. High-weight feature components dominate the joint feature information, while the influence of low-weight feature components is effectively suppressed. Through this attention mechanism, when both visual and acoustic aspects are clear and semantically consistent, the similarity between Q and K is high, resulting in a high fusion weight, and both visual and acoustic features dominate the fusion result. When visual aspects are blurry but speech is clear, low visual confidence leads to a reduction in the effective value of Q, and the fusion weight of visual features decreases accordingly, while high-weight speech features dominate the fusion result. When a circle is drawn but a "square" is spoken, the cosine similarity between Q and K is low, resulting in a low attention score, and the fusion weight of semantically conflicting features is suppressed, thus avoiding misbinding that could affect the inference of the large language model.

[0085] In step S38, the first feature subset and the second feature subset are weighted and fused according to the fusion weights to obtain multimodal joint feature information. Specifically, weighted fusion refers to applying the fusion weights of the first and second feature subsets, calculated through the attention mechanism in step S37, to the feature components of the first and second feature subsets respectively, and then merging the weighted visual features with the weighted acoustic features.

[0086] Specifically, each feature component in the first feature subset is multiplied by the corresponding visual fusion weight to obtain a weighted visual feature subset; each feature component in the second feature subset is multiplied by the corresponding acoustic fusion weight to obtain a weighted acoustic feature subset; then, the weighted visual feature subset and the weighted acoustic feature subset are added or concatenated element by element to generate a unified feature vector, which is the multimodal joint feature information.

[0087] It should be noted that multimodal joint feature information is a unified feature representation that integrates visual and acoustic modal information. In this joint feature information, the features of the source modality (whether visual or acoustic) with high weights dominate the fusion result, while the influence of features of low weights is effectively suppressed. Therefore, high-weight feature components become the core focus of large language models.

[0088] Specifically, in the matching scenario—where a circle is drawn and a "circle" is spoken—the acoustic features of "this circle" and the visual features of "circular outline" are both given high weights. The weighted fusion of joint feature information simultaneously preserves the semantic information of both with strong signals, and a precise cross-modal correspondence has been established between the two. After receiving the data, the large language model can directly infer that the object the user is asking about is the circle in the picture.

[0089] In mismatch scenarios where the visual image is blurry but the speech is clear, the speech features have a high weight and the visual features have a low weight. The joint feature information after weighted fusion is dominated by speech features, but the joint feature information still retains the temporal binding relationship between the speech "this circle" and the corresponding object in the vision. Based on this, the large language model can understand that the object referred to by the user is the object that was photographed in the vision, and give a reasonable response based on limited visual features.

[0090] In the semantic conflict scenario of drawing a circle and saying it's a square, the acoustic features of "square" and the visual features of "circle" are given low weights due to their low cosine correlation. The influence of these low-weighted feature components is effectively suppressed after weighted fusion, and no incorrect binding between "square" and "circle" is formed in the joint feature information. The large language model will not make inference errors due to receiving contradictory information.

[0091] Through the aforementioned weighted fusion, the multimodal joint feature information is no longer a simple superposition of visual and acoustic features, but a high-quality joint representation that has undergone semantic filtering and weight optimization through an attention mechanism. In this representation, high-quality and semantically matching features are enhanced, while low-quality or semantically mismatched features are suppressed. The final information input into the subsequent large language model is a clear and accurate cross-modal semantic association result.

[0092] In one application scenario, a user draws a black circle on paper using a smart pen while verbally stating, "What color is this circle?" The smart pen sends the image and voice data to the server. The server then executes steps S31 to S35 to complete time alignment and correlation calculation before proceeding to the fusion processing stage, steps S36 to S38.

[0093] First, the server evaluated visual and acoustic confidence scores separately. For the visual modality, the images captured by the camera were clear and complete—the circular outlines were distinct, the lighting was uniform, there were no obstructions, and there was no blurring. Both the sharpness index and edge strength were high. After evaluating all parameters, the visual confidence score was 0.92, indicating reliable image data quality. For the acoustic modality, the user's speech was clear, without noise interference, with a signal-to-noise ratio of 20dB (far exceeding the good standard of 15dB), and without clipping or dropouts. After comprehensive evaluation, the acoustic confidence score was 0.95, indicating that the speech data quality was also reliable.

[0094] Next, the server uses an attention mechanism to determine the fusion weights of each feature component. First, the visual feature subset is linearly transformed into a query vector Q, and the acoustic feature subset is linearly transformed into a key vector K and a value vector V. Then, the similarity between Q and K is calculated, which combines relevance and confidence information: according to the relevance information obtained in step S35, the cosine similarity between the acoustic features of "this circle" and the visual features of "circular region" is 0.93, indicating high relevance; simultaneously, both the visual confidence (0.92) and acoustic confidence (0.95) are high, and neither adjusts the weights of Q and K, thus maintaining a high attention score. After normalization, the acoustic feature component of "this circle" receives a high fusion weight of 0.9, and the corresponding visual feature component of "circular region" receives a high fusion weight of 0.88, both playing a dominant role in the fusion process.

[0095] Finally, the server performs weighted fusion of the first and second feature subsets based on the aforementioned fusion weights. The high-weighted acoustic feature component of "this circle" and the high-weighted visual feature component of "circular region" are firmly bound together after weighted summation. The generated multimodal joint feature information simultaneously retains the phonological semantics of "the user asked about a circle" and the visual semantics of "a circle exists in the image" with strong signals, and a precise cross-modal correspondence has been established between the two. After receiving this joint feature information, the large language model can accurately understand the user's intent—the object the user is pointing to is the black circle in the image—and thus directly provide the correct color judgment result.

[0096] In another scenario, the user draws a circle but verbally describes "what color is this square?" The confidence assessment result of step S36 remains unchanged (visual confidence 0.92, acoustic confidence 0.95, both reliable). According to the correlation information, the cosine similarity between the acoustic features of "this square" and the visual features of "circular area" is only 0.25, indicating low correlation.

[0097] When calculating the similarity between Q and K, the attention mechanism suffers from low cosine similarity, resulting in a low attention score between Q and K even with high confidence. After normalization, the acoustic feature component fusion weight for "this square" is reduced to 0.25, and the visual feature component fusion weight for "circular region" is also reduced to 0.2. In weighted fusion, the influence of these two low-weight features is effectively suppressed, preventing erroneous bindings between "square" and "circle" in the generated multimodal joint feature information. Consequently, the subsequent large language model will not make incorrect color judgments due to receiving contradictory information.

[0098] In another scenario, a user draws a circle and verbally states "What color is this circle?", but the captured image is blurry (e.g., motion blur due to hand tremors). Based on this, the visual confidence score is low (e.g., 0.35), while the acoustic confidence score remains at 0.95. Simultaneously, when the attention mechanism calculates the similarity between Q and K, the low visual confidence reduces the effective values ​​of each component in Q. Even though the cosine similarity between the acoustic features of "this circle" and the visual features of the "circular region" is high (0.93), the attention score corresponding to the visual features remains low due to the weakened effective value of Q. After normalization, the fusion weight of the visual feature components is reduced to 0.3, while the acoustic feature components, due to their high confidence and high correlation, receive a high weight of 0.85. Subsequently, the influence of low-weight visual features is effectively suppressed during weighted fusion, while high-weight acoustic features dominate the fusion result. However, the temporal binding relationship between the speech "this circle" and the corresponding visual object is still preserved in the joint feature information.

[0099] After receiving the joint feature information, the large language model mainly relies on the voice information to infer that the user is asking about "this circle", and combines limited visual features (albeit blurry) to infer and inform the user through the smart pen that the circle is black.

[0100] As can be seen from the comparison of the different scenarios above, this solution, through the coordinated efforts of confidence assessment, correlation calculation and dynamic weight allocation, can automatically generate reasonable fusion weights based on data quality and semantic consistency, ensuring that the fusion result is always dominated by the most reliable and relevant information at present.

[0101] In one embodiment of the present invention, please refer to Figure 6 Step 40 includes: S41: Input the multimodal joint feature information into the preset semantic model to obtain the processing result information output by the preset semantic model; S42: Send the processing result information to the smart pen.

[0102] In step S41, the multimodal joint feature information is input into a preset semantic model to obtain the processing result information output by the preset semantic model. Specifically, the preset semantic model is a pre-trained large language model deployed on the server side, used to perform semantic understanding and reasoning based on the input multimodal joint feature information, and generate corresponding natural language responses.

[0103] The pre-defined semantic model is obtained through multimodal joint training. The training process is as follows: A large amount of sample image data (including various geometric shapes, scene images, handwritten content, and other visual information) and corresponding sample speech data (including instructions describing graphic attributes, questions asking for graphic information, instructions requesting computational operations, and other speech content) are collected. Annotators annotate each pair of image-speech samples with corresponding text description samples (such as graphic attribute information, correct answers to questions, and corresponding operation results of instructions). The image samples are input into the first feature extraction network, and the speech samples are input into the second feature extraction network. The features output by the two networks are fused and then input into the large language model to be trained. The annotated text descriptions are used as supervision signals to train the model, enabling the model to learn to map the joint features of images and speech to the correct text output, thereby establishing a mapping relationship between multimodal input and natural language output.

[0104] During training, the model learns how to understand referential relationships in user speech. For example, when the speech mentions "this graphic," the model learns to look up associated visual features from the joint features to determine the referent and generate a corresponding response based on that referent.

[0105] It should be noted that the specific training methods of the preset semantic model (such as the method of collecting training data, annotation specifications, hyperparameter settings for model training, etc.) are well-known technologies in this field, and this invention does not limit their specific implementation methods.

[0106] When the server inputs the generated multimodal joint feature information into the preset semantic model, the model, based on its multimodal semantic understanding capabilities learned during training, infers from the input joint feature information and generates corresponding processing result information. The processing result information is a text response in natural language form. For example, when a user asks about the color of a shape, it outputs "The shape is black" or "The circle is black." When a user requests to calculate the area, it outputs "The area of ​​the shaded part of the shape is XX square centimeters" along with the corresponding calculation steps.

[0107] Understandably, to adapt to the input interface of the predefined semantic model, the server may include a feature adaptation layer (such as a projection layer) to map the multimodal joint feature information to the feature dimensions required by the predefined semantic model. When the server inputs the generated multimodal joint feature information into the predefined semantic model, the model performs reasoning based on its semantic understanding capabilities and generates the corresponding processing result information.

[0108] In step S42, the processing result information is sent to the smart pen. Specifically, after receiving the processing result information (i.e., the natural language text reply), the server transmits the processing result information back to the smart pen via a wireless network (such as Wi-Fi or a mobile communication network). After receiving the processing result information, the smart pen's communication module presents it to the user through its output module (such as a display module or speaker)—for example, displaying it on a long, narrow e-ink screen for silent reading in a quiet environment, or playing a voice reply through the speaker to meet the needs of scenarios where it is inconvenient to view the screen.

[0109] The multimodal data processing method proposed in this invention is applied to a smart pen in a data processing system. The data processing system also includes a server. Please refer to [link / reference]. Figure 7 The methods include: S11: Generate first modal data and second modal data; S21: Send the first modal data and the second modal data to the server so that the server can extract the first feature information of the first modal data and the second feature information of the second modal data, perform fusion processing on the first feature information and the second feature information to obtain multimodal joint feature information, and send the processing result information corresponding to the multimodal joint feature information to the smart pen.

[0110] In step S11, first modal data and second modal data are generated. Specifically, the first modal data and second modal data are different types of external information collected by the smart pen through its information acquisition module. For example, the first modal data can be image data of the writing medium surface (including visual content such as the user's drawings and written text) collected by the smart pen's visual acquisition module (such as a camera), and the second modal data can be user voice signal data (including voice content such as user's instructions, questions, or descriptions) collected by the smart pen's acoustic acquisition module (such as a microphone).

[0111] In step S21, the first modal data and the second modal data are sent to the server. Specifically, the smart pen establishes a wireless communication connection with the server through its communication module (such as a Wi-Fi module, Bluetooth module, or mobile communication module), and packages and sends the timestamped first modal data and second modal data generated in step S11 to the server. The server is a cloud computing node in the data processing system, which is equipped with program modules for executing multimodal data processing methods. It has abundant computing resources and can undertake computationally intensive processing tasks such as image recognition, speech recognition, semantic understanding, and multimodal fusion. After sending the data, the smart pen completes its core responsibility in this interaction—collection and transmission—without needing to perform any computationally intensive processing such as recognition, analysis, or understanding of the data.

[0112] Through the coordination of steps S11 to S21 above, the smart pen only performs data acquisition, labeling, and transmission operations. All data processing tasks are undertaken by the server; that is, the smart pen is responsible for "generating and sending," while the server is responsible for "receiving, processing, and transmitting back." During data generation, the smart pen only adds timestamps to establish temporal relationships between data points, without performing any semantic analysis, recognition, or understanding of the data content itself, thus ensuring extremely low computational load and power consumption at the pen end.

[0113] It should be noted that the multimodal data processing method executed by the server has been described in detail in steps S10 to S42 above, including receiving first modal data and second modal data, extracting feature information, performing fusion processing, and returning processing result information, etc., which will not be explained again here. The smart pen can simply wait to receive the processing result information returned by the server after sending the data.

[0114] In one embodiment of the present invention, please refer to Figure 8 Step 11 includes: S111: Obtain the raw data of the first mode and the raw data of the second mode; S112: Add timestamps to the original data of the first modality and the original data of the second modality respectively to generate the first modality data and the second modality data.

[0115] In step 111, the first modality raw data and the second modality raw data are acquired. Specifically, the first modality raw data is the raw image signal directly acquired by the smart pen's visual acquisition module (such as a camera). This signal contains the raw pixel information of visual content such as drawings and written text by the user, without any image recognition or feature extraction processing. The second modality raw data is the raw speech signal directly acquired by the smart pen's acoustic acquisition module (such as a microphone). This signal contains the raw audio waveform information of speech content such as instructions, questions, or descriptions issued by the user, without any speech recognition or semantic understanding processing. The smart pen's microprocessor module acquires the above two types of raw data from the visual acquisition module and the acoustic acquisition module respectively through its electrical connection path with the information acquisition module. In step S112 of the present invention, timestamps are added to the first modality raw data and the second modality raw data respectively to generate the first modality data and the second modality data. Specifically, the timestamp is a precise time tag generated by the hardware clock unit in the microprocessor module of the smart pen while acquiring the raw data. It records the precise time (e.g., a time value with millisecond precision) at which each frame of image data and each segment of voice data is acquired.

[0116] The microprocessor module adds the generated timestamps to the corresponding first modality raw data and second modality raw data respectively. For example, it embeds the timestamp information into the metadata field of the image frame or associates the timestamp information with the audio frame for storage, thereby generating first modality data and second modality data with timestamps and a unified time reference.

[0117] By adding timestamps as described above, a precise time correspondence is established between the two types of data. That is, image frames and audio frames collected at the same or similar times can be identified as related data collected at the same time in subsequent server-side processing, providing a time reference for the server to align and associate multimodal data based on time information.

[0118] In one application scenario, a user uses a smart pen to draw a shaded geometric shape on paper for 8 seconds, while simultaneously saying "Calculate the area of ​​the shaded part of this shape" for 4 seconds. First, the smart pen's camera captures raw image data at a preset frame rate throughout the drawing process, and the microphone captures raw audio data while the user speaks. The microprocessor acquires both streams of raw data. Then, the microprocessor's internal hardware clock unit records the acquisition time for each frame of image data (e.g., 0.00 second, 0.03 second, 0.06 second... up to 8.00 seconds) and the acquisition time for each segment of audio data (e.g., each sampling point between 3.00 seconds and 7.00 seconds). These timestamps are added to the corresponding raw data, generating timestamped image and audio data.

[0119] Therefore, when the server subsequently receives these two types of data, it can parse the timestamps to determine that the image data was collected from the 0th to the 8th second and the voice data was collected from the 3rd to the 7th second. This allows it to determine the overlap between the two data, which is the 3rd to the 7th second. Within this overlap, the voice pronouns are accurately associated with the corresponding visual features, thus avoiding incorrect association of voice data with irrelevant data from the beginning or end of the drawing process due to time misalignment.

[0120] The present invention also proposes a server, the server comprising: The communication module is connected to the smart pen and is configured to receive first modal data and second modal data sent by the smart pen. The feature extraction module is configured to extract first feature information from the first modality data and second feature information from the second modality data; The fusion processing module is configured to perform fusion processing on the first feature information and the second feature information to obtain multimodal joint feature information; The communication module is further configured to send the processing result information corresponding to the multimodal joint feature information to the smart pen.

[0121] Specifically, the server is a computing device deployed in the cloud. Together with the smart pen, it constitutes a data processing system, which is used to undertake computationally intensive processing tasks such as image recognition, speech recognition, semantic understanding, and multimodal fusion, and then sends the processing results back to the smart pen.

[0122] The communication module is communicatively connected to the smart pen and configured to receive first modal data and second modal data sent by the smart pen. The first modal data may be image data acquired by the visual acquisition module of the smart pen, and the second modal data may be voice data acquired by the acoustic acquisition module of the smart pen. The communication module is also used to send the processing result information generated by the fusion processing module to the smart pen.

[0123] The feature extraction module includes a first feature extraction network and a second feature extraction network. The first feature extraction network is a pre-trained deep learning model. The first feature extraction network is used to encode the features of the input first modality data (such as image data) and output first feature information. The first feature information represents the key visual semantic elements in the image, including the basic shape of the graphic, the position of the geometric boundary, the spatial relationship between different regions, etc.

[0124] The second feature extraction network is a pre-trained deep learning model. The second feature extraction network is used to encode the features of the input second modality data (such as speech signals) and output second feature information. The second feature information represents the key semantic information in the speech, including the user's expressed instruction intention, key entities, referential words, etc.

[0125] It should be noted that the specific network structure and training method used in the feature extraction module are well-known technologies in the field, and the present invention does not limit its specific implementation.

[0126] The fusion processing module is configured to fuse the first feature information and the second feature information to obtain multimodal joint feature information. Specifically, the fusion processing module includes a time alignment unit, a correlation calculation unit, a confidence evaluation unit, a weight determination unit, and a weighted fusion unit.

[0127] The time alignment unit is used to obtain the first timestamp information corresponding to the first modal data and the second timestamp information corresponding to the second modal data, and to determine the time overlap interval between the first feature information and the second feature information based on the first timestamp information and the second timestamp information. Further, the time alignment unit extracts a first feature subset corresponding to the overlap interval from the first feature information and extracts a second feature subset corresponding to the overlap interval from the second feature information based on the overlap interval.

[0128] The correlation calculation unit is used to calculate the similarity between the first feature subset and the second feature subset to obtain a similarity matrix. Based on the similarity matrix, the correlation calculation unit determines the correlation information between the first feature subset and the second feature subset.

[0129] The confidence assessment unit is used to assess the first confidence information of the first feature subset and the second confidence information of the second feature subset, wherein the confidence information characterizes the quality reliability of each modality data itself.

[0130] The weight determination unit is used to determine the fusion weights of the first feature subset and the second feature subset based on the correlation information, the first confidence information, and the second confidence information.

[0131] The weighted fusion unit is used to perform weighted fusion of the first feature subset and the second feature subset according to the fusion weight to generate the multimodal joint feature information.

[0132] The above-described division of modules is merely an exemplary implementation of the present invention. In actual deployment, the above-described multiple modules can be integrated into the same processor, with the processor uniformly executing the functions corresponding to each module. Of course, each function can also be completed collaboratively by multiple processors.

[0133] This invention does not limit the physical implementation of each module. The specific processing steps performed by each functional module of the server have been described in detail in the foregoing method embodiments, and will not be repeated here.

[0134] Since the server proposed in this invention adopts all the technical solutions of the above-described method embodiments, it has at least all the beneficial effects brought about by the above embodiments, which will not be repeated here.

[0135] This invention also proposes a smart pen, which includes: The information acquisition module is configured to acquire raw data from the first mode and raw data from the second mode. The processing module, electrically connected to the information acquisition module, is configured to add timestamps to the first modality raw data and the second modality raw data respectively, to generate timestamped first modality data and timestamped second modality data; and The communication module is electrically connected to the processing module. The communication module is configured to send first modal data with timestamps and second modal data with timestamps to the server, and to receive processing result information sent by the server.

[0136] Specifically, the smart pen is a handheld writing device that, together with the server, constitutes a data processing system. It is used to collect the user's writing image data and voice command data, add timestamps to the collected raw data and send it to the server, as well as receive the processing result information returned by the server and present it to the user.

[0137] The smart pen includes an information acquisition module, a processing module, and a communication module. The information acquisition module is configured to acquire raw data from both a first modality and a second modality. Specifically, the information acquisition module includes a visual acquisition unit (such as a camera) and an acoustic acquisition unit (such as a microphone). The visual acquisition unit is used to acquire raw image signals from the writing medium surface (including visual content such as drawings and written text by the user), and the acoustic acquisition unit is used to acquire raw voice signals emitted by the user (including voice content such as user commands, questions, or descriptions). The information acquisition module is electrically connected to the processing module, transmitting the acquired raw data from both the first and second modalities to the processing module.

[0138] The processing module is configured to add timestamps to the first modality raw data and the second modality raw data respectively to generate time-stamped first modality data and time-stamped second modality data.

[0139] Specifically, the processing module is the microprocessor of the smart pen, which integrates a hardware clock unit. This hardware clock unit generates precise timestamps (e.g., millisecond-level precision) for each frame of image data and each segment of audio data while the information acquisition module collects raw data. The processing module adds the generated timestamps to the corresponding raw data, for example, by embedding the timestamp information into the metadata field of the image frame or by associating the timestamp information with the audio frame, thereby generating first-modal data and second-modal data with a unified time reference. Through the above-mentioned operation of adding timestamps, a time correspondence is established between the two types of modal data, providing a time reference for the server to subsequently align and associate multimodal data based on time information. The communication module is electrically connected to the processing module. The communication module is configured to send first modal data with timestamps and second modal data with timestamps to the server, and to receive processing result information sent by the server.

[0140] Specifically, the communication module can be a low-power wireless communication module (such as a Wi-Fi module, Bluetooth module, or mobile communication module), which establishes a wireless communication connection with the server, packages the first and second modal data with timestamps generated by the processing module and sends them to the server, and receives the processing result information returned by the server after the server completes the data processing.

[0141] Since the smart pen proposed in this invention adopts all the technical solutions of the above-described method embodiments, it has at least all the beneficial effects brought about by the above embodiments, which will not be repeated here.

[0142] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural transformations made using the contents of the specification and drawings of the present invention under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the scope of patent protection of the present invention.

Claims

1. A multimodal data processing method, applied to a server in a data processing system, characterized in that, The data processing system also includes a smart pen, and the method includes: Receive first-modal data and second-modal data sent by the smart pen; Extract the first feature information from the first modality data, and extract the second feature information from the second modality data; The first feature information and the second feature information are fused to obtain multimodal joint feature information; The processing result information corresponding to the multimodal joint feature information is sent to the smart pen.

2. The multimodal data processing method as described in claim 1, characterized in that, The steps of extracting first feature information from the first modality data and extracting second feature information from the second modality data include: The first modality data is input into the first feature extraction network, and the first feature information of the first modality data is extracted through the first feature extraction network; The second modality data is input into the second feature extraction network, and the second feature information of the second modality data is extracted through the second feature extraction network.

3. The multimodal data processing method as described in claim 1, characterized in that, The step of fusing the first feature information and the second feature information includes: Obtain the first timestamp information corresponding to the first modal data and the second timestamp information corresponding to the second modal data; Based on the first timestamp information and the second timestamp information, determine the time overlap interval between the first feature information and the second feature information; Based on the overlapping interval, a first feature subset corresponding to the overlapping interval is extracted from the first feature information, and a second feature subset corresponding to the overlapping interval is extracted from the second feature information.

4. The multimodal data processing method as described in claim 3, characterized in that, The step of fusing the first feature information and the second feature information further includes: Calculate the similarity between the first feature subset and the second feature subset to obtain a similarity matrix; Based on the similarity matrix, the correlation information between the first feature subset and the second feature subset is determined.

5. The multimodal data processing method as described in claim 4, characterized in that, The step of fusing the first feature information and the second feature information further includes: Evaluate the first confidence information of the first feature subset and the second confidence information of the second feature subset; Based on the correlation information, the first confidence information, and the second confidence information, determine the fusion weights of the first feature subset and the second feature subset respectively; Based on the fusion weights, the first feature subset and the second feature subset are weighted and fused to obtain the multimodal joint feature information.

6. The multimodal data processing method as described in claim 1, characterized in that, The step of sending the processing result information corresponding to the multimodal joint feature information to the smart pen includes: The multimodal joint feature information is input into a preset semantic model to obtain the processing result information output by the preset semantic model; The processing result information is sent to the smart pen.

7. A multimodal data processing method, applied to a smart pen in a data processing system, characterized in that, The data processing system further includes a server, and the method includes: Generate first modal data and second modal data; The first modal data and the second modal data are sent to the server, so that the server extracts the first feature information of the first modal data and the second feature information of the second modal data, performs fusion processing on the first feature information and the second feature information to obtain multimodal joint feature information, and sends the processing result information corresponding to the multimodal joint feature information to the smart pen.

8. The multimodal data processing method as described in claim 7, characterized in that, The steps for generating the first modal data and the second modal data include: Obtain the raw data of the first mode and the raw data of the second mode; Timestamps are added to the original data of the first modality and the original data of the second modality respectively to generate the first modality data and the second modality data.

9. A server, characterized in that, The server includes: The communication module, which communicates with the smart pen, is configured to receive first-mode data and second-mode data sent by the smart pen; The feature extraction module is configured to extract first feature information from the first modality data and second feature information from the second modality data; The fusion processing module is configured to perform fusion processing on the first feature information and the second feature information to obtain multimodal joint feature information; The communication module is further configured to send the processing result information corresponding to the multimodal joint feature information to the smart pen.

10. A smart pen, characterized in that, The smart pen includes: An information acquisition module is configured to acquire raw data from a first modality and raw data from a second modality. A processing module, electrically connected to the information acquisition module, is configured to add timestamps to the first modality raw data and the second modality raw data, respectively, to generate timestamped first modality data and timestamped second modality data; and A communication module is electrically connected to the processing module. The communication module is configured to send the timestamped first modal data and the timestamped second modal data to the server, and to receive the processing result information sent by the server.

Images