Distributed screen multi-modal input fusion method and system based on open source honkong

Abstract

This invention discloses a distributed screen multimodal input fusion method and system based on the open-source HarmonyOS. The method includes: acquiring raw voice, touch, and gesture data from multiple acquisition screens via distributed bus technology and performing preprocessing operations to generate voice, touch, and gesture input data; recognizing the voice, touch, and gesture input data separately using the HarmonyOS recognition engine and aggregating them to generate multimodal input data; fusing the multimodal input data using an input fusion algorithm to generate input fusion commands; distributing the input fusion commands to target screens based on an intelligent scheduling and synchronization mechanism; and collecting user feedback on the input fusion effect on the target screens in real time based on a user feedback mechanism, and optimizing the recognition engine and input fusion algorithm based on the feedback results. This invention enables seamless switching and collaborative operation of multiple input methods across multiple screens, improving application interaction efficiency and user experience.

Description

Technical Field

[0001] This invention relates to the field of information technology, specifically to a distributed screen multimodal input fusion method and system based on the open-source HarmonyOS. Background Technology

[0002] With the rapid development of information technology, distributed systems and multi-screen interaction are becoming increasingly common in daily life and work, especially in fields such as smart offices, remote education, and entertainment, where multi-screen collaborative work is gradually gaining attention. In recent years, the open-source HarmonyOS system, with its distributed architecture and powerful cross-device collaboration capabilities, has provided a new platform for realizing multimodal input fusion.

[0003] In existing technologies, traditional multi-screen interaction systems are often limited to a single input method, such as only supporting touch or keyboard input. This greatly reduces the system's interaction efficiency, fails to provide users with a smooth and natural interactive experience, and has poor flexibility, making it unsuitable for different application scenarios and device requirements.

[0004] Therefore, how to achieve seamless switching and collaborative work of multiple input methods across multiple screens, and improve the interaction efficiency and user experience of applications, has become an urgent problem to be solved. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a distributed screen multimodal input fusion method and system based on the open-source HarmonyOS. The specific technical solution includes:

[0006] A distributed screen multimodal input fusion method based on open-source HarmonyOS, the method comprising:

[0007] S1, based on the open-source HarmonyOS system, acquires raw voice, touch and gesture data from multiple capture screens through distributed bus technology;

[0008] S2, perform preprocessing operations on the raw voice, touch and gesture data to generate corresponding voice, touch and gesture input data;

[0009] S3, the voice, touch and gesture input data are recognized by the recognition engine of the open source HarmonyOS system respectively, and the corresponding multimodal input data is generated by summarizing them;

[0010] S4, perform weighted fusion processing on the multimodal input data based on the input fusion algorithm to generate corresponding input fusion instructions;

[0011] S5, based on the intelligent scheduling and synchronization mechanism, the input fusion command is distributed to the target screen;

[0012] S6. Based on the user feedback mechanism, the user terminal's feedback results on the input fusion effect of the target screen are collected in real time, and the recognition engine and input fusion algorithm are optimized according to the feedback results.

[0013] A distributed screen multimodal input fusion system based on open-source HarmonyOS, used to execute the aforementioned distributed screen multimodal input fusion method based on open-source HarmonyOS, includes:

[0014] The acquisition module is used to acquire raw voice, touch and gesture data from multiple capture screens based on the open-source HarmonyOS system and through distributed bus technology.

[0015] The preprocessing module is used to perform preprocessing operations on the raw voice, touch and gesture data to generate corresponding voice, touch and gesture input data;

[0016] The recognition module is used to recognize the voice, touch and gesture input data respectively through the recognition engine of the open source HarmonyOS system, and summarize and generate corresponding multimodal input data;

[0017] The fusion module is used to perform weighted fusion processing on the multimodal input data based on the input fusion algorithm to generate corresponding input fusion instructions;

[0018] The distribution module is used to distribute the input fusion instructions to the target screen based on an intelligent scheduling and synchronization mechanism;

[0019] The feedback module is used to collect feedback results from the user terminal on the input fusion effect of the target screen in real time based on the user feedback mechanism, and optimize the recognition engine and input fusion algorithm according to the feedback results.

[0020] A computing device includes: at least one processor and a memory storing program instructions; when the program instructions are read and executed by the processor, the computing device performs the method.

[0021] A readable storage medium storing program instructions that, when read and executed by a computing device, cause the computing device to perform the method.

[0022] The beneficial effects of this invention are as follows: This invention proposes a distributed screen multimodal input fusion method and system based on the open-source HarmonyOS. Based on the open-source HarmonyOS system, it acquires raw voice, touch, and gesture data from multiple acquisition screens through distributed bus technology; preprocesses the raw voice, touch, and gesture data to generate corresponding voice, touch, and gesture input data; recognizes the voice, touch, and gesture input data separately through the recognition engine of the open-source HarmonyOS system, and aggregates them to generate corresponding multimodal input data; fuses the multimodal input data based on the input fusion algorithm to generate corresponding input fusion instructions; distributes the input fusion instructions to the target screen based on an intelligent scheduling and synchronization mechanism; and collects feedback results from the user terminal on the input fusion effect of the target screen in real time based on a user feedback mechanism, and optimizes the recognition engine and input fusion algorithm based on the feedback results. Thus, through the recognition engine and input fusion algorithm of the open-source HarmonyOS system, seamless switching and collaborative work of multiple input methods across multiple screens can be achieved, improving the interaction efficiency and user experience of the application. Attached Figure Description

[0023] Figure 1 A flowchart illustrating the distributed screen multimodal input fusion method based on open-source HarmonyOS provided in an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of the structure of a distributed screen multimodal input fusion system based on open-source HarmonyOS provided in an embodiment of the present invention;

[0025] Figure 3 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0026] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0027] See Figure 1 This is a flowchart illustrating the distributed screen multimodal input fusion method based on open-source HarmonyOS provided in this embodiment of the invention. Figure 1The execution entity of the method shown can be a software and / or hardware device. The execution entity of this application can include, but is not limited to, at least one of the following: user equipment, network equipment, etc. User equipment can include, but is not limited to, computers, smartphones, personal digital assistants (PDAs), and the aforementioned electronic devices. Network equipment can include, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of computers or network servers. Cloud computing is a type of distributed computing, consisting of a super virtual computer composed of a group of loosely coupled computers. This embodiment does not impose any limitations on this. Figure 1 As shown, the method of the present invention includes steps S1 to S6, as detailed below:

[0028] S1, based on the open-source HarmonyOS system, acquires raw voice, touch and gesture data from multiple capture screens through distributed bus technology;

[0029] Understandably, based on the open-source HarmonyOS system, raw data from multiple capture screens, including voice, touch, and gesture data, can be efficiently acquired and processed. Through distributed bus technology, flexible communication paths can be built between different physical or logical components to ensure efficient data transmission and synchronization.

[0030] Specifically, distributed bus technology can connect various acquisition screens to the central processing unit, capturing and transmitting various types of input data from different screens in real time and accurately, including raw data such as voice signals, touch events, and gestures.

[0031] S2, perform preprocessing operations on the raw voice, touch and gesture data to generate corresponding voice, touch and gesture input data;

[0032] For raw voice data, noise reduction, volume adjustment, and speech recognition preprocessing can be performed to ensure the accuracy of subsequent recognition. For raw touch data, position calibration, event classification, and touch point tracking are required to achieve more precise touch control. For gesture data, feature extraction, pattern recognition, and trajectory optimization are needed to improve the accuracy of gesture recognition. Through preprocessing, standardized and normalized voice, touch, and gesture input data can be generated, facilitating subsequent data recognition and processing.

[0033] S3, the voice, touch and gesture input data are recognized by the recognition engine of the open source HarmonyOS system respectively, and the corresponding multimodal input data is generated by summarizing them;

[0034] The open-source HarmonyOS recognition engine integrates speech recognition, touch recognition, and gesture recognition technologies, enabling it to recognize and analyze pre-processed speech, touch, and gesture input data respectively.

[0035] Specifically, the speech recognition engine converts speech signals into text information; the touch recognition engine parses touch events and extracts useful operation commands; and the gesture recognition engine recognizes the user's hand gestures. Finally, the recognition results can be aggregated to generate a dataset containing multiple modal information, i.e., multimodal input data.

[0036] S4, perform weighted fusion processing on the multimodal input data based on the input fusion algorithm to generate corresponding input fusion instructions;

[0037] To further enhance the system's flexibility and intelligence, an input fusion algorithm can be introduced. Based on preset fusion rules and strategies, this algorithm effectively integrates and processes multimodal input data, thereby generating accurate and comprehensive input fusion commands. These commands can reflect the user's true intentions and operational needs, providing a reliable basis for subsequent operations.

[0038] S5, based on the intelligent scheduling and synchronization mechanism, the input fusion command is distributed to the target screen;

[0039] Based on the intelligent scheduling and synchronization mechanism, input fusion commands can be efficiently distributed to the target screen. This mechanism intelligently determines the distribution time and method of input fusion commands based on factors such as the current system operating status, task priority, and target screen availability, ensuring that commands reach the target screen and execute corresponding operations in the shortest possible time.

[0040] S6. Based on the user feedback mechanism, the user terminal's feedback results on the input fusion effect of the target screen are collected in real time, and the recognition engine and input fusion algorithm are optimized according to the feedback results.

[0041] The user feedback mechanism allows for real-time collection of user feedback on the input fusion effect of the target screen, including key indicators such as user satisfaction, ease of use, and response time. Analyzing and processing this feedback data enables timely identification of problems and shortcomings in the current system's recognition engine, input fusion algorithm, and command distribution. Based on this feedback, targeted optimizations and improvements can be made to the system. Furthermore, this cyclical feedback and optimization mechanism ensures continuous system self-improvement, providing users with a higher quality and more efficient multimodal interactive experience.

[0042] S2, preprocessing the raw voice, touch, and gesture data to generate corresponding voice, touch, and gesture input data, specifically including:

[0043] S2-1, Remove abnormal data from the raw voice, touch and gesture data;

[0044] S2-2, unifies the data format of raw data for voice, touch and gesture;

[0045] S2-3 maps the raw voice, touch, and gesture data to the same numerical range to generate voice, touch, and gesture input data.

[0046] It's important to note that the first step is to remove outliers from the raw voice, touch, and gesture data. Outliers, or noisy data, can stem from sensor malfunctions, environmental interference, and user errors, significantly impacting the accuracy and reliability of the data. Therefore, statistical methods or machine learning algorithms can be used to clean the raw data, effectively identifying and eliminating these outliers to ensure the accuracy of subsequent analysis.

[0047] Secondly, since data collected by different sensors or devices may use different encoding methods, sampling rates, or data structures, these raw data can be converted into a unified format standard using data format conversion tools or specialized scripts. This includes adjusting the sampling rate of audio files, normalizing touch coordinates, and serializing and storing gesture trajectory data, thereby ensuring seamless integration of all data in subsequent processing stages.

[0048] Finally, the raw voice, touch, and gesture data can be mapped to the same numerical range to generate standardized input data. This step typically involves data normalization or standardization, which transforms the data to a specific numerical range, such as [0, 1] or [-1, 1], based on its distribution characteristics. This eliminates the influence of different feature units and improves the training efficiency and prediction performance of the model.

[0049] Specifically, for raw voice data, Mel-frequency cepstral coefficients (MFCC) can be used for extraction and normalization; for raw touch data, scaling can be performed according to the screen size; and for raw gesture data, smoothing of motion trajectories and scaling transformation can be considered, thereby ensuring that voice, touch, and gesture input data have a consistent numerical range, better reflecting the user's true intentions and providing support for intelligent decision-making in human-computer interaction systems.

[0050] S3, using the recognition engine of the open-source HarmonyOS system, recognizes the voice, touch, and gesture input data respectively, and aggregates them to generate corresponding multimodal input data, specifically including:

[0051] S3-1, Receive and recognize the voice input data through the voice recognition engine, and convert the voice input data into corresponding voice recognition data, including the following operations:

[0052] S3-1-1, the speech input data to be recognized is segmented into multiple frames and the features of each frame are extracted;

[0053] S3-1-2, After obtaining the feature sequences of all frames, the feature sequences and speech input data are modeled to obtain the correspondence between the feature sequences and speech input data, and the corresponding hidden state sequence, i.e. the speech recognition data, is determined according to the state transition probability, observation probability and initial state probability.

[0054] The expression for the state transition probability is:

[0055] ,

[0056] In the formula, Indicates from hidden state Move to hidden state The state transition probability; Indicates probability symbols; Indicates time Hidden state variables; Indicates time Hidden state variables; Indicates a hidden state;

[0057] The expression for the probability of observation is:

[0058] ,

[0059] In the formula, Indicates that it is in a hidden state. Generate observations The probability of observation; Indicates probability symbols; Indicates time The observed variables; Indicates time Hidden state variables; Represents the observed value; Indicates a hidden state;

[0060] The expression for the initial state probability is:

[0061] ,

[0062] In the formula, This indicates that the device is initially in a hidden state. The initial state probability; Indicates probability symbols; Represents the hidden state variables at the initial moment; Indicates a hidden state;

[0063] The hidden state sequence, i.e., speech recognition data, is determined based on the state transition probability, observation probability, and initial state probability.

[0064] S3-2, receiving and recognizing touch input data through the touch recognition engine, and calculating touch recognition data according to the objective function corresponding to the touch input data, including the following operations:

[0065] The objective function of the touch recognition engine is expressed as follows:

[0066] ,

[0067] In the formula, Represent the objective function; Indicates the total length of the sequence; Indicates the current position index in the sequence; Indicates the size of the window at the top and bottom positions; Indicates the current position index The touch point; Indicates the current position index after Touch points with location index; Indicates probability symbols; Indicates the current position index Predicted location index The touch point; Σ represents the summation symbol; Represents the logarithmic sign;

[0068] The result of the objective function calculation is the touch recognition data.

[0069] S3-3 receives and recognizes gesture input data through the gesture recognition engine, and converts the gesture input data into corresponding gesture recognition data, including the following operations:

[0070] S3-3-1, The gesture recognition engine recognizes the gesture input data and generates the corresponding gesture input image. Feature extraction is performed on the gesture input image to obtain the corresponding gesture image features. The expression for the gesture image features is:

[0071] ,

[0072] In the formula, Represents gesture image features; Coordinates representing the features of the gesture image; Indicates a gesture input image; Represents the convolution kernel; Indicates the kernel size;

[0073] S3-3-2, Obtain gesture recognition data based on gesture image features;

[0074] S3-4, summarize the voice, touch and gesture recognition data to obtain multimodal input data.

[0075] S4, perform weighted fusion processing on the multimodal input data based on the input fusion algorithm to generate a corresponding input fusion instruction, specifically including:

[0076] The input fusion instruction is obtained by weighting and summing the multimodal input data:

[0077] ,

[0078] In the formula, Indicates the input fusion command; Indicates the total number of input methods; Indicates the first Priority of different input methods hour, This indicates the first input method, namely voice input. hour, This indicates the second input method, namely touch input. hour, This indicates the third input method, namely gesture input; Indicates the first The output data after the input method is processed by the input fusion algorithm; Indicates the first Recognition data corresponding to each input method; Indicates the first The fusion parameters corresponding to the various input methods; Indicates the current screen status; Indicates the current task requirements;

[0079] In summary, the processed output data is weighted and fused according to the priority of each input method, and the input fusion instruction is obtained by summing.

[0080] Figure 2 This diagram illustrates a schematic structure of a distributed screen multimodal input fusion system based on the open-source HarmonyOS, provided in an embodiment of the present invention. Figure 2 As shown, the system includes:

[0081] The acquisition module is used to acquire raw voice, touch and gesture data from multiple capture screens based on the open-source HarmonyOS system and through distributed bus technology.

[0082] The preprocessing module is used to perform preprocessing operations on the raw voice, touch and gesture data to generate corresponding voice, touch and gesture input data;

[0083] The recognition module is used to recognize the voice, touch and gesture input data respectively through the recognition engine of the open source HarmonyOS system, and summarize and generate corresponding multimodal input data;

[0084] The fusion module is used to perform weighted fusion processing on the multimodal input data based on the input fusion algorithm to generate corresponding input fusion instructions;

[0085] The distribution module is used to distribute the input fusion instructions to the target screen based on an intelligent scheduling and synchronization mechanism;

[0086] The feedback module is used to collect feedback results from the user terminal on the input fusion effect of the target screen in real time based on the user feedback mechanism, and optimize the recognition engine and input fusion algorithm according to the feedback results.

[0087] Figure 2 The system of the illustrated embodiment can be used to perform corresponding operations. Figure 1 The steps in the method embodiments shown are implemented in a similar manner and have similar technical effects, and will not be repeated here.

[0088] Figure 3 This diagram illustrates the hardware structure of an electronic device 30 according to an embodiment of the present invention. The electronic device 30 includes: a processor 31, a memory 32, and a computer program; wherein:

[0089] The memory 32 is used to store the computer program, and the memory may be flash memory. The computer program is, for example, an application program or functional module that implements the above method.

[0090] The processor 31 is used to execute the computer program stored in the memory 32 to implement the various steps performed by the device in the above method. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0091] Alternatively, the memory 32 can be either standalone or integrated with the processor 31.

[0092] When the memory 32 is a device independent of the processor 31, the device may further include:

[0093] Bus 33 is used to connect the memory 32 and the processor 31.

[0094] The present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, is used to implement the methods provided in the various embodiments described above.

[0095] The readable storage medium can be a computer storage medium or a communication medium. A communication medium includes any medium that facilitates the transfer of computer programs from one location to another. A computer storage medium can be any available medium accessible to a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application-Specific Integrated Circuit (ASIC). Alternatively, the ASIC can be located in a user equipment. Of course, the processor and the readable storage medium can also exist as discrete components in a communication device. The readable storage medium can be a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0096] The present invention also provides a program product including executable instructions stored in a readable storage medium. At least one processor of the device can read the executable instructions from the readable storage medium, and the at least one processor executes the executable instructions to cause the device to implement the methods provided in the various embodiments described above.

[0097] In the embodiments of the above-described device, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0098] Through the above embodiments, this invention, based on the open-source HarmonyOS distributed screen multimodal input fusion method and system, acquires raw voice, touch, and gesture data from multiple acquisition screens via distributed bus technology. It preprocesses the raw voice, touch, and gesture data to generate corresponding voice, touch, and gesture input data. The HarmonyOS recognition engine separately recognizes the voice, touch, and gesture input data, summarizing them to generate corresponding multimodal input data. An input fusion algorithm fuses the multimodal input data to generate corresponding input fusion instructions. An intelligent scheduling and synchronization mechanism distributes the input fusion instructions to the target screen. A user feedback mechanism collects real-time feedback from users on the input fusion effect on the target screen, and optimizes the recognition engine and input fusion algorithm based on the feedback. Thus, through the HarmonyOS recognition engine and input fusion algorithm, seamless switching and collaborative work of multiple input methods across multiple screens can be achieved, improving application interaction efficiency and user experience.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed screen multimodal input fusion method based on open-source HarmonyOS, characterized in that, The method includes: S1, based on the open-source HarmonyOS system, acquires raw voice, touch and gesture data from multiple capture screens through distributed bus technology; S2, perform preprocessing operations on the raw voice, touch and gesture data to generate corresponding voice, touch and gesture input data; S3, the voice, touch and gesture input data are recognized by the recognition engine of the open source HarmonyOS system respectively, and the corresponding multimodal input data is generated by summarizing them; S4, perform weighted fusion processing on the multimodal input data based on the input fusion algorithm to generate corresponding input fusion instructions; S5, based on the intelligent scheduling and synchronization mechanism, the input fusion command is distributed to the target screen; S6. Based on the user feedback mechanism, the user terminal's feedback results on the input fusion effect of the target screen are collected in real time, and the recognition engine and input fusion algorithm are optimized according to the feedback results.

2. The distributed screen multimodal input fusion method based on open-source HarmonyOS according to claim 1, characterized in that, S2 include: S2-1, Remove abnormal data from the original voice, touch, and gesture data; S2-2, unify the data format of the raw voice, touch and gesture data; S2-3, map the raw voice, touch and gesture data to the same numerical range to generate the voice, touch and gesture input data.

3. The distributed screen multimodal input fusion method based on open-source HarmonyOS according to claim 1, characterized in that, S3 include: S3-1, The voice input data is received and recognized by the voice recognition engine, and the voice input data is converted into corresponding voice recognition data; S3-2, The touch input data is received and recognized by the touch recognition engine, and the touch recognition data is calculated according to the objective function corresponding to the touch input data; S3-3, The gesture recognition engine receives and recognizes the gesture input data, and converts the gesture input data into corresponding gesture recognition data; S3-4, Summarize the voice, touch and gesture recognition data to obtain the multimodal input data.

4. The distributed screen multimodal input fusion method based on open-source HarmonyOS according to claim 3, characterized in that, S3-1 includes: S3-1-1, The voice input data that needs to be recognized is segmented into multiple frames, and the features of each frame are extracted; S3-1-2, After obtaining the feature sequences of all frames, the feature sequences and speech input data are modeled to obtain the correspondence between the feature sequences and speech input data, and the corresponding hidden state sequence, i.e. the speech recognition data, is determined according to the state transition probability, observation probability and initial state probability. The expression for the state transition probability is: ， In the formula, Indicates from hidden state Move to hidden state The state transition probability; Indicates probability symbols; Indicates time Hidden state variables; Indicates time Hidden state variables; Indicates a hidden state; The expression for the observation probability is: ， In the formula, Indicates that it is in a hidden state. Generate observations The probability of observation; Indicates probability symbols; Indicates time The observed variables; Indicates time Hidden state variables; Represents the observed value; Indicates a hidden state; The expression for the initial state probability is: ， In the formula, This indicates that the device is initially in a hidden state. The initial state probability; Indicates probability symbols; Represents the hidden state variables at the initial moment; Indicates a hidden state.

5. The distributed screen multimodal input fusion method based on open-source HarmonyOS according to claim 4, characterized in that, S3-2 includes: The expression for the objective function corresponding to the touch input data is: ， In the formula, Represent the objective function; Indicates the total length of the sequence; Indicates the current position index in the sequence; Indicates the size of the window at the top and bottom positions; Indicates the current position index The touch point; Indicates the current position index after Touch points with location index; Indicates probability symbols; Indicates the current position index Predicted location index The touch point; Σ represents the summation symbol; Represents the logarithmic sign; The result of the objective function is the touch recognition data.

6. The distributed screen multimodal input fusion method based on open-source HarmonyOS according to claim 5, characterized in that, S3-3 includes: S3-3-1, The gesture recognition engine recognizes the gesture input data and generates a corresponding gesture input image. Feature extraction is performed on the gesture input image to obtain the corresponding gesture image features. The expression for the gesture image features is: ， In the formula, Represents gesture image features; Coordinates representing the features of the gesture image; Indicates a gesture input image; Represents the convolution kernel; Indicates the kernel size; S3-3-2, Obtain the gesture recognition data based on the gesture image features.

7. The distributed screen multimodal input fusion method based on open-source HarmonyOS according to claim 1, characterized in that, S4 include: The input fusion instruction is obtained by weighting and summing the multimodal input data: ， In the formula, Indicates the input fusion command; Indicates the total number of input methods; Indicates the first Priority of different input methods hour, This indicates the first input method, namely voice input. hour, This indicates the second input method, namely touch input. hour, This indicates the third input method, namely gesture input; Indicates the first The output data after the input method is processed by the input fusion algorithm; Indicates the first Recognition data corresponding to each input method; Indicates the first The fusion parameters corresponding to the various input methods; Indicates the current screen status; This indicates the current task requirements.

8. A distributed screen multimodal input fusion system based on open-source HarmonyOS, used to execute the distributed screen multimodal input fusion method based on open-source HarmonyOS according to any one of claims 1-7, characterized in that, include: The acquisition module is used to acquire raw voice, touch and gesture data from multiple capture screens based on the open-source HarmonyOS system and through distributed bus technology. The preprocessing module is used to perform preprocessing operations on the raw voice, touch and gesture data to generate corresponding voice, touch and gesture input data; The recognition module is used to recognize the voice, touch and gesture input data respectively through the recognition engine of the open source HarmonyOS system, and summarize and generate corresponding multimodal input data; The fusion module is used to perform weighted fusion processing on the multimodal input data based on the input fusion algorithm to generate corresponding input fusion instructions; The distribution module is used to distribute the input fusion instructions to the target screen based on an intelligent scheduling and synchronization mechanism; The feedback module is used to collect feedback results from the user terminal on the input fusion effect of the target screen in real time based on the user feedback mechanism, and optimize the recognition engine and input fusion algorithm according to the feedback results.

9. A computing device, characterized in that, include: At least one processor and a memory storing program instructions; When the program instructions are read and executed by the processor, the computing device performs the method as described in any one of claims 1-7.

10. A readable storage medium storing program instructions, characterized in that, When the program instructions are read and executed by the computing device, the computing device performs the method as described in any one of claims 1-7.

Images