A multi-modal data processing method and apparatus
By constructing a multimodal data processing method, the target model is used to perform multimodal data processing on the data acquisition components and running application data of electronic devices. This solves the problem of inaccurate determination of the usage scenarios of electronic devices, achieves more efficient and accurate scene recognition, and improves the intelligent experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- LENOVO (BEIJING) LTD
- Filing Date
- 2026-01-30
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, the use cases of electronic devices are determined by a single judgment rule with a limited scope of application, which leads to inaccurate determination of the use cases by intelligent models.
By constructing a multimodal data processing method, the target model is used to perform multimodal data processing on the acquisition components and operational application data of electronic devices. Feature extraction and splicing are performed using processing units corresponding to different types of environmental data, and inference is performed by combining deep neural networks to determine the application scenarios of electronic devices.
It improves the efficiency and accuracy of data processing, enables accurate determination of the application scenarios of electronic devices, and enhances the intelligent experience of electronic devices.
Smart Images
Figure CN122153286A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing for intelligent models, and in particular to a multimodal data processing method and apparatus. Background Technology
[0002] To enhance the intelligent user experience when using computers and other electronic devices, these devices need to more accurately understand user intent in order to proactively execute operations, adjust configurations, and present the content the user needs. User intent is determined by the surrounding environment, the user's state, and the device's own operational status. However, currently, the rules for determining the usage scenarios of electronic devices are too simplistic and have limited applicability, leading to inaccurate determinations of usage scenarios by intelligent models. Summary of the Invention
[0003] The purpose of this application is to provide a multimodal data processing method, including:
[0004] The environmental data of the electronic device is determined, wherein the environmental data comes from at least one of the following: data collected by the electronic device's acquisition components and application data in operation; the environmental data includes at least a first type of environmental data and a second type of environmental data; The first type of environmental data and the second type of environmental data are input into the target model, which includes processing units corresponding to the different types of environmental data. Based on the first type of environmental data, a first processing unit of the target model is determined, and the first processing unit processes the first type of environmental data to obtain a first feature unit corresponding to the first type of environmental data. Based on the second type of environmental data, a second processing unit of the target model is determined, and the second processing unit processes the second type of environmental data to obtain the second feature unit corresponding to the second type of environmental data; The target model performs inference based on the first feature unit and the second feature unit to determine the application scenario of the electronic device.
[0005] Optionally, the determination of environmental data for the electronic device includes: According to the data formats of different types of environmental data, at least one type of data from the acquisition components of the electronic device and the running application data are processed to form a first type of environmental data and a second type of environmental data; the data formats of the different types of environmental data are determined by the component information and / or data attributes of the acquisition components.
[0006] Optionally, the first type of environmental data includes time-series data, and the first processing unit processes the first type of environmental data, including: The first processing unit determines a first time feature and a second time feature of the time series data, wherein the period of the first time feature is greater than the period of the second time feature. Based on the first and second time features of the time series data, determine the multi-channel time series data corresponding to the time series data; The multi-channel time-series data is quantized to form the first feature unit.
[0007] Optionally, the method further includes: Based on a preset time identifier, determine whether the environmental data has time data characteristics; If the environmental data has time data characteristics, the environmental data is identified as the time series data.
[0008] Optionally, the second type of environmental data includes image data, and the second processing unit processes the second type of environmental data, including: Using the image encoder in the second processing unit, the encoding of the image data is aligned with the encoding of the reference image. The alignment operation includes upsampling and / or downsampling the image data through the convolutional layer of the target model. Based on the image data after the alignment operation, a corresponding image data vector is generated; The image data vector is described with text to obtain the corresponding second feature unit.
[0009] Optionally, generating a corresponding image data vector based on the image data after the alignment operation includes: Based on the image data after the alignment operation and the output classification layer in the image encoder, determine multiple image type vectors with relatively high classification probabilities. Multiple image type vectors are fused together to form the image data vector.
[0010] Optionally, the environmental data includes at least a third type of environmental data, the third type of environmental data including text data, and the method further includes: The third processing unit of the target model processes the third type of environmental data, including: when there is no other modal data in the text data, the third processing unit performs vectorization processing on the text data to generate the corresponding third feature unit.
[0011] Optionally, the environmental data collected by the acquisition component may include a first type of environmental data, a second type of environmental data, or a third type of environmental data, and the application data running on the electronic device may include the third type of environmental data.
[0012] Optionally, the target model performs inference based on the first feature unit and the second feature unit, including: The target model concatenates at least two of the first feature unit, the second feature unit, and the third feature unit, and inputs the concatenation result into the inference unit of the target model to determine the application scenario of the electronic device.
[0013] This application also provides a multimodal data processing apparatus, including: A data acquisition component configured to determine environmental data of an electronic device, wherein the environmental data originates from at least one of the following: data acquired by the data acquisition component of the electronic device and running application data; the environmental data includes at least a first type of environmental data and a second type of environmental data. An operation module is configured to input first-type environmental data and second-type environmental data into a target model, wherein the target model includes processing units corresponding to different types of environmental data. The target model is configured to determine a first processing unit of the target model based on the first type of environmental data, and the first processing unit processes the first type of environmental data to obtain a first feature unit corresponding to the first type of environmental data. Based on the second type of environmental data, a second processing unit of the target model is determined, and the second processing unit processes the second type of environmental data to obtain a second feature unit corresponding to the second type of environmental data; Reasoning is performed based on the first feature unit and the second feature unit to determine the application scenario of the electronic device. Attached Figure Description
[0014] Figure 1 This is a flowchart of a multimodal data processing method according to an embodiment of this application; Figure 2 Examples of embodiments of this application Figure 1 A flowchart of one embodiment of step S300; Figure 3 This is a flowchart of one embodiment of the multimodal data processing method according to this application; Figure 4 Examples of embodiments of this application Figure 1 A flowchart of one embodiment of step S400; Figure 5 This is a flowchart of another specific embodiment of the multimodal data processing method according to the present application; Figure 6 This is a structural block diagram of a multimodal data processing apparatus according to an embodiment of this application. Detailed Implementation
[0015] Various embodiments and features of this application are described herein with reference to the accompanying drawings.
[0016] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this application will be apparent to those skilled in the art.
[0017] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.
[0018] These and other features of this application will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.
[0019] It should also be understood that although this application has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of this application.
[0020] The above and other aspects, features and advantages of this application will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.
[0021] Specific embodiments of this application are described thereafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this application, which can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the application. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as the basis and representative basis for the claims to teach those skilled in the art to use this application in a variety of substantially any suitable detailed structures.
[0022] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in other embodiments,” all of which may refer to one or more of the same or different embodiments according to this application.
[0023] This application provides a multimodal data processing method that can be applied to electronic devices with target models, such as user terminals, and can infer the application scenario of the electronic device based on environmental data collected by the electronic device.
[0024] The data processing method of this application will be described in detail below with reference to the accompanying drawings. Figure 1 This is a flowchart of a multimodal data processing method according to an embodiment of this application, such as... Figure 1 As shown and combined Figure 5 The method includes the following steps: S100, determine the environmental data of the electronic device, wherein the environmental data comes from at least one of the following: data collected by the electronic device's acquisition component and application data in operation; the environmental data includes at least a first type of environmental data and a second type of environmental data.
[0025] For example, an electronic device can be a terminal device, a service device, etc. Environmental data comes from at least one of the following: data collected by the electronic device's acquisition components and application data during operation. Specifically, the electronic device has acquisition components, which include various types of acquisition devices, each capable of collecting corresponding environmental data, such as image data, audio data, light signals, and motion data. On the other hand, the electronic device also has programs installed, including operating systems and applications, which generate corresponding application data during operation. This application data characterizes the program's operating status and its impact on the electronic device's usage scenario.
[0026] In this embodiment, the environmental data includes at least a first type of environmental data and a second type of environmental data. Of course, other types of environmental data may also be included, such as a third type, a fourth type, etc. These various types of environmental data form multimodal data, and each type of environmental data can represent one modality. The data format and content of different types of environmental data can differ.
[0027] In one embodiment of this application, after determining the environmental data, the environmental data can be preprocessed to adapt it to the target model so that the target model can process the environmental data, such as data format processing, noise removal, and data size adjustment. For example, the environmental data can be preprocessed separately by each acquisition component. Alternatively, the environmental data can be preprocessed separately by the operating system of the electronic device.
[0028] In one embodiment, preprocessing can transform environmental data into different types of input data, such as text data, time-series data, and image data. The data formats, content, and characteristics of these different input data can vary. Then, the processed input data of each type is fed into the target model, which will further process each type of input data accordingly.
[0029] In one embodiment, preprocessing includes adding identification information corresponding to the type of multiple types of environmental data, such as adding first identification information to environmental data of the first type and second identification information to environmental data of the second type, so that the target model can identify the identification information to determine the type of environmental data.
[0030] S200, input the first type of environmental data and the second type of environmental data into the target model, the target model including processing units corresponding to the different types of environmental data.
[0031] For example, the target model can be a model built on a deep neural network. For instance, the target model can be built on a large language model (LLM), which achieves deep understanding and efficient generation of natural language through training on massive amounts of text data and a complex neural network architecture.
[0032] In this embodiment, the target model includes multiple processing units, each of which can process its corresponding environmental data. Since the types of environmental data differ, the processing units can also handle different types of environmental data in different ways. Multiple types of environmental data form multimodal data. For the multimodal data of electronic devices, the target model uses multiple processing units corresponding to each modality to process the corresponding modal data, improving processing efficiency and ensuring high accuracy without interference between them.
[0033] S300, based on the first type of environmental data, determine the first processing unit of the target model, and have the first processing unit process the first type of environmental data to obtain the first feature unit corresponding to the first type of environmental data; After inputting the first type and the second type of environmental data into the target model, the target model can determine the processing unit corresponding to each type of environmental data based on relevant indication information in the environmental data. For example, the processing unit corresponding to the environmental data can be determined based on the data characteristics and / or uploaded device information. This includes determining the processing unit of the target model corresponding to the first type of environmental data as the first processing unit, and having the first processing unit process the first type of environmental data. For example, if the first type of environmental data is time series data, the second processing unit is the corresponding time processing unit.
[0034] In one embodiment, during the processing of the first type of environmental data, the first processing unit vectorizes the first type of environmental data to facilitate its use by the inference unit of the target model. In another embodiment, the vectorized first type of environmental data can be digitized to form a first feature unit. The first feature unit can be directly input into the inference unit of the target model for inference.
[0035] S400, based on the second type of environmental data, determine the second processing unit of the target model, and have the second processing unit process the second type of environmental data to obtain the second feature unit corresponding to the second type of environmental data.
[0036] Similar to the embodiments described above, after the second type of environmental data is input into the target model, the target model can determine the second processing unit corresponding to the second type of environmental data based on the relevant indication information in the second type of environmental data. For example, the second processing unit can be determined based on the data characteristics of the second type of environmental data and / or uploaded device information. The second processing unit then processes the second type of environmental data. For example, if the second type of environmental data is image data, the second processing unit is the corresponding image processing unit.
[0037] In one embodiment, during the processing of the second type of environmental data, the second processing unit vectorizes the second type of environmental data to facilitate its use by the inference unit of the target model. In another embodiment, the vectorized second type of environmental data can be text-based to form a second feature unit. The second feature unit can be directly input into the inference unit of the target model for inference.
[0038] S500, the target model performs reasoning based on the first feature unit and the second feature unit to determine the application scenario of the electronic device.
[0039] For example, the inference unit in the target model can be built based on a deep neural network, which can be used to perform inference based on input data and obtain inference results. In one embodiment, the target model can concatenate the first feature unit and the second feature unit, for example, according to the time sequence of the input, to form a complete input data. The inference unit performs inference based on this complete input data to determine the application scenario of the electronic device, such as a work scenario, a meeting scenario, an entertainment scenario, a mobile scenario, a static scenario, etc. In one embodiment, both the first feature unit and the second feature unit can be embeddings, which are feature information that the inference unit can understand.
[0040] In one embodiment, the inference unit can be built based on Transformer, and its decoder interprets the contents of the first feature unit and the second feature unit. The application scenario of the electronic device is then determined based on the interpreted content.
[0041] The multimodal data processing method of this application can classify multimodal data and process the corresponding multimodal data using multiple processing units in the target model. Due to the comprehensiveness of multimodal data, the target model can effectively improve the efficiency and accuracy of data processing, thereby accurately determining the application scenario of the electronic device.
[0042] In one embodiment of this application, determining the environmental data of the electronic device includes: According to the data formats of different types of environmental data, at least one type of data, namely, the data collected from the acquisition component of the electronic device and the application data being run, is processed to form a first type of environmental data and a second type of environmental data; the data formats of the different types of environmental data are determined by the component information and / or data attributes of the acquisition component.
[0043] For example, different types of environmental data have different data formats. For instance, the data format of images acquired by an image acquisition device differs from the data format of audio acquired by an audio acquisition component. The data acquired by the acquisition component also differs in format from the application data generated during program execution. In this embodiment, the acquisition device can process the acquired data from the acquisition component according to the data formats of different types of environmental data; the operating system of the electronic device can also process the application data from the running system according to the data formats of different types of environmental data. This forms a first type of environmental data and a second type of environmental data, which are then input into the target model. This allows the target model to identify the data promptly and accurately, determine the processing unit corresponding to that type of environmental data, and ensure the accuracy of environmental data processing.
[0044] For example, the acquisition components include at least one of the following: an RGB camera, an infrared camera, an ultrasonic sensor, an ambient light sensor, a microphone, an accelerometer, and a gyroscope. Different acquisition devices collect environmental data in different formats, and the collected data can be processed according to different formats to form either Type I or Type II environmental data. For example, data collected by an ambient light sensor can be processed into Type I environmental data, while data collected by an RGB camera and an infrared camera can be processed into Type II environmental data.
[0045] In one embodiment, the data format of different types of environmental data can be determined by the component information and / or data attributes of the acquisition components. For example, based on the component information of RGB cameras and infrared cameras (both image acquisition devices), the acquired data is determined to be the second type of environmental data. Data acquired by ultrasonic sensors has electromagnetic signal attributes, and data with these electromagnetic signal attributes can be determined to be the first type of environmental data. Data acquired by microphones has audio-to-text conversion attributes, and data with these conversion attributes is determined to be the third type of environmental data.
[0046] In one embodiment of this application, the first type of environmental data includes time-series data, and the first processing unit processes the first type of environmental data, such as... Figure 2 As shown, it includes the following steps: S310, the first processing unit determines a first time feature and a second time feature of the time series data, wherein the period of the first time feature is greater than the period of the second time feature.
[0047] For example, the first type of environmental data includes time-series data, which has temporal characteristics. For instance, data collected by ultrasonic sensors and ambient light sensors exhibits temporal characteristics, changing over time. Time-series data possesses a first time characteristic and a second time characteristic. The first time characteristic can be the long-term trend of the time-series data, while the second time characteristic can be the short-term fluctuations. The long-term trend (T) and short-term fluctuations (including S / C / I) are the core components of time-series data. The former determines the long-term evolution direction of the time-series data, while the latter represents short-term deviations and oscillations superimposed on the trend. Together, they comprehensively characterize the dynamic changes of the time-series data.
[0048] S320, Based on the first time feature and the second time feature of the time series data, determine the multi-channel time series data corresponding to the time series data.
[0049] For example, multi-channel time-series data is a collection of time-series data of multiple related indicators collected synchronously under the same time dimension. Based on the first and second time features characterizing the dynamic changes of time-series data, multi-channel filtering and scaling operations are performed on the time-series data to determine the corresponding multi-channel time-series data. The multi-channel filtering operation involves weighted smoothing and feature extraction of the time and / or spatial dimensions of the time-series data, extending the filtering operation to multiple synchronous feature channels. An appropriate filtering method is selected for the numerical characteristics of each channel (such as dimensions, fluctuation patterns, and noise types), ultimately outputting filtered data that is perfectly aligned with the number of input channels and time steps. The scaling operations involve upsampling (enlarging, scaling ratio > 1) or downsampling (reducing, scaling ratio < 1) the time dimension (time series: T) or spatial dimension (image: H / W) of the time-series data according to a specified scaling factor, while ensuring that the channel dimensions remain unchanged, multi-channel synchronous scaling, and dimension alignment are maintained. This is an operation for adapting multi-channel data to model input, feature fusion, and resolution adjustment.
[0050] S330, quantize the multi-channel time-series data to form a first feature unit.
[0051] For example, quantizing multi-channel time series data involves preprocessing and optimizing the data to reduce data storage costs and improve computational efficiency, while preserving the core characteristics of the data (such as trends, correlations, and details) within an acceptable range.
[0052] In one embodiment, after quantizing the multi-channel time-series data, a text encoder can be used to process the quantized multi-channel time-series data into text, thereby forming the first feature unit.
[0053] In one embodiment of this application, such as Figure 3 As shown, the method further includes the following steps: S600, based on a preset time identifier, determine whether the environmental data has time data characteristics; S700, if the environmental data has time data characteristics, the environmental data is determined to be the time series data.
[0054] For example, during the preprocessing of environmental data, a preset time identifier can be added to the time series data. This preset time identifier can indicate that the data is time series data, or it can represent the ontology of the time series data. For example, time series data includes data definition, time series data ontology, and expected target. Preset time identifiers can be added before and after the time series data ontology, such as...<time_start> and<time_end> Among them,<time_start> Add it before the time series data body, and then...<time_end> After being added to the time-series data ontology, the target model can determine that the time-series data ontology in the environmental data has time data characteristics based on the preset time identifier, thereby identifying the environmental data as time-series data.
[0055] In one embodiment of this application, the second type of environmental data includes image data, and the second processing unit processes the second type of environmental data, such as... Figure 4 As shown, it includes: S410, using the image encoder in the second processing unit, the encoding of the image data is aligned with the encoding of the reference image. The alignment operation includes upsampling and / or downsampling the image data through the convolutional layer of the target model.
[0056] For example, feature extraction of image data can be trained based on a reference image. This reference image has the same or similar content as the image data and can be used as a reference during the image data processing in the second processing unit, such as when performing feature extraction. Specifically, the image encoder in the second processing unit can be used to align the encoding of the image data with the encoding of the reference image. Aligning the encoding of the image data with the encoding of the reference image maps image encodings extracted from different sources, scales, modalities, or networks to a unified dimensional space, feature distribution, or semantic space. This makes the aligned encodings comparable, fusionable, and computable, thereby ensuring that the number of channels and spatial dimensions of different encodings are consistent and the feature distributions are similar.
[0057] In one embodiment, the specific means of implementing the above alignment operation includes upsampling and / or downsampling the image data through the convolutional layers of the target model. Upsampling enlarges the spatial dimensions (height H, width W) of the image data with a scaling ratio greater than 1. The core principle is to generate new pixels / grid points through interpolation, feature learning, etc., thereby improving the spatial resolution of the data. The final output H and W dimensions are larger than the original data. Downsampling reduces the spatial dimensions (height H, width W) of the image data with a scaling ratio less than 1. The core principle is to retain key information through selection, aggregation, feature extraction, etc., thereby reducing the spatial resolution of the data. The final output H and W dimensions are smaller than the original data.
[0058] For example, if the image data is a generalized image, the convolutional layer of the target model can perform upsampling and / or downsampling on the generalized image, scaling the spatial dimension while keeping the channel dimension unchanged and the spatial alignment between channels unchanged. The appropriate scaling method can be selected according to the data characteristics of the generalized image (such as continuous / discrete, whether it contains semantic information, and the type of noise).
[0059] S420, Based on the image data after the alignment operation, generate a corresponding image data vector.
[0060] For example, the target model will perform vectorization processing on the image data after the alignment operation. Vectorization processing is to transform the multi-channel grid-like features with spatial dimensions into a one-dimensional semantic feature vector through spatial dimension compression and channel feature aggregation, while retaining the core visual and semantic features of the image data (such as generalized images), in preparation for subsequent image retrieval, classification, clustering, cross-modal fusion and other operations.
[0061] S430, perform text description on the image data vector to obtain the corresponding second feature unit.
[0062] For example, a text encoder can be used to textify the image data vector to obtain the second feature unit.
[0063] Preferably, based on the image data after the alignment operation, a corresponding image data vector is generated, such as... Figure 5 As shown, it includes the following steps: Based on the image data after the alignment operation and the output classification layer in the image encoder, determine multiple image type vectors with relatively high classification probabilities. Multiple image type vectors are fused together to form the image data vector.
[0064] Specifically, taking generalized images as an example, the output classification layer in the image encoder aims to optimize the high cosine similarity of the classification probability distribution between the reference image and the generalized image data. During the image encoder process, when the generalized image is input, multiple image type vectors with high feature classification probabilities are obtained by aligning the convolutional layer (CNN) and the output classification layer. These multiple image type vectors are then fused to serve as the encoding for the generalized image. This fusion can be achieved through methods such as concatenation or probability superposition. Finally, an image data vector (generalized image vector) is formed based on the encoding of the generalized image.
[0065] In one embodiment, if the second type of environmental data is an RGB image, the second feature unit can be obtained by aligning it with the corresponding reference text using an image encoder via the Clip method.
[0066] In one embodiment of this application, the environmental data includes at least a third type of environmental data, the third type of environmental data including text data, and the method further includes: The third processing unit of the target model processes the third type of environmental data, including: when there is no other modal data in the text data, the third processing unit performs vectorization processing on the text data to generate the corresponding third feature unit.
[0067] For example, the third type of environmental data includes text data, such as text data formed by data collected by the microphone of an electronic device. Based on the data characteristics of the third type of environmental data and / or uploaded device information, it is determined that it is the third type of environmental data, and a third processing unit corresponding to the third type of environmental data is determined. If the third type of environmental data only contains text data and no other modal data, the text data can be directly vectorized, and a text encoder can be used to describe the vectorized text data to generate the corresponding third feature unit.
[0068] In one embodiment of this application, the environmental data collected by the acquisition component includes a first type of environmental data, a second type of environmental data, or a third type of environmental data, and the application data running on the electronic device includes the third type of environmental data.
[0069] For example, the acquisition components of electronic devices include various types, such as Figure 5As shown, examples include RGB cameras, infrared cameras, ultrasonic sensors, ambient light sensors, microphones, accelerometers, and gyroscopes. This type of data acquisition component can collect relevant data about the environment in which the electronic device operates. Specifically, the data that can be collected includes first-type, second-type, or third-type environmental data. Application data generated by the electronic device, such as data generated by program execution, is considered third-type environmental data; for example, application data generated by the electronic device can be represented solely by text data.
[0070] In one embodiment of this application, the target model performs inference based on a first feature unit and a second feature unit, including: The target model concatenates at least two of the first feature unit, the second feature unit, and the third feature unit, and inputs the concatenation result into the inference unit of the target model to determine the application scenario of the electronic device.
[0071] For example, the inference unit of the target model can be built based on Transformer and is capable of inference. The data input to the inference unit can be a whole. In this embodiment, at least two of the first, second, and third feature units can be concatenated according to the chronological order of the collected environmental data of various types and / or the chronological order of the generation of the first, second, and third feature units. For example, the first and second feature units can be concatenated, or the first and third feature units can be concatenated, or the second and third feature units can be concatenated, or the first, second, and third feature units can be concatenated. The concatenation result is then input into the inference unit of the target model, which performs inference to determine the application scenario of the electronic device.
[0072] In one embodiment of this application, the first type of environmental data is constructed based on a first data definition, a first time-series data, and a first desired target; the second type of environmental data is constructed based on a second data definition, a second image data, and a second desired target; and the third type of environmental data is constructed based on a third data definition, a third data format, third text data, and a third desired target.
[0073] For example, the first type of environmental data Contains time series data It also contains the definition of this data. and the expected goals of preprocessing The formula is as follows:
[0074] Use data to identify the front and back. <time_start><time_end>Separate.
[0075] The ultrasonic data collected by the aforementioned ultrasonic sensor can be converted into first-type environmental data, such as time-series data. Furthermore, data such as sound, ultrasound, and A+G data can be converted into time-series descriptions.
[0076] Second type of environmental data Contains spatial distribution data It should also include the definition of this data. and the expected goals of preprocessing The formula is as follows:
[0077] Use data to identify the front and back. <figure_start> <figure _end> Separate.
[0078] The data collected by the aforementioned RGB and infrared cameras can be converted into a second type of environmental data, such as generalized images.
[0079] Third type of environmental data Contains data values It should also include the definition of this data. Data format and the expected goals of preprocessing The formula is as follows:
[0080] Ambient light data collected by ambient light sensors, as well as data on the application and performance ratio of electronic devices, can be converted into a third type of environmental data, such as text data.
[0081] In one scenario embodiment, the electronic device acquires multimodal data features, which can identify the electronic device as being in a mobile usage scenario.
[0082] Image: Rapidly changing environment (drastic changes in lighting, blurred background), image recognition of vehicle features (seat back, tray table).
[0083] Sound: Continuous low-frequency noise (about 85dB from an aircraft engine or the sound of high-speed rail wheels), accompanied by irregular vibrations.
[0084] A+G timing: Key criteria - The accelerometer shows continuous low-frequency vibration above 1Hz (characteristic of vehicles), the gyroscope detects random high-frequency jitter (road bumps), and the overall composite acceleration fluctuates between 0.8-1.2g (non-stationary state).
[0085] PC applications: very little keyboard input (occasionally arrow keys), unstable network traffic (cellular network characteristics), and running lightweight applications.
[0086] By combining the moving environment in the image, traffic noise in the sound, strong A+G motion characteristics, and light interactive applications, a mobile usage scenario is determined. Based on this application scenario, the system can perform corresponding operations, such as reducing the touchpad sensitivity to prevent accidental triggering, or switching to power-saving mode.
[0087] In another embodiment, the electronic device acquires multimodal data features, which can identify that the electronic device is in a multi-person scenario, such as sharing or a meeting.
[0088] Image: Multiple faces detected, with faces facing the center of the screen.
[0089] Ultrasonic: Detects users at a slightly greater distance (ultrasonic distance 80-100cm).
[0090] Sound: Multiple voice sources were detected, and keyword information such as "turn page" and "please see" was identified.
[0091] A+G timing: The device is placed on the desktop, and the acceleration is stable, but the gyroscope detects periodic slight rotations (the user adjusts the screen angle for multiple people to view).
[0092] PC Applications: The foreground is a PPT presentation or image viewer. Keyboard input uses regular PageDown / PageUp switching, and mouse activity is reduced. Based on image + distance detection + multi-person audio interaction + A+G screen angle adjustment + presentation applications, etc., the system determines that it is a content sharing scenario. The system automatically adjusts the brightness to be suitable for viewing from multiple angles, or enables "presentation mode" (disabling screen saver and automatic sleep), or adjusts the audio from single-person meeting mode to multi-person meeting mode, and collects the voices of multiple people, etc.
[0093] This application also provides a multimodal data processing device that can be applied to electronic devices, such as... Figure 6 As shown and combined Figure 5 The data processing device includes: A data acquisition component configured to determine environmental data of an electronic device, wherein the environmental data originates from at least one of the following: data acquired by the data acquisition component of the electronic device and running application data; the environmental data includes at least a first type of environmental data and a second type of environmental data. An operation module is configured to input first-type environmental data and second-type environmental data into a target model, wherein the target model includes processing units corresponding to different types of environmental data. The target model is configured to determine a first processing unit of the target model based on the first type of environmental data, and the first processing unit processes the first type of environmental data to obtain a first feature unit corresponding to the first type of environmental data. Based on the second type of environmental data, a second processing unit of the target model is determined, and the second processing unit processes the second type of environmental data to obtain a second feature unit corresponding to the second type of environmental data; Reasoning is performed based on the first feature unit and the second feature unit to determine the application scenario of the electronic device.
[0094] For example, an electronic device can be a terminal device, a service device, etc. Environmental data comes from at least one of the following: data collected by the electronic device's acquisition components and application data during operation. Specifically, the electronic device has acquisition components, which include various types of acquisition devices, each capable of collecting corresponding environmental data, such as image data, audio data, light signals, and motion data. On the other hand, the electronic device also has programs installed, including operating systems and applications, which generate corresponding application data during operation. This application data characterizes the program's operating status and its impact on the electronic device's usage scenario.
[0095] In this embodiment, the environmental data includes at least a first type of environmental data and a second type of environmental data. Of course, other types of environmental data may also be included, such as a third type, a fourth type, etc. These various types of environmental data form multimodal data, and each type of environmental data can represent one modality. The data format and content of different types of environmental data can differ.
[0096] In one embodiment of this application, after determining the environmental data, the acquisition component or operation module can preprocess the environmental data so that the preprocessed environmental data can be adapted to the target model so that the target model can process the environmental data, such as data format processing, noise removal, and data size adjustment. For example, the acquired environmental data can be preprocessed separately by each acquisition component. Alternatively, the acquired environmental data can be preprocessed separately by the operating system of the electronic device.
[0097] In one embodiment, preprocessing can transform environmental data into different types of input data, such as text data, time-series data, and image data. The data formats, content, and characteristics of these different input data can vary. Then, the processed input data of each type is fed into the target model, which will further process each type of input data accordingly.
[0098] In one embodiment, preprocessing includes adding identification information corresponding to the type of multiple types of environmental data, such as adding first identification information to environmental data of the first type and second identification information to environmental data of the second type, so that the target model can identify the identification information to determine the type of environmental data.
[0099] The target model can be a model built on deep neural networks. For example, the target model can be built on a large language model (LLM), which achieves deep understanding and efficient generation of natural language through training on massive amounts of text data and a complex neural network architecture.
[0100] In this embodiment, the target model includes multiple processing units, each of which can process its corresponding environmental data. Since the types of environmental data differ, the processing units can also handle different types of environmental data in different ways. Multiple types of environmental data form multimodal data. For the multimodal data of electronic devices, the target model uses multiple processing units corresponding to each modality to process the corresponding modal data, improving processing efficiency and ensuring high accuracy without interference between them.
[0101] After inputting the first type and the second type of environmental data into the target model, the target model can determine the processing unit corresponding to each type of environmental data based on relevant indication information in the environmental data. For example, the processing unit corresponding to the environmental data can be determined based on the data characteristics and / or uploaded device information. This includes determining the processing unit of the target model corresponding to the first type of environmental data as the first processing unit, and having the first processing unit process the first type of environmental data. For example, if the first type of environmental data is time series data, the second processing unit is the corresponding time processing unit.
[0102] In one embodiment, during the processing of the first type of environmental data, the first processing unit vectorizes the first type of environmental data to facilitate its use by the inference unit of the target model. In another embodiment, the vectorized first type of environmental data can be digitized to form a first feature unit. The first feature unit can be directly input into the inference unit of the target model for inference.
[0103] Similar to the embodiments described above, after the second type of environmental data is input into the target model, the target model can determine the second processing unit corresponding to the second type of environmental data based on the relevant indication information in the second type of environmental data. For example, the second processing unit can be determined based on the data characteristics of the second type of environmental data and / or uploaded device information. The second processing unit then processes the second type of environmental data. For example, if the second type of environmental data is image data, the second processing unit is the corresponding image processing unit.
[0104] In one embodiment, during the processing of the second type of environmental data, the second processing unit vectorizes the second type of environmental data to facilitate its use by the inference unit of the target model. In another embodiment, the vectorized second type of environmental data can be text-based to form a second feature unit. The second feature unit can be directly input into the inference unit of the target model for inference.
[0105] The inference unit in the target model can be constructed based on a deep neural network and can be used to perform inference based on input data and obtain inference results. In one embodiment, the target model can concatenate a first feature unit and a second feature unit, for example, concatenating them according to the time sequence of the input, to form a complete input data. The inference unit performs inference based on this complete input data to determine the application scenario of the electronic device, such as a work scenario, a meeting scenario, an entertainment scenario, a mobile scenario, a static scenario, etc. In one embodiment, both the first feature unit and the second feature unit can be embeddings, which are feature information that the inference unit can understand.
[0106] In one embodiment, the inference unit can be built based on Transformer, and its decoder interprets the contents of the first feature unit and the second feature unit. The application scenario of the electronic device is then determined based on the interpreted content.
[0107] In one embodiment of this application, the target model is further configured to: process at least one type of data, namely, the collected data from the electronic device's acquisition component and the running application data, according to the data format of different types of environmental data, to form a first type of environmental data and a second type of environmental data; the data format of the different types of environmental data is determined by the component information and / or data attributes of the acquisition component.
[0108] In one embodiment of this application, the first type of environmental data includes time-series data, and the target model is further configured as follows: The first processing unit determines a first time feature and a second time feature of the time series data, wherein the period of the first time feature is greater than the period of the second time feature. Based on the first and second time features of the time series data, determine the multi-channel time series data corresponding to the time series data; The multi-channel time-series data is quantized to form the first feature unit.
[0109] In one embodiment of this application, the target model is further configured as follows: Based on a preset time identifier, determine whether the environmental data has time data characteristics; If the environmental data has time data characteristics, the environmental data is identified as the time series data.
[0110] In one embodiment of this application, the target model is further configured as follows: Using the image encoder in the second processing unit, the encoding of the image data is aligned with the encoding of the reference image. The alignment operation includes upsampling and / or downsampling the image data through the convolutional layer of the target model. Based on the image data after the alignment operation, a corresponding image data vector is generated; The image data vector is described with text to obtain the corresponding second feature unit.
[0111] In one embodiment of this application, the target model is further configured as follows: Based on the image data after the alignment operation and the output classification layer in the image encoder, determine multiple image type vectors with relatively high classification probabilities. Multiple image type vectors are fused together to form the image data vector.
[0112] In one embodiment of this application, the environmental data includes at least a third type of environmental data, the third type of environmental data including text data, and the target model is further configured as follows: The third processing unit of the target model processes the third type of environmental data, including: when there is no other modal data in the text data, the third processing unit performs vectorization processing on the text data to generate the corresponding third feature unit.
[0113] In one embodiment of this application, the environmental data collected by the acquisition component includes a first type of environmental data, a second type of environmental data, or a third type of environmental data, and the application data running on the electronic device includes the third type of environmental data.
[0114] In one embodiment of this application, the target model is further configured to: concatenate at least two of the first feature unit, the second feature unit, and the third feature unit, and input the concatenation result into the inference unit of the target model to determine the application scenario of the electronic device.
[0115] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.< / figure>
Claims
1. A multimodal data processing method, comprising: The environmental data of the electronic device is determined, wherein the environmental data comes from at least one of the following: data collected by the electronic device's acquisition components and application data in operation; the environmental data includes at least a first type of environmental data and a second type of environmental data; The first type of environmental data and the second type of environmental data are input into the target model, which includes processing units corresponding to the different types of environmental data. Based on the first type of environmental data, a first processing unit of the target model is determined, and the first processing unit processes the first type of environmental data to obtain a first feature unit corresponding to the first type of environmental data. Based on the second type of environmental data, a second processing unit of the target model is determined, and the second processing unit processes the second type of environmental data to obtain the second feature unit corresponding to the second type of environmental data; The target model performs inference based on the first feature unit and the second feature unit to determine the application scenario of the electronic device.
2. The multimodal data processing method according to claim 1, wherein determining the environmental data of the electronic device includes: According to the data formats of different types of environmental data, at least one type of data from the acquisition components of the electronic device and the running application data are processed to form a first type of environmental data and a second type of environmental data; the data formats of the different types of environmental data are determined by the component information and / or data attributes of the acquisition components.
3. The multimodal data processing method according to claim 2, wherein the first type of environmental data includes time series data, and the first processing unit processes the first type of environmental data, including: The first processing unit determines a first time feature and a second time feature of the time series data, wherein the period of the first time feature is greater than the period of the second time feature. Based on the first and second time features of the time series data, determine the multi-channel time series data corresponding to the time series data; The multi-channel time-series data is quantized to form the first feature unit.
4. The multimodal data processing method according to claim 3, further comprising: Based on a preset time identifier, determine whether the environmental data has time data characteristics; If the environmental data has time data characteristics, the environmental data is identified as the time series data.
5. The multimodal data processing method according to claim 2, wherein the second type of environmental data includes image data, and the second processing unit processes the second type of environmental data, including: Using the image encoder in the second processing unit, the encoding of the image data is aligned with the encoding of the reference image. The alignment operation includes upsampling and / or downsampling the image data through the convolutional layer of the target model. Based on the image data after the alignment operation, a corresponding image data vector is generated; The image data vector is described with text to obtain the corresponding second feature unit.
6. The multimodal data processing method according to claim 5, wherein generating a corresponding image data vector based on the image data after the alignment operation comprises: Based on the image data after the alignment operation and the output classification layer in the image encoder, determine multiple image type vectors with relatively high classification probabilities. Multiple image type vectors are fused together to form the image data vector.
7. The multimodal data processing method according to claim 2, wherein the environmental data includes at least a third type of environmental data, the third type of environmental data includes text data, and the method further includes: The third processing unit of the target model processes the third type of environmental data, including: when there is no other modal data in the text data, the third processing unit performs vectorization processing on the text data to generate the corresponding third feature unit.
8. The multimodal data processing method according to claim 7, wherein the environmental data collected by the acquisition component includes a first type of environmental data, a second type of environmental data, or a third type of environmental data, and the application data running on the electronic device includes the third type of environmental data.
9. The multimodal data processing method according to claim 7, wherein the target model performs inference based on a first feature unit and a second feature unit, comprising: The target model concatenates at least two of the first feature unit, the second feature unit, and the third feature unit, and inputs the concatenation result into the inference unit of the target model to determine the application scenario of the electronic device.
10. A multimodal data processing apparatus, comprising: A data acquisition component configured to determine environmental data of an electronic device, wherein the environmental data originates from at least one of the following: data acquired by the data acquisition component of the electronic device and running application data; the environmental data includes at least a first type of environmental data and a second type of environmental data. An operation module is configured to input first-type environmental data and second-type environmental data into a target model, wherein the target model includes processing units corresponding to different types of environmental data. The target model is configured to determine a first processing unit of the target model based on the first type of environmental data, and the first processing unit processes the first type of environmental data to obtain a first feature unit corresponding to the first type of environmental data. Based on the second type of environmental data, a second processing unit of the target model is determined, and the second processing unit processes the second type of environmental data to obtain a second feature unit corresponding to the second type of environmental data; Reasoning is performed based on the first feature unit and the second feature unit to determine the application scenario of the electronic device.