Multi-modal data acquisition method and apparatus based on sidecar container

Abstract

Embodiments of the present disclosure provide a multi-modal data collection method and device based on a Sidecar container, which comprises: encapsulating a cloud desktop by using a Kubernetes containerization technology, so that each user instance runs in an independent container; deploying a Sidecar container beside a main container of the cloud desktop, the Sidecar container and the main container running in parallel; loading a data collection component in the Sidecar container, and monitoring and collecting various types of user behavior data in real time through API Hook, log monitoring, OCR and voice analysis; and transmitting the collected user behavior data to a local user terminal by using end-to-end encryption technology.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of cloud desktop, in particular, to a multi-modal data collection method and device based on Sidecar container. BACKGROUND

[0002] Cloud desktop is a virtual desktop solution based on cloud computing technology, which provides desktop operating systems (such as Windows, Linux, etc.) and application programs to users through cloud servers. Users can remotely access and use these desktop environments through terminal devices (such as computers, mobile phones, tablets, etc.). The core idea of cloud desktop is to host computing resources, storage and applications in the cloud, so that users do not need to install and manage these resources on local devices, providing high flexibility, scalability and ease of management advantages.

[0003] In the cloud desktop environment, user operations involve multiple data types, such as text input, operation logs, screen content, voice instructions, etc. Traditional data collection methods usually rely on centralized monitoring or client plug-ins, which have high resource occupation, large delay, and insufficient security, etc. SUMMARY

[0004] In order to efficiently and non-invasively monitor and collect data of cloud desktop user behavior in real time, the embodiments described herein provide a multi-modal data collection method and device based on Sidecar container and a computer readable storage medium storing a computer program.

[0005] According to a first aspect of the present disclosure, a multi-modal data collection method based on Sidecar container is provided, comprising: encapsulating a cloud desktop using Kubernetes containerization technology, so that each user instance runs in an independent container; deploying a Sidecar container beside the cloud desktop master container, the Sidecar container and the master container run in parallel; loading a data collection component in the Sidecar container, and monitoring and collecting multiple types of user behavior data in real time through API Hook, log monitoring, OCR and voice analysis; and using end-to-end encryption technology to transmit the collected user behavior data to the local user terminal.

[0006] In some embodiments of the present disclosure, encapsulating a cloud desktop using Kubernetes containerization technology, so that each user instance runs in an independent container, comprises: packaging an operating system, a desktop environment, commonly used application programs and user configurations into a Docker image, generating an independent user instance based on the image after each user logs in; creating a Kubernetes resource configuration file for each user instance, each configuration file defining a Pod to run each user instance; and configuring persistent storage for each user instance to save user data.

[0007] In some embodiments of the present disclosure, deploying a Sidecar container beside the cloud desktop main container, the Sidecar container and the main container run in parallel, including: running an application in the cloud desktop main container, the Sidecar container and the main container share a network and a storage system through a container orchestration tool; and keeping the Sidecar container and the main container running synchronously through a heartbeat mechanism, and automatically restarting the Sidecar container when an exception occurs.

[0008] In some embodiments of the present disclosure, loading a data collection component in the Sidecar container, monitoring and collecting various types of user behavior data in real time through API Hook, log monitoring, OCR and voice analysis, including: embedding an APIHook in the main container, intercepting application calls through the Sidecar container to obtain application data; the Sidecar container obtains log data about system behavior, user operation and application state by monitoring system operation logs, application logs and service logs; the Sidecar container analyzes visual data on the application interface through OCR technology and extracts text data from the visual data; and the Sidecar container captures user voice input in real time through voice collection technology and converts the voice input into text data.

[0009] In some embodiments of the present disclosure, the application data includes user text input in a document, a chat application or a code editor, the log data includes application startup, shutdown, usage frequency, interaction mode and user read, modify, delete operation and operation track on a file, the visual data includes visible text information on the user interface, and the voice input includes user verbal instructions, conversations or other voice interactions.

[0010] In some embodiments of the present disclosure, the method further comprises: utilizing a multi-modal large language model deployed on the user terminal to perform cross-modal data fusion analysis on the user behavior data; and feeding back the analysis result to the Sidecar container, and adjusting the data collection strategy through a real-time feedback mechanism when detecting a change in the user's behavior pattern.

[0011] In some embodiments of the present disclosure, the cross-modal data fusion analysis of user behavior data by the multi-modal large language model deployed on the user terminal includes: inputting the text data converted from the voice input and the text data converted from the visual data into the multi-modal large language model based on the Transformer architecture, capturing the semantic structure in the text through the self-attention mechanism, understanding the text content, and generating personalized work suggestions; and fusing the features of the log data and the application data by the multi-modal large language model based on the Transformer architecture, learning the relationship between the logs and the behavior patterns, and identifying sensitive operations, abnormal behaviors, and potential security risks.

[0012] In some embodiments of the present disclosure, the analysis results are fed back to the Sidecar container, and when the change in the user's behavior pattern is detected, the data collection strategy is adjusted through a real-time feedback mechanism, including: modifying the frequency, type, or granularity of data collection according to the change in the user's behavior pattern; and adjusting the storage method and processing mechanism of the data according to the change in the data collection strategy.

[0013] According to a second aspect of the present disclosure, a multi-modal data collection device based on a Sidecar container is provided. The device includes at least one processor; and at least one memory storing a computer program. When the computer program is executed by the at least one processor, the device is caused to: encapsulate the cloud desktop using the Kubernetes containerization technology, so that each user instance runs in an independent container; deploy a Sidecar container beside the cloud desktop master container, and the Sidecar container and the master container run in parallel; load a data collection component in the Sidecar container, and monitor and collect multiple types of user behavior data in real time through API Hook, log monitoring, OCR, and voice analysis; and transmit the collected user behavior data to the local user terminal using end-to-end encryption technology.

[0014] According to a third aspect of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein the computer program, when executed by a processor, implements the steps of the method according to the first aspect of the present disclosure.

[0015] According to the Sidecar container-based multi-modal data collection method and device of the embodiments of the present disclosure, through containerization technology, each user instance runs in an independent container, which can ensure that the data of different users do not interfere with each other, enhances data isolation, and helps to improve security in a multi-user environment. By using end-to-end encryption technology, it is ensured that user behavior data is encrypted and protected from the collection to the transmission process, which enhances the confidentiality and integrity of the data. By loading various data collection components such as API Hook, log monitoring, OCR, and voice analysis in the Sidecar container, various types of user behavior data can be collected in real time. This multi-modal data collection capability enables comprehensive monitoring of various user interaction methods, thereby improving the comprehensiveness and accuracy of data analysis. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly described below. It should be noted that the drawings described below only relate to some embodiments of the present disclosure, but not limit the present disclosure, wherein:

[0017] Figure 1 is an exemplary flowchart of a Sidecar container-based multi-modal data collection method according to an embodiment of the present disclosure;

[0018] Figure 2 is a schematic block diagram of a Sidecar container-based multi-modal data collection device according to an embodiment of the present disclosure.

[0019] It should be noted that the elements in the drawings are schematic and not drawn to scale. DETAILED DESCRIPTION

[0020] In order to make the purposes, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative labor also belong to the scope of protection of the present disclosure.

[0021] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the specification and relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein. Additionally, terms such as "first" and "second" are used merely as labels to distinguish between two (2) elements (or portions thereof) from one another.

[0022] Cloud desktop itself refers to a remote desktop service that provides desktop operating systems and applications to users through virtualization technology, while containerization is a technology that packages applications and their dependencies into independent containers, enabling lightweight, convenient deployment and management. Containerized cloud desktop combines the advantages of both, using container technology to manage and deploy desktop instances, making each user's desktop environment more efficient, flexible, and able to scale resources on demand. The embodiments of the present disclosure aim to achieve efficient cloud desktop environment management through containerization technology, while understanding each user's needs, habits, and behavior patterns by collecting and analyzing user behavior and data in the cloud desktop, thereby providing personalized services.

[0023] Figure 1 An exemplary flowchart of a Sidecar container-based multi-modal data collection method according to an embodiment of the present disclosure is shown. In Figure 1 At block S102 of the method 100 shown, Kubernetes containerization technology is used to encapsulate the cloud desktop, with each user instance running in an independent container.

[0024] According to one embodiment of the present disclosure, the operating system, desktop environment, commonly used applications, and user configurations are packaged into a Docker image, and an independent user instance is generated for each user based on the image after login; a Kubernetes resource configuration file is created for each user instance, and each configuration file defines a Pod to run each user instance; persistent storage is configured for each user instance to save user data.

[0025] Kubernetes, as a container orchestration platform, can automate the management of container deployment, expansion, scheduling, load balancing, and other tasks. By using Docker container technology, a completely isolated running environment is provided for each user, ensuring that users do not interfere with each other or leak data. Using Kubernetes, each container instance is defined by a Pod, and each user's desktop environment runs in a separate Pod, which contains one or more containers. User data such as files, application configurations, etc. can be managed through the persistent storage mechanism provided by Kubernetes. This architecture can not only improve resource utilization, but also effectively support large-scale, multi-user cloud desktop environments.

[0026] In block S104, a Sidecar container is deployed alongside the cloud desktop master container, and the Sidecar container and the master container run in parallel.

[0027] In Kubernetes, the master container is responsible for running basic services and applications of the cloud desktop, such as operating systems, desktop applications, UIs, and configurations. The Sidecar container is a container that runs in parallel with the master container and shares the same Pod with the master container, so it can share the same network namespace, storage system, and other resources. The Sidecar container can be scaled or replaced as needed without affecting the applications in the master container, thereby improving the decoupling and maintainability of the system.

[0028] For example, the Sidecar container can monitor user operation behavior, input, system resource usage, etc., collect real-time data for analysis, record user operation logs, error logs, and monitor the health status of the cloud desktop. The Sidecar container manages communication with external networks to ensure the security and traffic management of network access. The Sidecar container can also act as a proxy, responsible for routing requests or providing specific services, such as running Nginx reverse proxy through the Sidecar container to provide load balancing functions for the master container.

[0029] To ensure the synchronized operation of the master container and the Sidecar container, a heartbeat mechanism can be used to detect the health status of the container and restart it when necessary. The heartbeat mechanism can detect whether the container is alive through Liveness Probe. If the container does not respond to the heartbeat, it will be restarted. Alternatively, Readiness Probe can be used to detect whether the container is ready to receive traffic, and when the container is not ready, Kubernetes will forward traffic to other healthy containers.

[0030] In Figure 1At block S106, the Sidecar container loads the data collection components, which monitor and collect various types of user behavior data in real-time through API Hook, log monitoring, OCR, and voice analysis.

[0031] Users generate various types of data when operating systems or applications, including: user text input behavior on keyboards or touch devices, which can reflect the user's thinking process and usage habits when interacting with the system. User operation behavior, including mouse clicks, keyboard strokes, application switching, menu selection, etc., these logs can reflect the user's operation mode and habits. Real-time capture or screenshot of the content displayed on the user's screen, which helps to obtain the user's current working state, interaction content or in-use application. Information input by the user through voice assistants or voice recognition systems, voice data not only contains voice text, but also analyzes the user's voice emotion, tone, etc. Load data collection components in the Sidecar container.

[0032] In some embodiments of the present disclosure, API Hook is embedded in the main container, and application data is obtained by intercepting application calls through the Sidecar container. Application data includes user text input in documents, chat applications or code editors, etc. By intercepting text data interaction in the application, the user's input text in document editors, chat applications, development environments, etc. is obtained.

[0033] The Sidecar container obtains log data about system behavior, user operation and application state by monitoring system operation logs, application logs and service logs. Log data includes application startup, shutdown, usage frequency, interaction mode, and user operations on files (opening, saving, deleting, editing) and operation traces, etc. Logs can also include data transmission behavior, such as file download, upload or sharing. Through the analysis of logs, the user's usage mode and demand for data can be obtained.

[0034] The Sidecar container analyzes the visual data on the application interface through OCR technology and extracts text data from the visual data. Visual data includes visible text information on the user interface, such as button labels, menu options, dialog box content, etc., which is particularly important for text information that cannot be directly obtained through API. The Sidecar container captures user voice input in real-time through voice collection technology and converts voice input into text data. Voice input includes user's verbal instructions, conversations or other voice interactions, and the transcription of voice content provides data support for subsequent semantic understanding and task summary.

[0035] To ensure the real-time and efficiency of multiple data collection methods, message queues such as Kafka, RabbitMQ, or NATS can be used to asynchronously transmit collected data to the backend processing system. Each collection method (API Hook, log monitoring, OCR, speech analysis) can send data through the message queue to ensure real-time data transmission and avoid loss. Collected data can be stored in a distributed database (such as MongoDB, Cassandra) or search engine (such as Elasticsearch) for subsequent data analysis and query.

[0036] Finally, in block S108, the collected user behavior data is transmitted to the local user terminal using end-to-end encryption technology.

[0037] By real-time analysis and classification of the sensitivity of collected data, automatically filtering and encrypting data related to sensitive information such as financial data and personal information, the privacy and security of data during analysis are ensured. Sensitive information is automatically de-identified, and end-to-end encryption technology is used to ensure the security of data during cloud storage and transmission. Users can choose to enable transparency mode to view and manage their own generated data at any time. When encrypted data reaches the local user terminal, the session key is first decrypted using asymmetric encryption (using the recipient's private key). Once the session key is decrypted, the user terminal uses the session key to decrypt the encrypted data, restoring the original user behavior data. The decrypted data can be used for further analysis on the local user terminal.

[0038] In some embodiments of the present disclosure, a multi-modal large language model deployed on the user terminal is used to perform cross-modal data fusion analysis on the user behavior data.

[0039] The multi-modal large language model (such as a Transformer-based model) can handle multiple different types of data sources, such as text, speech, and visual information. Specifically, the text data converted from speech input and the text data converted from visual data are input into a multi-modal large language model based on the Transformer architecture, which captures the semantic structure in the text through self-attention mechanisms, understands the text content, and generates personalized work recommendations.

[0040] In addition to personal productivity improvement, communication, file sharing, and collaborative workflows among team members can also be analyzed to provide suggestions for improving collaboration efficiency and help teams better coordinate and communicate. Seamless collaboration between different platforms is supported, allowing users to continuously process work tasks on different devices, and through containerization and cloud synchronization, maintaining the real-time and consistency of data.

[0041] The multi-modal large language model based on the Transformer architecture is used to fuse the features of log data and application data, learn the relationship between log and behavior patterns, and identify sensitive operations, abnormal behaviors, and potential security risks. For example, operations such as user access to confidential files, modification of system settings, etc. can be marked as sensitive operations. After identifying sensitive operations or abnormal behaviors, the system can automatically trigger response measures according to preset rules, such as warning the user, locking the account, or suspending certain services, etc.

[0042] By analyzing the user's behavior patterns, such as click frequency, access path, input content, etc. within a certain period of time, the system can automatically identify significant changes in behavior patterns (e.g. sudden increase in user click frequency, change in usage patterns, etc.) and generate feedback signals. The user terminal will feed back the analysis results to the Sidecar container, and when it detects changes in the user's behavior patterns, it will adjust the data collection strategy through real-time feedback mechanisms. The feedback can be sent from the terminal to the Sidecar container through message queues, WebSockets, or RESTful APIs, etc. The data collection strategy inside the container will receive these feedbacks and adjust as needed, such as modifying the frequency, type, or granularity of data collection; and according to the changes in the data collection strategy, adjust the storage method and processing mechanism of the data.

[0043] For example, if significant changes in user behavior are detected, the container can decide to increase the frequency of data collection in order to more accurately track changes in user behavior. If some behavior patterns have changed (e.g. the user starts to interact frequently with certain interface elements), the type of data collected can be adjusted based on feedback results, such as increasing the collection of certain specific fields or operations. If the user's behavior patterns become more stable or do not change significantly, the system can reduce the collection frequency or stop collecting certain low-priority data according to needs in order to save resources. If the user is mainly active in document editing, writing, report generation, etc., the system can prioritize the collection of text input, document access frequency, keyword usage, etc. If the user is mainly active in code development, debugging, version control, etc., the system can strengthen the collection of code input, programming language usage, development tool usage frequency, etc.

[0044] Figure 2 is a schematic block diagram of a Sidecar container-based multi-modal data collection device according to an embodiment of the present disclosure. As Figure 2 shown, the device 200 can include a processor 210 and a memory 220 storing a computer program. When the computer program is executed by the processor 210, the device 200 can perform the following operations: Figure 1The steps of the method 100 are shown. In one example, the device 200 can be a computer device or a cloud computing node. The device 200 can encapsulate the cloud desktop using the Kubernetes containerization technology, so that each user instance runs in an independent container; deploy a Sidecar container beside the cloud desktop main container, and the Sidecar container and the main container run in parallel; load a data collection component in the Sidecar container, and monitor and collect various types of user behavior data in real time through API Hook, log monitoring, OCR and voice analysis; and use end-to-end encryption technology to transmit the collected user behavior data to the local user terminal.

[0045] In some embodiments of the present disclosure, the device 200 can run an application program in the cloud desktop main container, and the Sidecar container binds with the main container to share the network and storage system through a container orchestration tool; and keeps the Sidecar container and the main container running in synchronization through a heartbeat mechanism, and automatically restarts the Sidecar container when an exception occurs.

[0046] In some embodiments of the present disclosure, the device 200 can embed an API Hook in the main container, intercept application program calls through the Sidecar container, and obtain application data; the Sidecar container obtains log data about system behavior, user operation and application state by monitoring system operation logs, application logs and service logs; the Sidecar container analyzes visual data on the application program interface through OCR technology, and extracts text data from the visual data; and the Sidecar container captures user voice input in real time through voice collection technology, and converts the voice input into text data.

[0047] In some embodiments of the present disclosure, the device 200 can use a multi-modal large language model deployed on the user terminal to perform cross-modal data fusion analysis on the user behavior data; and feed back the analysis result to the Sidecar container, and adjust the data collection strategy through a real-time feedback mechanism when detecting that the user's behavior pattern has changed.

[0048] In embodiments of the present disclosure, the processor 210 can be, for example, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a processor based on a multi-core processor architecture, etc. The memory 220 can be any type of memory implemented using data storage technology, including but not limited to random access memory, read-only memory, semiconductor-based memory, flash memory, disk storage, etc.

[0049] In addition, in embodiments of the present disclosure, the device 200 can also include an input device 230, such as a keyboard, a mouse, etc. In addition, the device 200 can also include an output device 240, such as a display, etc.

[0050] In other embodiments of the present disclosure, a computer readable storage medium storing a computer program is also provided, wherein the computer program can implement the steps of the method as shown in Figure 1 when executed by a processor.

[0051] In summary, according to the Sidecar container-based multi-modal data collection method and device of the embodiments of the present disclosure, through containerization technology, each user instance runs in an independent container, which can ensure that the data of different users do not interfere with each other, enhances data isolation, and helps to improve security in a multi-user environment. By using end-to-end encryption technology, it ensures that user behavior data is encrypted and protected from the collection to the transmission process, which enhances the confidentiality and integrity of the data. Loading multiple data collection components such as API Hook, log monitoring, OCR, and voice analysis in the Sidecar container can simultaneously collect multiple types of user behavior data in real time. This multi-modal data collection capability enables comprehensive monitoring of various user interaction methods, thereby improving the comprehensiveness and accuracy of data analysis.

[0052] The flowcharts and block diagrams in the attached drawings show the architecture, function and operation of possible implementations of the apparatus and methods according to the embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks can also occur in different order from that noted in the attached drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0053] Unless the context clearly indicates otherwise, as used herein and in the appended claims, the singular form of a word includes the plural and vice versa. Thus, the use of the singular will include its meaning of the plural whenever appropriate. Also, the use of "comprise", "comprises" or "comprising" will be understood to encompass the meaning of "including", "includes" or "include" and vice versa. Similarly, the use of "or" will be understood to encompass the meaning of "and" and vice versa. Also, the use of "having" will be understood to encompass the meaning of "including" and vice versa. Also, the use of "associated with" will be understood to encompass the meaning of "includes" and vice versa. Also, the use of "one" will be understood to encompass the meaning of "at least one" and vice versa. Also, the use of "example" will be understood to encompass the meaning of "exemplary" and vice versa. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0054] Further aspects and ranges of adaptation become apparent from the description provided herein. It should be understood that various aspects of the application can be implemented alone or in combination with one or more other aspects. It should also be understood that the description and specific examples herein are intended to be illustrative only and are not intended to limit the scope of the present application.

[0055] The above detailed description of several embodiments of the disclosure has been presented for the purposes of illustration and description. It is apparent to those skilled in the art that various modifications and variations can be made to the embodiments of the disclosure without departing from the spirit and scope of the disclosure. The scope of protection of the disclosure is defined by the appended claims.

Claims

1. A method for multi-modal data acquisition based on Sidecar containers, characterized in that, The application relates to a cloud desktop behavior data collection and analysis method based on Kubernetes containerization technology. The cloud desktop is encapsulated by using Kubernetes containerization technology, so that each user instance runs in an independent container; A Sidecar container is deployed beside the cloud desktop main container, and the Sidecar container and the main container run in parallel; A data collection component is loaded in the Sidecar container, and various types of user behavior data are monitored and collected in real time through API Hook, log monitoring, OCR and voice analysis; End-to-end encryption technology is used to transmit the collected user behavior data to a local user terminal; A multi-modal large language model deployed on the user terminal is used to perform cross-modal data fusion analysis on the user behavior data, including: inputting the converted text data of voice input and the converted text data of visual data into a multi-modal large language model based on a Transformer architecture, capturing the semantic structure in the text through a self-attention mechanism, understanding the text content and generating personalized work suggestions; the features of log data and application data are fused by using a multi-modal large language model based on a Transformer architecture, the relationship between log and behavior patterns is learned, sensitive operations, abnormal behaviors and potential security risks are identified; and The analysis result is fed back to the Sidecar container, and when the user's behavior pattern changes, the data collection strategy is adjusted through a real-time feedback mechanism, including: modifying the frequency, type or granularity of data collection according to the change of the user behavior pattern; and adjusting the storage mode and processing mechanism of data according to the change of the data collection strategy.

2. The Sidecar container-based multi-modality data acquisition method of claim 1, wherein, The cloud desktop is encapsulated by using Kubernetes containerization technology, so that each user instance runs in an independent container; An operating system, a desktop environment, commonly used application programs and user configurations are packaged into a Docker image, and an independent user instance is generated based on the image after each user logs in; A Kubernetes resource configuration file is created for each user instance, and each configuration file defines a Pod to run each user instance; and Persistent storage is configured for each user instance to save user data.

3. The Sidecar container-based multi-modality data acquisition method of claim 1, wherein, The cloud desktop main container runs application programs, and the Sidecar container shares the network and storage system with the main container through container orchestration tools; and The Sidecar container and the main container are kept running in synchronization through a heartbeat mechanism, and the Sidecar container is automatically restarted when an exception occurs. An API Hook is embedded in the main container, and application data is obtained by intercepting application program calls through the Sidecar container; 4. The Sidecar container-based multi-modality data acquisition method of claim 1, wherein, The Sidecar container obtains log data about system behavior, user operation and application state by monitoring system operation logs, application logs and service logs; ​ ​ The Sidecar container parses visual data on an application interface through an OCR technique and extracts text data from the visual data; and The Sidecar container captures voice input of a user in real time through a voice collection technique and converts the voice input into text data.

5. The Sidecar container-based multi-modality data acquisition method of claim 4, wherein, The application data includes text input of a user in a document, a chat application or a code editor, the log data includes startup, shutdown, usage frequency, interaction mode of an application and read, modify, delete operation and operation track of a user on a file, the visual data includes visible text information on a user interface, and the voice input includes verbal instructions, conversations or other voice interactions of a user.

6. A Sidecar container-based multi-modal data acquisition apparatus, characterized in that, The apparatus comprises: at least one processor; and at least one memory having computer program stored therein; wherein the computer program, when executed by the at least one processor, causes the apparatus to perform the steps of the method according to any one of claims 1 to 5.

7. A computer readable storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 5.

Images