Multi-modal data acquisition method and device based on Sidecar container

Abstract

The embodiment of the invention provides a multi-modal data acquisition method and device based on a Sidecar container, and the method comprises the steps: packaging a cloud desktop through employing a Kubernetes containerization technology, and enabling each user instance to run in an independent container; the method comprises the following steps: deploying a Sidecar container beside a main container of a cloud desktop, wherein the Sidecar container and the main container run in parallel; the method comprises the following steps: loading a data acquisition component in a Sidecar container, and monitoring and acquiring various types of user behavior data in real time through API Hook, log monitoring, OCR (Optical Character Recognition) and voice analysis; and transmitting the acquired user behavior data to a local user terminal by adopting an end-to-end encryption technology.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of cloud desktops, and more specifically, to a multi-modal data collection method and device based on Sidecar containers. Background Art

[0002] A cloud desktop is a virtual desktop solution based on cloud computing technology. It provides desktop operating systems (such as Windows, Linux, etc.) and applications to users through a cloud server. Users can remotely access and use these desktop environments through terminal devices (such as computers, mobile phones, tablets, etc.). The core idea of a cloud desktop is to host computing resources, storage, and applications in the cloud. Users do not need to install and manage these resources on local devices, providing advantages such as high flexibility, scalability, and ease of management.

[0003] In a cloud desktop environment, user operations involve various data types, such as text input, operation logs, screen content, voice commands, etc. Traditional data collection methods usually rely on centralized monitoring or client plugins, and have problems such as high resource occupancy, large latency, and insufficient security. Summary of the Invention

[0004] To efficiently and non-intrusively monitor and collect cloud desktop user behaviors in real time, the embodiments described herein provide a multi-modal data collection method, device, and computer-readable storage medium storing a computer program based on Sidecar containers.

[0005] According to a first aspect of the present disclosure, there is provided a multi-modal data collection method based on Sidecar containers, including: encapsulating a cloud desktop using Kubernetes containerization technology so that each user instance runs in an independent container; deploying a Sidecar container beside the cloud desktop main container, with the Sidecar container and the main container running in parallel; loading a data collection component in the Sidecar container, and through API Hook, log monitoring, OCR, and voice parsing, real-time monitoring and collecting various types of user behavior data; and transmitting the collected user behavior data to a local user terminal using end-to-end encryption technology.

[0006] In some embodiments of the present disclosure, encapsulating a cloud desktop using Kubernetes containerization technology so that each user instance runs in an independent container includes: packaging an operating system, a desktop environment, common application programs, and user configurations into a Docker image, and generating an independent user instance based on this image after each user logs in; creating a Kubernetes resource configuration file for each user instance, with each configuration file defining a Pod to run each user instance; and configuring persistent storage for each user instance to save user data.

[0007] In some embodiments of the present disclosure, a Sidecar container is deployed beside the cloud desktop main container. The parallel operation of the Sidecar container and the main container includes: running an application in the cloud desktop main container, and the Sidecar container is bound to the main container through a container orchestration tool to share the network and storage system; and maintaining the synchronous operation of the Sidecar container and the main container through a heartbeat mechanism, and automatically restarting the Sidecar container in case of an exception.

[0008] In some embodiments of the present disclosure, a data collection component is loaded in the Sidecar container. Through API Hook, log monitoring, OCR, and voice parsing, various types of user behavior data are monitored and collected in real time, including: embedding an APIHook in the main container, intercepting application calls through the Sidecar container to obtain application data; the Sidecar container obtains log data on system behavior, user operations, and application status by monitoring system operation logs, application logs, and service logs; the Sidecar container parses the visual data on the application interface through OCR technology and extracts text data from the visual data; and the Sidecar container captures the user's voice input in real time through voice collection technology and converts the voice input into text data.

[0009] In some embodiments of the present disclosure, the application data includes the user's text input in documents, chat applications, or code editors. The log data includes the startup, shutdown, usage frequency, interaction mode of the application, and the user's read, modify, delete operations on files and operation traces. The visual data includes the visible text information on the user interface. The voice input includes the user's oral instructions, conversations, or other voice interactions.

[0010] In some embodiments of the present disclosure, the method further includes: performing cross-modal data fusion analysis on the user behavior data by using a multi-modal large language model deployed on the user terminal; and feeding back the analysis result to the Sidecar container, and adjusting the data collection strategy through a real-time feedback mechanism when it is detected that the user's behavior pattern has changed.

[0011] In some embodiments of the present disclosure, cross-modal data fusion analysis of user behavior data using a multi-modal large language model deployed on a user terminal includes: inputting the text data after voice input conversion and the text data after visual data conversion into a multi-modal large language model based on the Transformer architecture, capturing the semantic structure in the text through the self-attention mechanism, understanding the text content, and generating personalized work suggestions; and fusing the features of log data and application data using the multi-modal large language model based on the Transformer architecture, learning the relationship between log and behavior patterns, and identifying sensitive operations, abnormal behaviors, and potential security risks.

[0012] In some embodiments of the present disclosure, feedback the analysis result to the Sidecar container, and when it is detected that the user's behavior pattern changes, adjust the data collection strategy through a real-time feedback mechanism, including: modifying the frequency, type, or granularity of data collection according to the change in the user's behavior pattern; and adjusting the storage method and processing mechanism of the data according to the change in the data collection strategy.

[0013] According to the second aspect of the present disclosure, there is provided a multi-modal data collection device based on a Sidecar container. The device includes at least one processor; and at least one memory storing a computer program. When the computer program is executed by the at least one processor, the device: encapsulates the cloud desktop using Kubernetes containerization technology so that each user instance runs in an independent container; deploys a Sidecar container beside the main container of the cloud desktop, and the Sidecar container and the main container run in parallel; loads a data collection component in the Sidecar container, and monitors and collects various types of user behavior data in real time through API Hook, log monitoring, OCR, and voice parsing; and transmits the collected user behavior data to the local user terminal using end-to-end encryption technology.

[0014] According to the third aspect of the present disclosure, there is provided a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method according to the first aspect of the present disclosure.

[0015] The multi-modal data collection method and device based on Sidecar containers according to embodiments of the present disclosure, through containerization technology, each user instance runs in an independent container, which can ensure that the data of different users do not interfere with each other, enhance data isolation, and contribute to improving security in a multi-user environment. By adopting end-to-end encryption technology, it is ensured that user behavior data is encrypted and protected during the process from collection to transmission, which enhances the confidentiality and integrity of the data. Loading various data collection components in the Sidecar container, such as API Hook, log monitoring, OCR, and voice parsing, can simultaneously collect various types of user behavior data in real time. This multi-modal data collection ability enables comprehensive monitoring of various interaction methods of users, thereby improving the comprehensiveness and accuracy of data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] To briefly describe the technical solutions of the embodiments of the present disclosure more clearly, the accompanying drawings of the embodiments will be briefly described below. It should be understood that the following-described drawings only relate to some embodiments of the present disclosure and do not limit the present disclosure, where: Figure 1 is an exemplary flowchart of a multi-modal data collection method based on Sidecar containers according to embodiments of the present disclosure; Figure 2 is a schematic block diagram of a multi-modal data collection device based on Sidecar containers according to embodiments of the present disclosure.

[0017] It should be noted that the elements in the drawings are schematic and not drawn to scale. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are some, but not all, of the embodiments of the present disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of the present disclosure without creative efforts also belong to the scope of protection of the present disclosure.

[0019] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which the subject matter of the present disclosure belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the specification and the related art, and will not be interpreted in an idealized or overly formal form unless otherwise clearly defined herein. Additionally, terms such as "first" and "second" are only used to distinguish one component (or a part of a component) from another component (or another part of a component).

[0020] A cloud desktop itself refers to a remote desktop service that provides a desktop operating system and applications to users through virtualization technology. Containerization, on the other hand, packages an application and its dependencies into independent containers to achieve lightweight and convenient deployment and management. The containerized cloud desktop combines the advantages of both, using container technology to manage and deploy desktop instances, making each user's desktop environment more efficient, flexible, and capable of scaling resources on demand. The embodiments of the present disclosure aim to achieve efficient management of the cloud desktop environment through containerization technology, and at the same time, by collecting and analyzing the behaviors and data of users in the cloud desktop, understand the needs, habits, and behavior patterns of each user, so as to achieve personalized services.

[0021] Figure 1 An exemplary flowchart showing a multi-modal data collection method based on Sidecar containers according to an embodiment of the present disclosure is shown. At Figure 1 In block S102 of the method 100 shown, the cloud desktop is encapsulated using Kubernetes containerization technology, and each user instance runs in an independent container.

[0022] According to an embodiment of the present disclosure, the operating system, desktop environment, common applications, and user configurations are packaged into a Docker image, and an independent user instance is generated based on this image after each user logs in; a Kubernetes resource configuration file is created for each user instance, and each configuration file defines a Pod to run each user instance; persistent storage is configured for each user instance to save user data.

[0023] As a container orchestration platform, Kubernetes can automate tasks such as container deployment, scaling, scheduling, and load balancing. By using Docker container technology to provide a completely isolated running environment for each user, it can ensure that users do not interfere with each other or leak data. Each container instance is defined using Kubernetes' Pod, and each user's desktop environment runs in an independent Pod, which contains one or more containers. User data (such as files, application configurations, etc.) can be managed through the persistent storage mechanism provided by Kubernetes. This architecture can not only improve resource utilization but also effectively support large-scale, multi-user cloud desktop environments.

[0024] In block S104, a Sidecar container is deployed beside the cloud desktop main container, and the Sidecar container runs in parallel with the main container.

[0025] In Kubernetes, the main container is responsible for running the basic services and applications of the cloud desktop, such as the operating system, desktop applications, UI, and configurations. The Sidecar container is a container that runs in parallel with the main container and shares the same Pod with the main container. Therefore, it can share resources such as the same network namespace and storage system. The Sidecar container can be scaled or replaced as needed without affecting the applications in the main container, thereby improving the decoupling and maintainability of the system.

[0026] For example, the Sidecar container can monitor the user's operation behaviors, inputs, system resource usage, etc., collect real-time data for analysis, record the user's operation logs and error logs, and monitor the health status of the cloud desktop. Manage the communication with the external network through the Sidecar container to ensure the security of network access and traffic management. The Sidecar container can also act as a proxy, responsible for routing requests or providing specific services. For example, run the Nginx reverse proxy through the Sidecar container to provide the load balancing function for the main container.

[0027] To ensure the synchronous operation of the main container and the Sidecar container, a heartbeat mechanism can be used to detect the health status of the container and restart it when necessary. The heartbeat mechanism can detect whether the container is alive through the Liveness Probe. If the container does not respond to the heartbeat, it will be restarted. Or, detect whether the container is ready to receive traffic through the Readiness Probe. When the container is not ready, Kubernetes will forward the traffic to other healthy containers.

[0028] At Figure 1 box S106, load the data collection component in the Sidecar container, and monitor and collect various types of user behavior data in real time through API Hook, log monitoring, OCR, and voice parsing.

[0029] When the user is using the operating system or application, various types of data are generated, including: the user's text input behavior on the keyboard or touch device, which can reflect the user's thinking process and usage habits when interacting with the system. The user's operation behaviors, including mouse clicks, keyboard taps, application switching, menu selection, etc., and these logs can reflect the user's operation patterns and habits. The real-time capture or screenshot of the content displayed on the user's screen, which helps to obtain the user's current working status, interaction content, or the applications in use. The information input by the user through the voice assistant or voice recognition system, and the voice data not only contains the voice text, but also can analyze the user's voice emotions, intonations and other characteristics. Load the data collection component in the Sidecar container.

[0030] In some embodiments of the present disclosure, an API Hook is embedded in the main container, and the Sidecar container intercepts application calls to obtain application data. The application data includes text inputs of users in documents, chat applications, or code editors, etc. By intercepting the text data interaction in the application, the text input by the user in places such as document editors, chat applications, and development environments can be obtained.

[0031] The Sidecar container obtains log data regarding system behavior, user operations, and application status by monitoring system operation logs, application logs, and service logs. The log data includes the startup, shutdown, usage frequency, interaction mode of the application, as well as the operations (opening, saving, deleting, editing) and operation traces of the user on files. The logs can also include data transfer behaviors, such as file downloads, uploads, or sharing. Through the analysis of the logs, the usage patterns and requirements of users for data can be obtained.

[0032] The Sidecar container parses the visual data on the application interface through OCR technology and extracts text data from the visual data. The visual data includes visible text information on the user interface, such as button labels, menu options, dialog box contents, etc., which is particularly important for text information that cannot be directly obtained through the API. The Sidecar container captures the user's voice input in real time through voice collection technology and converts the voice input into text data. The voice input includes the user's oral instructions, conversations, or other voice interactions, and the transcribing of the voice content provides data support for subsequent semantic understanding and task summary.

[0033] To ensure the real-time performance and efficiency of multiple data collection methods, a message queue (such as Kafka, RabbitMQ, or NATS) can be used to asynchronously transmit the collected data to the backend processing system. Each data collection method (API Hook, log monitoring, OCR, voice parsing) can send data through the message queue to ensure real-time data transmission and avoid data loss. The collected data can be stored in a distributed database (such as MongoDB, Cassandra) or a search engine (such as Elasticsearch) for subsequent data analysis and query.

[0034] Finally, in block S108, the collected user behavior data is transmitted to the local user terminal using end-to-end encryption technology.

[0035] By performing real-time analysis and classification on the sensitivity of the collected data, automatically screen and encrypt data involving sensitive information, such as financial data, personal information, etc., to ensure the privacy and security of the data during analysis. Automatically de-identify sensitive information and adopt end-to-end encryption technology to ensure the security of the data during cloud storage and transmission. Users can choose to enable the transparency mode to view and manage the data they generate at any time. When the encrypted data reaches the local user terminal, first decrypt the session key through asymmetric encryption (using the private key of the receiving end to decrypt). Once the session key is decrypted, the user terminal uses the session key to decrypt the encrypted data to restore the original user behavior data, and the decrypted data can be used for further analysis on the local user terminal.

[0036] In some embodiments of the present disclosure, a multi-modal large language model deployed on the user terminal is used to perform cross-modal data fusion analysis on the user behavior data.

[0037] The multi-modal large language model (such as a Transformer-based model) can process various different types of data sources, such as text, voice, and visual information. Specifically, the text data after converting the voice input and the text data after converting the visual data are input into the multi-modal large language model based on the Transformer architecture, and the self-attention mechanism is used to capture the semantic structure in the text, understand the text content, and generate personalized work suggestions.

[0038] In addition to improving personal productivity, it is also possible to analyze the communication, file sharing, collaborative workflows, etc. among team members, give suggestions for improving collaboration efficiency, and help the team better coordinate and communicate. Support seamless collaboration between different platforms, allow users to continuously process work tasks on different devices, and maintain the real-time nature and consistency of the data through containerization and cloud synchronization.

[0039] Utilize the multi-modal large language model based on the Transformer architecture to fuse the characteristics of log data and application data, learn the relationship between log and behavior patterns, and identify sensitive operations, abnormal behaviors, and potential security risks. For example, operations such as users accessing confidential files and modifying system settings can be marked as sensitive operations. After identifying sensitive operations or abnormal behaviors, the system can automatically trigger response measures according to preset rules, such as warning users, locking accounts, or suspending certain services, etc.

[0040] Analyze the user's behavior patterns through algorithms, such as click frequency, access path, input content, etc. within a certain period of time. If a significant change in the behavior pattern is detected (for example, the user's click frequency suddenly increases, the usage method changes, etc.), the system can automatically identify this change and generate a feedback signal. The user terminal feeds back the analysis result to the Sidecar container. When a change in the user's behavior pattern is detected, the data collection strategy is adjusted through a real-time feedback mechanism. The feedback can be sent from the terminal to the Sidecar container through methods such as message queues, WebSocket, or RESTful APIs. The data collection strategy inside the container receives these feedbacks and makes adjustments as needed. For example, modify the frequency, type, or granularity of data collection; and, according to the change in the data collection strategy, adjust the data storage method and processing mechanism.

[0041] For example, if an important change in user behavior is detected, the container can decide to increase the data collection frequency to more accurately track the user's changes. If certain behavior patterns change (for example, the user starts to interact frequently with specific interface elements), the types of data collected can be adjusted according to the feedback results, such as increasing the collection of certain specific fields or operations. If the user's behavior pattern becomes more stable or there is no significant change, the system can reduce the collection frequency or stop collecting some low-priority data according to the needs to save resources. If the user is mainly active in aspects such as document editing, writing, and report generation, the system can preferentially collect data such as text input, document access frequency, and keyword usage. If the user is mainly active in aspects such as code development, debugging, and version control, the system can strengthen the collection of data such as code input, programming language usage, and development tool usage frequency.

[0042] Figure 2 is a schematic block diagram of a multi-modal data collection device based on a Sidecar container according to an embodiment of the present disclosure. As Figure 2 shown, the device 200 may include a processor 210 and a memory 220 storing a computer program. When the computer program is executed by the processor 210, the device 200 can execute the steps of the method 100 as Figure 1 shown. In one example, the device 200 can be a computer device or a cloud computing node. The device 200 can encapsulate the cloud desktop using Kubernetes containerization technology, enabling each user instance to run in an independent container; deploy a Sidecar container beside the main container of the cloud desktop, and the Sidecar container and the main container run in parallel; load a data collection component in the Sidecar container, and through API Hook, log monitoring, OCR, and voice parsing, monitor and collect various types of user behavior data in real time; use end-to-end encryption technology to transmit the collected user behavior data to the local user terminal.

[0043] In some embodiments of the present disclosure, the device 200 may run an application in the main container of the cloud desktop. The Sidecar container is bound to the main container through a container orchestration tool to share the network and storage system; and the Sidecar container and the main container are kept running synchronously through a heartbeat mechanism, and the Sidecar container is automatically restarted in case of an exception.

[0044] In some embodiments of the present disclosure, the device 200 may embed an API Hook in the main container to intercept application calls through the Sidecar container and obtain application data; the Sidecar container obtains log data on system behavior, user operations, and application status by monitoring system operation logs, application logs, and service logs; the Sidecar container parses visual data on the application interface through OCR technology and extracts text data from the visual data; and the Sidecar container captures the user's voice input in real time through voice capture technology and converts the voice input into text data.

[0045] In some embodiments of the present disclosure, the device 200 may perform cross-modal data fusion analysis on user behavior data by using a multi-modal large language model deployed on the user terminal; and feedback the analysis result to the Sidecar container, and adjust the data collection strategy through a real-time feedback mechanism when it is detected that the user's behavior pattern has changed.

[0046] In an embodiment of the present disclosure, the processor 210 may be, for example, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a processor based on a multi-core processor architecture, etc. The memory 220 may be any type of memory implemented using data storage technology, including but not limited to random access memory, read-only memory, semiconductor-based memory, flash memory, disk memory, etc.

[0047] In addition, in an embodiment of the present disclosure, the device 200 may also include an input device 230, such as a keyboard, a mouse, etc. Additionally, the device 200 may further include an output device 240, such as a display, etc.

[0048] In other embodiments of the present disclosure, a computer-readable storage medium storing a computer program is also provided, wherein the computer program, when executed by a processor, can implement the steps of the method as Figure 1 shown.

[0049] In summary, according to the multi-modal data acquisition method and device based on Sidecar containers of the embodiments of the present disclosure, through containerization technology, each user instance runs in an independent container, which can ensure that the data of different users do not interfere with each other, enhance data isolation, and contribute to improving security in a multi-user environment. By adopting end-to-end encryption technology, it is ensured that the user behavior data is encrypted and protected during the process from collection to transmission, which enhances the confidentiality and integrity of the data. Loading various data acquisition components in the Sidecar container, such as API Hook, log monitoring, OCR, and voice parsing, can simultaneously collect various types of user behavior data in real time. This multi-modal data acquisition capability enables comprehensive monitoring of various interaction methods of users, thereby improving the comprehensiveness and accuracy of data analysis.

[0050] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the devices and methods according to multiple embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of an instruction, and the module, the segment of a program, or the part of an instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0051] Unless the context clearly indicates otherwise, the singular forms of the words used in this specification and the appended claims include the plural, and vice versa. Thus, when referring to the singular, it generally includes the plural of the corresponding term. Similarly, the terms "comprising" and "including" shall be construed as inclusive rather than exclusive. Likewise, the term "including" and "or" shall be construed as inclusive, unless such construction is clearly prohibited herein. Where the term "example" is used in this specification, particularly when it is placed after a group of terms, the "example" is merely exemplary and illustrative, and should not be considered exclusive or extensive.

[0052] Further aspects and scopes of adaptability become apparent from the description provided herein. It should be understood that the various aspects of the present application may be implemented alone or in combination with one or more other aspects. It should also be understood that the description herein and the specific embodiments are for illustrative purposes only and are not intended to limit the scope of the present application.

[0053] The above has described several embodiments of the present disclosure in detail. However, it is obvious that those skilled in the art can make various modifications and variations to the embodiments of the present disclosure without departing from the spirit and scope of the present disclosure. The protection scope of the present disclosure is defined by the appended claims.

Claims

1. A multi-modal data acquisition method based on Sidecar containers, characterized in that, Including: Encapsulating the cloud desktop using Kubernetes containerization technology so that each user instance runs in an independent container; Deploying a Sidecar container beside the main container of the cloud desktop, and the Sidecar container runs in parallel with the main container; Loading a data collection component in the Sidecar container, and through API Hook, log monitoring, OCR, and voice parsing, real-time monitoring and collecting various types of user behavior data; And Using end-to-end encryption technology to transmit the collected user behavior data to the local user terminal.

2. The multi-modal data acquisition method based on Sidecar containers according to claim 1, wherein, The encapsulating the cloud desktop using Kubernetes containerization technology so that each user instance runs in an independent container includes: Packaging the operating system, desktop environment, common applications, and user configurations into a Docker image, and generating an independent user instance based on this image after each user logs in; Creating a Kubernetes resource configuration file for each user instance, and each configuration file defines a Pod to run each user instance; and Configuring persistent storage for each user instance to save user data.

3. The multimodal data acquisition method based on Sidecar containers according to claim 1, wherein, The deploying a Sidecar container beside the main container of the cloud desktop, and the Sidecar container runs in parallel with the main container includes: Running an application in the main container of the cloud desktop, and the Sidecar container is bound to the main container through a container orchestration tool to share the network and storage system; and Keeping the Sidecar container and the main container running synchronously through a heartbeat mechanism, and automatically restarting the Sidecar container in case of an exception.

4. The multi-modal data acquisition method based on Sidecar containers according to claim 1, wherein, The loading a data collection component in the Sidecar container, and through API Hook, log monitoring, OCR, and voice parsing, real-time monitoring and collecting various types of user behavior data includes: Embedding an API Hook in the main container, and intercepting application calls through the Sidecar container to obtain application data; The Sidecar container obtains log data on system behavior, user operations, and application status by monitoring system operation logs, application logs, and service logs; The Sidecar container parses visual data on the application interface through OCR technology and extracts text data from the visual data; and The Sidecar container captures the user's voice input in real time through voice collection technology and converts the voice input into text data.

5. The multi-modal data acquisition method based on Sidecar containers according to claim 4, wherein The application data includes the user's text input in documents, chat applications, or code editors, the log data includes the startup, shutdown, usage frequency, interaction mode of the application, and the user's read, modify, delete operations on files and operation traces, the visual data includes visible text information on the user interface, and the voice input includes the user's oral instructions, conversations, or other voice interactions.

6. The multimodal data acquisition method based on Sidecar containers according to claim 1, characterized in that, The method further includes: Performing cross-modal data fusion analysis on the user behavior data using a multi-modal large language model deployed on the user terminal; and Feeding back the analysis results to the Sidecar container, and when it is detected that the user's behavior pattern changes, adjusting the data collection strategy through a real-time feedback mechanism.

7. The multimodal data acquisition method based on Sidecar containers according to claim 6, wherein The cross-modal data fusion analysis of the user behavior data by using the multi-modal large language model deployed on the user terminal includes: Inputting the text data converted from the voice input and the text data converted from the visual data into the multi-modal large language model based on the Transformer architecture, capturing the semantic structure in the text through the self-attention mechanism, understanding the text content, and generating personalized work suggestions; and Fusing the features of the log data and the application data by using the multi-modal large language model based on the Transformer architecture, learning the relationship between the log and the behavior pattern, and identifying sensitive operations, abnormal behaviors, and potential security risks.

8. The multi-modal data acquisition method based on Sidecar containers according to claim 6, characterized in that The feedback of the analysis result to the Sidecar container and the adjustment of the data collection strategy through the real-time feedback mechanism when it is detected that the user's behavior pattern changes include: Modifying the frequency, type, or granularity of data collection according to the change of the user behavior pattern; and Adjusting the storage method and processing mechanism of the data according to the change of the data collection strategy.

9. A multi-modal data acquisition device based on Sidecar containers, characterized in that, The device includes: At least one processor; and At least one memory storing a computer program; Wherein, when the computer program is executed by the at least one processor, the device executes the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images