A data management method and system based on an instant messaging platform

Abstract

The application provides a data management method and system based on an instant messaging platform, and relates to the technical field of data processing.The method comprises the following steps: step 1, using a micro-service architecture, distributing the function services of the instant messaging device in a container environment, and realizing dynamic registration and discovery of nodes through a service management component to build a communication basic platform; step 2, based on the communication basic platform, establishing bidirectional communication through a web socket protocol, and performing version checking and integrity checking on the structured messages exchanged between the client and the server to generate verified message packages; and step 3, based on the verified message packages, building a layered storage system, writing communication data into a distributed relational database in real time, and storing attachments through a distributed file device to realize unified storage of structured and unstructured data. The application realizes dynamic expansion through the micro-service architecture and containerized deployment, improves the retrieval efficiency, and realizes data full-process traceability.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data management method and system based on an instant messaging platform. Background Art

[0002] In the context of enterprise instant messaging, traditional instant messaging platform data management technologies have limitations. For example, at a large enterprise, traditional data management technologies lack a unified architecture for data storage. Different departments may use different instant messaging tools. The sales department's tool stores message logs and files on local servers, while the R&D department's tool relies on third-party cloud storage. This results in decentralized data storage within the enterprise, making unified management and analysis difficult. When the company needs to review all communication records and files for a project, it must search across multiple storage locations, consuming significant time and labor costs.

[0003] Furthermore, traditional instant messaging platforms have limited data retrieval capabilities. Companies generate massive amounts of instant messaging data daily. For example, if an employee wants to find specific details about a product design proposal discussed in a project last year, traditional technology may only support simple searches based on time or keywords. If the employee forgets the exact time and keywords, it can be difficult to quickly locate the relevant information. Furthermore, traditional technology cannot efficiently link file attachments to message history. If an employee only remembers the file name, they struggle to find the relevant communication context, severely impacting work efficiency and information accuracy. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a data management method and system based on an instant messaging platform, which realizes efficient storage of instant messaging data through distributed deployment of a microservice architecture and improves the efficiency and flexibility of data management.

[0005] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0006] In a first aspect, a data management method based on an instant messaging platform is provided, the method comprising:

[0007] Step 1: Using a microservices architecture, the functional services of the instant messaging device are distributed and deployed in a container environment. Dynamic node registration and discovery are implemented through service governance components to build a communication infrastructure platform.

[0008] Step 2: Based on the communication infrastructure platform, establish two-way communication through the web socket protocol, perform version verification and integrity verification on the structured messages exchanged between the client and the server, and generate a verified message package;

[0009] Step 3: Based on the verified message packets, a hierarchical storage system is constructed to write the communication data into a distributed relational database in real time. Attachments are stored in a distributed file device, achieving unified storage of structured and unstructured data.

[0010] Step 4: Based on the uniformly stored data, a multi-dimensional indexing mechanism is established. Elasticsearch is used to create a composite index for the metadata of messages and files, generating indexable data that can be searched based on multiple conditions such as time, user, session, keyword, and file type.

[0011] Step 5: Implement fine-grained permission management based on indexable data, set permission levels through role-based access control, and generate permission-controlled data;

[0012] Step 6: Based on the permission-controlled data, execute the periodic data archiving strategy, automatically migrate the expired data to the cloud storage device, and set three fixed detection points in sequence during the archiving process. Generate dynamic verification values ​​based on the feedback results of each detection point. Based on the dynamic verification values, adjust the microservice architecture deployment parameters to achieve data traceability.

[0013] Furthermore, a microservices architecture is adopted to distribute the functional services of instant messaging devices in a container environment, and dynamic node registration and discovery are achieved through service governance components to build a basic communication platform, including:

[0014] Based on the functional requirements of instant messaging devices, the functions are divided into independent microservice units, including user authentication service, message routing service, user status management service and contact management service;

[0015] Encapsulate microservice units in independent container images and deploy them in a distributed container cluster environment using container orchestration tools;

[0016] Deploy service governance components in a container cluster environment, including a service registry and service discovery mechanism;

[0017] Each microservice unit automatically registers its node with the service registration center when it starts, reporting the service identifier, network address, health status information, and past call frequency and response time data;

[0018] The service discovery mechanism dynamically monitors the status changes of microservice nodes in the service registration center, calculates the load trend of the reported call frequency and response time through a time series prediction algorithm, and generates future load forecast values;

[0019] Based on the future load forecast, an API gateway is deployed at the entrance of the container cluster environment to perform unified routing, load balancing, and access control on the service access endpoints provided by the microservice units.

[0020] A communication infrastructure platform is built through the distributed deployment of microservice units, dynamic node registration and discovery of service governance components, and unified access management of API gateways.

[0021] Furthermore, based on the communication infrastructure platform, two-way communication is established through the web socket protocol, and version verification and integrity verification are performed on the structured messages exchanged between the client and the server to generate a verified message package, including:

[0022] Based on the API gateway unified access point provided by the communication infrastructure platform, a persistent two-way communication link is established between the client and the API gateway via the web socket protocol;

[0023] Receive structured messages sent by the client and response messages returned by the server through a persistent two-way communication link;

[0024] At the API gateway layer, a version identifier and a digital signature generated based on the message content are attached to each structured message of the interaction;

[0025] In the server-side message processing component, the version identifier is extracted and compared with the preset valid version range to generate a version verification result;

[0026] Based on the version verification result, the digital signature is decrypted using the verification key corresponding to the signature algorithm to generate a message digest hash value. At the same time, a comparison hash value is calculated in real time based on the current message content. When the message digest hash value matches the comparison hash value calculated in real time, an abnormality confidence calculation is performed. If the confidence level is lower than the threshold, the message is determined to be complete and has not been tampered with.

[0027] The structured message that has passed the dual verification of version verification and integrity verification is generated into a verified message package carrying a verification status mark.

[0028] Furthermore, based on the verified message packets, a hierarchical storage system is built to write communication data into a distributed relational database in real time. Attachments are stored through distributed file devices, achieving unified storage of structured and unstructured data, including:

[0029] Write the text content, timestamp field, and user relationship field in the verified message package into the distributed relational database shard to generate a message record including a primary key;

[0030] Based on the message record, extract the file attachment from the corresponding verified message package and upload the file attachment to the object storage service to generate a unique storage path;

[0031] Bind the message record primary key to the storage path, create a primary key-path mapping table in the relational database, and form a unified storage structure in which text data and file attachments can be associated and accessed.

[0032] Furthermore, based on the uniformly stored data, a multi-dimensional indexing mechanism is established. Elastic search is used to create composite indexes for message and file metadata, generating indexable data that can be searched based on multiple conditions such as time, user, session, keyword, and file type. This includes:

[0033] Extract the message sending time field, user ID field, session ID field, keyword tag, and file media type field from the unified storage to form a structured metadata set;

[0034] Based on the structured metadata collection, build a composite index structure in the Elasticsearch cluster, including time range filters, user session aggregators, and keyword analyzers;

[0035] Continuously collect user query logs, dynamically adjust index field weights based on query frequency statistics, and generate pre-calculated index tables for high-frequency combined queries;

[0036] By using a composite index structure and a pre-calculated index table, an index data structure is generated for performing multi-field combined queries on the message sending time field, user ID field, session ID field, keyword tag, and file media type field, thereby indexing the data.

[0037] Furthermore, based on indexable data, fine-grained permission management is implemented. Through role-based access control, permission levels are set to generate permission-controlled data, including:

[0038] In the data management device, a role-based access control role set is defined, including an administrator role, a department head role, and a common member role, and role definition information is generated;

[0039] Based on the role definition information, an access control role identifier that allows access to the data entity is associated with each data entity in the indexable data to form a role access tag, so as to generate a set of indexable data entities with role access tags;

[0040] Deploy a permission interception function at the query interface layer and input a collection of indexable data entities with role access tags into the permission interception function. When a user initiates a data query request through the query interface layer, the permission interception function identifies the current user's access control role and analyzes the user behavior log in real time to calculate the permission deviation degree.

[0041] The permission interception function filters the data entities that meet the query conditions in the indexable data entity set with role access tags based on the identified current user access control role, returns the data entities retained after filtering as query results, and generates data controlled by permissions.

[0042] Furthermore, based on the data under permission control, a periodic data archiving strategy is implemented to automatically migrate overdue data to cloud storage devices. During the archiving process, three fixed checkpoints are set in sequence. Dynamic check values ​​are generated based on the feedback results of each checkpoint. Based on the dynamic check values, the deployment parameters of the microservice architecture are adjusted to achieve data traceability, including:

[0043] Based on the predefined lifecycle configuration table of permission-controlled data, the data archiving task is automatically initiated when the preset archiving trigger conditions are detected.

[0044] In the data archiving task execution process, checkpoints are preset for three key operation stages, including the data extraction stage checkpoint, which is set after the permission-controlled data extraction operation is completed from the source storage device; the encrypted transmission stage checkpoint, which is set after the data encryption is completed and the transmission operation to the cloud is started; and the cloud storage stage checkpoint, which is set after the data is successfully written to the cloud persistent storage device.

[0045] During the execution of the data archiving task, when each set checkpoint is reached, the corresponding check operation is performed and the check parameters are generated, including the compression ratio deviation value, the transmission hash value and the number of storage copies;

[0046] The compression ratio deviation, transmission hash value, and number of stored copies are used to form a dynamic check matrix that reflects the multi-stage integrity of the archiving task. The dynamic check matrix is ​​analyzed and matrix anomaly indicators reflecting the potential risks of the archiving process are calculated according to predefined rules.

[0047] Based on the value of the matrix anomaly indicator, the number of running instances of the log audit service responsible for recording data operation details is dynamically increased, and data traceability is achieved through the increased running instances of the log audit service.

[0048] In a second aspect, a data management system based on an instant messaging platform includes:

[0049] The architecture deployment module is used to build a communication infrastructure platform to achieve distributed deployment of instant messaging device functional services and dynamic node registration and discovery;

[0050] A communication verification module is used to establish two-way communication based on the communication infrastructure platform through the web socket protocol and perform version and integrity verification on structured messages to generate verified message packets;

[0051] The data storage module is used to build a hierarchical storage system based on verified message packets, write communication data into a distributed relational database in real time and store attachments through distributed file devices, realizing unified storage of structured and unstructured data;

[0052] An index building module, which is used to create composite indexes for message and file metadata based on uniformly stored data, generating indexable data that can be searched based on multiple conditions such as time, user, session, keyword, and file type.

[0053] The permission control module is used to implement fine-grained permission management based on indexable data, set permission levels through role-based access control, and generate permission-controlled data;

[0054] The data archiving module is used to execute periodic data archiving strategies based on permission-controlled data, migrate expired data to cloud storage, and generate dynamic checksums based on feedback from checkpoints during the archiving process to adjust microservice architecture deployment parameters and achieve data traceability.

[0055] According to a third aspect, a computing device includes:

[0056] one or more processors;

[0057] The storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method.

[0058] In a fourth aspect, a computer-readable storage medium stores a program, which implements the method when executed by a processor.

[0059] The above solution of the present invention includes at least the following beneficial effects:

[0060] Using a microservices architecture, the instant messaging device functionality is distributed across a container environment. Dynamic node registration and discovery are enabled through a service governance component, automatically adjusting resource allocation based on real-time load, effectively handling high-concurrency scenarios. Furthermore, dynamic monitoring and automatic repair mechanisms for service nodes enhance stability and availability, reducing the risk of single points of failure. Bidirectional communication is established using the Web Sockets protocol, and version and integrity checks are performed on structured messages. This mechanism ensures the accuracy and integrity of messages exchanged between the client and server, effectively preventing tampering and version incompatibilities during transmission. This provides a solid security guarantee for instant messaging data exchange and enhances user trust in data transmission. A tiered storage system is constructed, writing communication data to a distributed relational database in real time, while storing attachments through a distributed file system, achieving unified storage of structured and unstructured data. This storage approach not only leverages the advantages of relational databases in managing structured data and the efficiency of distributed file systems in handling massive amounts of unstructured data, but also establishes data associations to enable unified management and rapid access to both types of data, improving data storage and retrieval efficiency.

[0061] A multidimensional indexing mechanism was established, using ElasticSearch to create composite indexes for message and file metadata. Users can quickly search based on a combination of criteria, such as time, user, session, keyword, and file type, shortening data lookup time and improving information retrieval efficiency. The composite index structure also effectively reduces the data scan scope during searches, reducing resource consumption and improving overall search performance. Role-based access control was implemented to achieve fine-grained permission management. Users with different roles can only access data within their permission scope, effectively ensuring data security and privacy and preventing the leakage of sensitive information. Furthermore, this permission management approach allows enterprises or organizations to flexibly configure permissions based on the needs of different roles, enhancing the flexibility and adaptability of data management. A periodic data archiving strategy was implemented, and dynamic checksums were generated at checkpoints during the archiving process to adjust the deployment parameters of the microservice architecture. This mechanism automated data management and reduced manual operation and maintenance costs. Dynamic checksums were used to monitor and adjust the archiving process, ensuring the integrity and accuracy of the archived data. Furthermore, the addition of log auditing service instances enhanced data traceability. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a flow chart of a data management method based on an instant messaging platform provided by an embodiment of the present invention.

[0063] Figure 2 This is a schematic diagram of a data management system based on an instant messaging platform provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0065] like Figure 1 As shown, an embodiment of the present invention provides a data management method based on an instant messaging platform, the method comprising the following steps:

[0066] Step 1: Using a microservices architecture, the functional services of the instant messaging device are distributed and deployed in a container environment. Dynamic node registration and discovery are implemented through service governance components to build a communication infrastructure platform.

[0067] Step 2: Based on the communication infrastructure platform, establish two-way communication through the web socket protocol, perform version verification and integrity verification on the structured messages exchanged between the client and the server, and generate a verified message package;

[0068] Step 3: Based on the verified message packets, a hierarchical storage system is constructed to write the communication data into a distributed relational database in real time. Attachments are stored in a distributed file device, achieving unified storage of structured and unstructured data.

[0069] Step 4: Based on the uniformly stored data, a multi-dimensional indexing mechanism is established. Elasticsearch is used to create a composite index for the metadata of messages and files, generating indexable data that can be searched based on multiple conditions such as time, user, session, keyword, and file type.

[0070] Step 5: Implement fine-grained permission management based on indexable data, set permission levels through role-based access control, and generate permission-controlled data;

[0071] Step 6: Based on the permission-controlled data, execute the periodic data archiving strategy, automatically migrate the expired data to the cloud storage device, and set three fixed detection points in sequence during the archiving process. Generate dynamic verification values ​​based on the feedback results of each detection point. Based on the dynamic verification values, adjust the microservice architecture deployment parameters to achieve data traceability.

[0072] In an embodiment of the present invention, a microservices architecture is used to split functions into independent units and deploy them in a container environment. Dynamic node registration and discovery are achieved in conjunction with service governance components, allowing for automatic increase and decrease of service instances based on load, flexibly addressing high-concurrency scenarios. A service discovery mechanism monitors node status in real time, automatically isolates faulty nodes, and ensures continuous service availability, reducing the risk of downtime. The microservice-based splitting of functional modules avoids the coupling issues of traditional monolithic architectures, allowing each service to be independently developed, tested, and deployed, greatly improving development efficiency. At the same time, containerized deployment simplifies the environment configuration and deployment process, reduces operation and maintenance costs, and facilitates rapid iteration and upgrades of functions. Bidirectional communication is established based on the web socket protocol, enabling real-time, full-duplex data exchange between the client and the server, ensuring the timeliness of message transmission. Version and integrity verification prevents messages from being tampered with, lost, or experiencing version incompatibility during transmission, effectively ensuring the accuracy and security of data exchange and enhancing user trust in the system. The design of appending a version identifier to messages enables flexible compatibility with different versions of clients and servers. The digital signature and hash value verification mechanism provides a standardized verification process for data transmission, making it easily adaptable to a variety of security requirements. Structured and unstructured data are stored in distributed relational databases and distributed file devices respectively, giving full play to the storage advantages of both and realizing efficient data storage; by establishing a primary key-path mapping table, the associated storage and unified management of the two types of data are realized, simplifying the data access process and improving data query and processing efficiency.

[0073] The distributed storage architecture supports dynamic expansion of storage resources, allowing flexible adjustment of storage capacity based on data growth. Object storage services are used for unstructured files, leveraging their cost-effective storage capabilities to reduce the cost of large-scale file storage. Elastic Search is used to construct composite indexes, supporting combined searches based on multiple criteria, including time, user, session, keyword, and file type, significantly reducing data search time and meeting users' complex search needs. The index structure optimizes metadata processing, reducing the data scan scope during searches, reducing resource consumption, and improving overall search performance. The multi-dimensional index data structure facilitates in-depth analysis of communication data, such as analyzing session trends and user behavior patterns within a statistical time period, providing data support for business decision-making and unlocking the data's potential value. Role-based access control assigns different permission levels to different users, ensuring that sensitive data is accessible only to authorized users, effectively preventing data leaks and ensuring data security and privacy. Flexible permission levels meet the diverse permission management needs of enterprises or organizations, enhancing the controllability of data management. By setting three fixed checkpoints to generate dynamic check values, the integrity and consistency of data during the archiving process are monitored in real time to ensure the accuracy of data migration; based on the dynamic check values, the microservice architecture deployment parameters are adjusted, log audit service instances are added, data operation records are strengthened, and data traceability is improved.

[0074] In a preferred embodiment of the present invention, step 1 above adopts a microservice architecture to distribute the functional services of the instant messaging device in a container environment, and implements dynamic node registration and discovery through a service governance component to build a communication infrastructure platform, which may include:

[0075] Step 110 , based on the functional requirements of the instant messaging device, split the functions into independent microservice units, including user authentication service, message routing service, user status management service, and contact management service;

[0076] Step 111: encapsulate the microservice unit into an independent container image and deploy it in a distributed container cluster environment using a container orchestration tool;

[0077] Step 112: deploy service governance components in the container cluster environment, including a service registry and a service discovery mechanism.

[0078] Step 113: Each microservice unit automatically registers its node with the service registration center upon startup, reporting its service identifier, network address, health status information, and past call frequency and response time data.

[0079] Step 114: The service discovery mechanism dynamically monitors the status changes of the microservice nodes in the service registry, calculates the load trend of the reported call frequency and response time using a time series prediction algorithm, and generates a future load forecast value;

[0080] Step 115: Based on the future load prediction value, deploy an API gateway at the entrance of the container cluster environment to perform unified routing, load balancing, and access control on the service access endpoints provided by the microservice units;

[0081] Step 116: Build a communication infrastructure platform through the distributed deployment of microservice units, dynamic node registration and discovery of service governance components, and unified access management of API gateways.

[0082] In an embodiment of the present invention, various business requirements of instant messaging devices are comprehensively sorted out. Taking user authentication service as an example, starting from the user entering the account password, the password needs to be encrypted. Usually, an irreversible hash encryption algorithm (such as SHA-256) is used to convert the plaintext password entered by the user into a hash value of a fixed length, which is then compared with the corresponding hash value stored in the database to determine whether the password is correct. At the same time, the validity of the account must be verified, such as whether it exists or is frozen. This series of operations constitutes an independent user authentication function module.

[0083] When sending a message, the message routing service needs to resolve the target address of the message (which may be a single user, group or channel) and determine the message transmission path based on the preset routing rules (such as selecting online devices for push based on the user's online status, and temporarily storing the message on the server if none of them are online). For example, when sending a message to a group, it is necessary to traverse the device information of all members in the group and distribute the message to the online device corresponding to each member. This message path planning and distribution process forms the message routing service.

[0084] The user status management service continuously monitors the user's network connection status, application status, and other aspects. It periodically sends heartbeat packets (small data packets used to detect device onlineness) to the user's device. If a response is received within a specified timeframe, the user is considered online. If no response is received, and no feedback is received after multiple attempts, the user is considered offline. The service also records the timestamps of user status changes, such as online to offline and offline to online, to facilitate access by other services. These status monitoring and recording operations constitute the user status management function.

[0085] The contact management service is responsible for handling operations such as adding, deleting, and modifying contacts. When adding a contact, the legitimacy of the addition request must be verified (such as whether the number of contacts exceeds the upper limit, whether the other party has set an addition limit, etc.), and then the corresponding contact relationship record must be created in the database; when deleting a contact, the relevant contact records and chat history-related information in the database must be deleted at the same time. These contact data management operations constitute the contact management service. Through such detailed functional decomposition, the overall function of the instant messaging device is divided into independent microservice units.

[0086] Taking the user authentication service as an example, first collect and organize the service code (including user authentication logic code, database operation code, etc.) and configuration files (such as database connection configuration, encryption algorithm configuration, etc.). At the same time, determine the operating environment required to run the service, such as a specific version of the Java operating environment (assuming that the user authentication service is developed based on Java) and related dependent libraries (such as database driver library, encryption algorithm library, etc.).

[0087] These resources are integrated according to the packaging specifications of the container image to form a complete container image. During the packaging process, the files and directories are reasonably organized and configured to ensure that the image can correctly load and execute the service code at runtime. Container orchestration tools (such as Kubernetes) are used according to pre-established deployment strategies. For example, it is stipulated that the user authentication service is initially deployed with 3 replicas to cope with the initial number of users, and a certain amount of CPU and memory resources are allocated to each replica (such as 1 core CPU and 2GB of memory for each replica). The nodes in the container cluster environment on which the deployment is carried out are specified (the selection can be based on factors such as node load and resource utilization), and the container image is deployed to the corresponding nodes.

[0088] In a container cluster environment, deploy a service registry (such as Eureka), which will start a service registration and management process within the cluster. The service registry will initialize its own database or data storage structure to store service registration information, such as service name, IP address, port number, health status, etc. At the same time, deploy a service discovery mechanism, which will establish a connection with the service registry and monitor changes in service information in the service registry in real time. The service discovery mechanism will maintain a service information cache internally and regularly synchronize the latest service information from the service registry to ensure that the service information it has is up to date.

[0089] When the container instance of the user authentication service is started, the registration logic is triggered within the service. The service generates its own unique service identifier (which can be composed of the service name and a randomly generated UUID), obtains the network address (IP address and port number) of the currently running container, and uses the built-in monitoring module to monitor the health status of the service in real time. For example, it checks whether the database connection is normal (try to perform a simple database query operation. If successful, the database connection is considered normal) and whether the service process responds to external requests (by sending a test request to check whether a response can be received within the specified time), etc., to obtain health status information. At the same time, the monitoring module also records the number of times the service is called in the past period of time (such as the past 1 hour), as well as the time from receiving the request to returning the response for each call, and calculates the call frequency and response time data such as the average response time. Then, these service identifiers, network addresses, health status information, and past call frequency and response time data are packaged into a registration request and sent to the service registration center for node registration.

[0090] The service discovery mechanism continuously obtains call frequency and response time data reported by each microservice node from the service registry. The collected data is arranged chronologically at 5-minute intervals to form time series data. Trend analysis focuses on changes in call frequency and response time within adjacent time intervals. For example, if the call frequency of the user authentication service increases over several consecutive time intervals, along with a corresponding increase in response time, this indicates an increase in service load. To identify periodicity in the data, analysis of past data reveals a significant peak in call frequency between 8:00 and 9:00 AM each day. This is a periodic characteristic of the data. Combining these trends and periodicity with other relevant factors (such as recent user growth trends and business promotions), the load is predicted for a future period (e.g., the next hour) by analogy with similar load change patterns in the past. This generates a future load forecast. For example, if, based on past data and current trends, the call frequency of the user authentication service between 8:00 and 9:00 AM tomorrow is likely to increase by 20% compared to the same period today, this prediction is used as the future load forecast.

[0091] Based on the future load forecast generated in step 114, an API gateway is deployed at the entrance of the container cluster environment. When an external request (such as a request from a client to obtain a contact list) arrives at the API gateway, the API gateway first checks the authentication information (such as a token) carried in the request. It then communicates with an authentication server (which can be a standalone authentication service or integrated into the user authentication service) to verify the validity of the token (checking whether the token has expired, been tampered with, or matches the user account). Only requests that pass authentication will be processed by the API gateway based on the load forecast and a preset load balancing strategy (such as a round-robin strategy or a least-connection strategy). If a round-robin strategy is used and the load forecast indicates that a particular instance of the contact management service has a low load, the API gateway will forward the request to that instance. If a least-connection strategy is used, the request will be assigned to the contact management service instance with the fewest connections, achieving load balancing. Furthermore, the API gateway controls the access rights of the request, checking whether the request has permission to access the corresponding service resources (for example, whether a regular user has permission to access the administrator's exclusive contact management functions). Only requests that pass permission verification will be forwarded to the corresponding microservice.

[0092] Through the distributed deployment of microservice units, services with different functions can operate independently and flexibly expand or contract based on business needs. When message traffic increases, the number of message routing service instances can be increased; when user management needs change, the user authentication service and contact management service can be upgraded separately. The dynamic registration and discovery of nodes in the service governance component enables services to perceive each other's status and location in real time. When a microservice fails or a new microservice instance is launched, other services can obtain this information and make corresponding adjustments. The unified access management of the API gateway standardizes the interaction process between external requests and microservices, ensuring the security and legitimacy of the requests. These three components work closely together to build a comprehensive, efficient, secure, and reliable communications infrastructure platform, providing solid support for the data management and communication functions of the instant messaging platform.

[0093] The instant messaging function is split into microservice units and deployed in a distributed manner. Each service can be upgraded, expanded or maintained independently. For example, when the increase in the number of users causes the message routing service to be overloaded, it is only necessary to increase the container instance of the message routing service without making large-scale changes to the entire system, thereby improving the system's ability to cope with business changes. The node dynamic registration and discovery mechanism of the service governance component can monitor the service status in real time. Once a microservice fails (such as the user status management service crashes), the service registration center will promptly remove it from the list of available services, and the service discovery mechanism will no longer forward requests to the failed service. At the same time, the API gateway will reallocate the requests to other normally operating service instances to ensure the overall stability and reliability of the system and reduce The risk of system crash due to single point failure; load trend calculation is performed through time series prediction algorithm, and load balancing is achieved through API gateway based on this, which can reasonably allocate system resources and assign requests to service instances with lower load, avoiding the situation where some services are overly busy while other services are idle, improving resource utilization, and at the same time reducing request response time and improving user experience; unified access management of API gateway, centralized authentication and access control of all requests, only legitimate requests can access the corresponding microservices, effectively preventing illegal access and malicious attacks, and ensuring the security of instant messaging platform data and services; at the same time, the unified management of services by service registration center and service discovery mechanism facilitates system administrators to monitor and maintain services.

[0094] In a preferred embodiment of the present invention, step 2, establishing bidirectional communication via the web socket protocol based on the communication infrastructure platform, performing version verification and integrity verification on the structured messages exchanged between the client and the server, and generating a verified message package, may include:

[0095] Step 220 , establishing a persistent two-way communication link between the client and the API gateway via the web socket protocol based on the API gateway unified access point provided by the communication infrastructure platform;

[0096] Step 221: Receive the structured message sent by the client and the response message returned by the server through the persistent two-way communication link;

[0097] Step 222: At the API gateway layer, a version identifier and a digital signature generated based on the message content are added to each structured message of the interaction;

[0098] Step 223: In the server-side message processing component, the version identifier is extracted and compared with a preset valid version range to generate a version verification result.

[0099] Step 224: Based on the version verification result, the digital signature is decrypted using the verification key corresponding to the signature algorithm to generate a message digest hash value. At the same time, a comparison hash value is calculated in real time based on the current message content. When the message digest hash value matches the comparison hash value calculated in real time, an abnormality confidence level is calculated. If the confidence level is lower than a threshold, the message is determined to be intact and has not been tampered with.

[0100] In step 225 , the structured message that has passed the dual verification of version verification and integrity verification is generated into a verified message package that carries a verification status mark.

[0101] In an embodiment of the present invention, after the client is started, it first parses the URL of the API gateway unified access point, extracts the protocol type (such as ws: / / or wss: / / ), domain name and port number, and initiates an HTTP request to the URL. The request header contains an Upgrade field with a value of "websocket", indicating that the HTTP connection is desired to be upgraded to a WebSocket connection. At the same time, the request header also contains a Sec-WebSocket-Key field, which is a random Base64-encoded string generated by the client. After receiving the request, the API gateway checks whether the Upgrade field is "websocket" and whether the Sec-WebSocket-Key field exists and is in the correct format. If the verification is successful, the API gateway generates a Sec-WebSocket-Accept response header. The calculation process of its value is to first concatenate the Sec-WebSocket-Key value sent by the client with a fixed globally unique identifier (GUID, with a value of "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"), then perform SHA-1 hash calculation on the concatenated string, and finally perform Base64 encoding on the hash result. The resulting encoded value is the value of Sec-WebSocket-Accept.

[0102] After receiving the response, the client verifies whether the value of Sec-WebSocket-Accept is correct. The verification method is that the client uses the Sec-WebSocket-Key value it sent and recalculates it according to the same algorithm as the server (GUID concatenation, SHA-1 hash, Base64 encoding). The calculation result is compared with the Sec-WebSocket-Accept value returned by the server. If they are the same, the verification is successful. The two parties will also negotiate the subprotocol and extension to be used. If both the client and the server support the same subprotocol and extension, the specific protocol and extension to be used are determined during the handshake process. After the handshake is successful, the HTTP connection is upgraded to a WebSocket connection, forming a persistent two-way communication link.

[0103] After a WebSocket connection is established, the client and server begin transmitting messages over the link. Before transmission, messages are segmented into multiple frames. Each frame consists of a header and frame data. The header includes an opcode (Opcode), which indicates the frame type (such as text, binary, or close), a mask (Mask), which indicates whether the frame data has been masked, and a payload (Payload Length), which indicates the length of the frame data. When the client sends a message, it encapsulates the message according to the WebSocket protocol format, forming one or more frames. If the message length exceeds a certain threshold, the message is segmented into multiple frames. Each frame's opcode has a corresponding identifier (for example, the first frame's opcode is 0x01, indicating a text frame, and subsequent frames' opcodes are 0x00, indicating continuation frames). The client generates a random 32-bit masking key for each frame and uses it to perform an exclusive-OR operation on the frame data to obtain the masked frame data.

[0104] After receiving the frame, the API gateway first parses the frame header information and extracts the operation code, mask bit, payload length, etc. If the mask bit is 1, it means that the frame data has been masked. The API gateway will use the mask key carried in the frame header to perform an XOR operation on the frame data to restore the original frame data. Then, the type of frame is determined based on the operation code. If it is a text frame or binary frame, the frame data will be accumulated until all continuation frames (if any) are received. The data of all frames are combined into a complete message content. If it is a control frame (such as a close frame, a Ping frame, etc.), it will be processed according to the corresponding protocol rules (such as responding to a Pong frame, closing the connection, etc.).

[0105] After receiving the complete message content, the API gateway first determines the type and format of the message and determines the version identifier that needs to be attached. The version identifier is usually determined based on the structure and function of the message. For example, if the message is generated based on a specific protocol version, the protocol version number is used as the version identifier. The API gateway obtains the corresponding version identifier string from the configuration file or the internally maintained version mapping table. To generate a digital signature, the API gateway first hashes the message content using the SHA-256 hash algorithm, taking the message content as input and calculating a 256-bit (32-byte) hash value. This hash value is unique to the message content. If the message content changes, the hash value will be different. The API gateway then encrypts this hash value using a pre-configured private key. The private key is a confidential key that only the API gateway knows. The encryption process uses an asymmetric encryption algorithm (such as RSA) to convert the hash value into an encrypted string, which is the digital signature.

[0106] The API gateway adds the version identifier and digital signature to specific fields of the message to form a new message structure. For example, you can add a "version" field to the message header to store the version identifier and a "signature" field to store the digital signature. Then, you can serialize this new message structure and convert it into a format suitable for transmission over the network (such as JSON, XML, etc.).

[0107] After receiving the serialized message, the server-side message processing component first deserializes it into an internal data structure and extracts the value of the version identifier field. The server then obtains the valid version range supported by the current system from the configuration file or database. The valid version range is usually a continuous version number range, such as "v1.0-v2.5", or a discrete version number list, such as ["v1.0", "v1.5", "v2.0"]. The server compares the extracted version identifier with the valid version range. If the valid version range is a range, the server parses the version identifier into a numeric or alphanumeric version number and then compares the version number to see if it is between the minimum and maximum values ​​of the range. For example, if the valid version range is "v1.0-v2.5" and the extracted version identifier is "v2.0", "v2.0" is between "v1.0" and "v2.5", and the version verification passes. If the extracted version identifier is "v3.0", it is not within the valid range and the version verification fails.

[0108] If the valid version range is a discrete list, the server will directly check whether the extracted version identifier is in the list. For example, if the valid version list is ["v1.0", "v1.5", "v2.0"], and the extracted version identifier is "v1.5", then it is in the list and the version verification passes; if it is "v2.5", it is not in the list and the version verification fails. Based on the comparison result, the version verification result is generated, which is mostly a Boolean value indicating whether the version is valid.

[0109] After the message passes the version verification, the server begins to perform integrity verification. First, the server uses the same hash algorithm (SHA-256) as the API gateway to recalculate the currently received message content (excluding the digital signature field) to obtain a new hash value, called the comparison hash value. Then, the server uses the pre-configured public key to decrypt the digital signature in the message. The public key and the private key used by the API gateway to generate the digital signature are a pair of keys. Only the corresponding public key can correctly decrypt the digital signature.

[0110] After decrypting the digital signature, the result is the hash value originally calculated by the API gateway for the original message content, called the message digest hash value. The server compares the message digest hash value with the recalculated comparison hash value bit by bit. If the two hash values ​​are exactly the same, it means that the message has not been tampered with during transmission. If any bit is different, it means that the message content has changed and the integrity check has failed.

[0111] To further ensure the reliability of the message, the server will also perform anomaly confidence calculations. The server will collect information from multiple dimensions to evaluate the degree of abnormality of the message. The first is the credibility of the source of the message. The server will maintain a list of trusted clients. If the message comes from a client in the list, the credibility is high; if it comes from an unknown client, the credibility is low. The second is the transmission time of the message. The server will calculate the time interval from the sending to the receiving of the message. If the time interval is too long or too short, exceeding the normal range, there may be an abnormality.

[0112] The server also analyzes message frequency and patterns. If a client sends a large number of identical or similar messages in a short period of time, or if the message content and format do not conform to normal business logic, this is considered an anomaly. The server assigns a weight to each evaluation dimension—for example, a weight of 0.4 for message source credibility, 0.3 for transmission time, and 0.3 for message frequency and pattern. Based on the evaluation results for each dimension, the server calculates a comprehensive anomaly confidence score. This is calculated by multiplying each dimension's score by its weight and then adding all the results together. For example, if the message source credibility score is 0.8 (out of 1), the transmission time score is 0.6, and the message frequency and pattern score is 0.7, the anomaly confidence score is 0.8 × 0.4 + 0.6 × 0.3 + 0.7 × 0.3 = 0.71. The server compares this anomaly confidence score with a preset threshold (for example, 0.8). If the anomaly confidence score is below the threshold, the message is considered normal; if it is above the threshold, the message is considered an anomaly.

[0113] For messages that pass version and integrity checks, the server adds a verification status flag. This flag is typically a Boolean field, such as "verified." Set to true, it indicates the message has passed all verifications. The server also adds a verification timestamp to the message, recording the exact time the message passed verification. This is typically formatted in ISO 8601 format (e.g., "2023-05-15T10:30:00Z"). The server generates a unique identifier for the message for tracking and management. This identifier is typically a UUID (Universally Unique Identifier), a 128-bit string of numbers and letters that ensures global uniqueness. The server adds the verification status flag, verification timestamp, and unique identifier to the message data structure, forming a new message object. The server then further formats and encapsulates this message object based on processing needs. The server may convert the message to a specific message queue format for sending to other services for processing, or store the message in a database. This processed and encapsulated message is a verified message package, which can be safely circulated and used within the system.

[0114] Through the version verification mechanism, different versions of messages can be automatically identified and processed to ensure that the new version of the client and server are compatible with the old version of the message. At the same time, it can also prevent the old version of the client from sending incompatible messages to the new version of the server, improving compatibility and maintainability; the digital signature and hash value comparison mechanism can effectively prevent the message from being tampered with or forged during transmission. Any modification to the message content will cause the hash value to change, thus causing the integrity check to fail. The use of asymmetric encryption algorithms ensures that only the API gateway with the correct private key can generate a valid digital signature, and only the corresponding public key can verify the validity of the signature, which improves the authenticity and credibility of the message and ensures security; the abnormal confidence calculation evaluates the reliability of the message from multiple dimensions, and can promptly detect and handle abnormal messages to prevent malicious attacks and abnormal traffic from affecting the system. The persistent two-way communication link of WebSocket reduces the overhead of connection establishment and disconnection, improves communication efficiency, and also enhances stability, reducing communication interruptions and data loss caused by network fluctuations; the verified message package carries a clear verification status mark and a unique identifier, which facilitates the rapid identification and processing of legitimate messages by the subsequent service, reducing the overhead of repeated verification. The unique identifier also facilitates the tracking and management of messages, and can quickly locate and troubleshoot problems when they arise.

[0115] In a preferred embodiment of the present invention, step 3 above, based on the verified message packets, constructs a hierarchical storage system to write communication data into a distributed relational database in real time, and simultaneously stores attachments via a distributed file device to achieve unified storage of structured and unstructured data, which may include:

[0116] Step 330 , writing the text content, timestamp field, and user relationship field in the verified message package into a distributed relational database shard to generate a message record including a primary key;

[0117] Step 331: extract the file attachment from the corresponding verified message package based on the message record, and upload the file attachment to the object storage service to generate a unique storage path;

[0118] Step 332: Bind the message record primary key to the storage path, create a primary key-path mapping table in the relational database, and form a unified storage structure in which text data and file attachments can be associated and accessed.

[0119] In an embodiment of the present invention, structured text content (e.g., message body, title, sender ID, recipient ID), a timestamp field (UTC time accurate to the millisecond), and user relationship fields (e.g., session ID, message type, and permission level) are extracted from a verified message packet. A shard key is calculated based on the message's session ID or user ID. For example, a hash algorithm (e.g., MurmurHash3) is used to convert the session ID into a 64-bit integer, which is then modulo the total number of database shards to determine the target shard location. Atomic operations within a message packet are divided into database transactions. For example, the sending operation of a message may include inserting a message record, updating the session state, and incrementing an unread count. These operations must be completed within the same transaction.

[0120] SQL generation and execution: dynamically generate INSERT statements containing all extracted fields and calculated shard keys; execute distributed transaction coordination protocols (such as two-phase commit or TCC mode) to ensure consistency of cross-shard operations; generate globally unique primary keys (such as UUIDv1 or IDs generated by the Snowflake algorithm) to ensure non-duplicate operations in a distributed environment.

[0121] Create B-Tree indexes for commonly searched fields (such as timestamp, session ID, and sender ID), and inverted indexes for full-text search fields to improve query performance. parse the binary stream of the file attachment from the message packet, verifying the file type (using a magic number checksum and file extension) and size (compared to a preconfigured upper limit). Use the SHA-256 algorithm to calculate a hash value for the file content for deduplication and integrity verification. Create a directory structure based on the first 6 characters of the content hash value, with the remaining character used as the file name.

[0122] Add a timestamp prefix to ensure file name uniqueness and prevent conflicts with concurrent uploads. Use multipart uploads to split large files into 5MB chunks and upload them in parallel to increase upload speed. After the upload is complete, calculate and verify the ETag to ensure that the uploaded content matches the locally calculated hash value. Create a metadata JSON file containing information such as the file name, original size, MIME type, and upload time, and store it with the file.

[0123] Associate the message record primary key generated in step 330 with the object storage path generated in step 331 to form a key-value pair. Map the ID (auto-incrementing primary key), message record primary key (foreign key), storage path, file status (such as uploading, completed, deleted), and version number. Use a clustered index to sort and store the messages by primary key to improve the efficiency of associated queries. Use batch write mode to submit multiple mappings to the database at once, reducing network overhead and implementing an optimistic locking mechanism (using the version number field) to prevent concurrent update conflicts. Create a secondary index for the storage path to support reverse queries of associated message records by file path. Load frequently accessed mappings into a distributed cache (such as Redis) and set a reasonable TTL (such as 5 minutes) to reduce database access pressure.

[0124] Separate storage for structured text data and unstructured file attachments avoids the performance bottlenecks associated with large file storage in traditional relational databases. The tiered storage system allows different types of data to scale independently based on their access characteristics and growth patterns, improving overall system scalability. Sharding within a distributed relational database improves write throughput and query concurrency for structured data, making it particularly suitable for high-concurrency messaging scenarios. The content-addressing mechanism and distributed architecture of the object storage service naturally support efficient storage and fast access to massive files. Object storage's multi-copy redundancy mechanism (such as triple-copy storage) ensures high availability of file attachments, even in the event of partial node failures. The transactional mechanism of the distributed relational database ensures the consistency of message records and mapping relationships, preventing data loss or inconsistency. Low-cost object storage tiers (such as archive storage) are suitable for long-term preservation of historical attachments, offering significant cost savings compared to traditional storage solutions. A deduplication mechanism (based on content hashing) eliminates the storage of duplicate files, further optimizing storage space utilization. A unified mapping table clearly identifies the relationship between text data and file attachments, simplifying data retrieval and management. The associated storage of metadata with files provides complete file context, facilitating file lifecycle management. The distributed architecture supports horizontal scalability, easily accommodating growth in data volume and access volume by adding database nodes and object storage clusters. The independent storage tier design enables the system to flexibly adapt to evolving storage technologies, protecting technology investments. Diverse indexing strategies (B-Tree, inverted index, and auxiliary index) meet diverse query requirements and improve data retrieval efficiency. The caching mechanism reduces database access for hot data and reduces system response time.

[0125] In a preferred embodiment of the present invention, step 4 above establishes a multi-dimensional indexing mechanism based on the uniformly stored data, uses elastic search to create a composite index for the metadata of messages and files, and generates indexable data that can be searched by multiple conditions based on time, user, session, keyword, and file type, which may include:

[0126] Step 440: extract the message sending time field, user ID field, session ID field, keyword tag, and file media type field from the unified storage to form a structured metadata set;

[0127] Step 441 , based on the structured metadata set, construct a composite index structure in the elastic search cluster, including a time range filter, a user session aggregator, and a keyword analyzer;

[0128] Step 442: Continuously collect user query logs, dynamically adjust index field weights based on query frequency statistics, and generate a pre-calculated index table for high-frequency combined queries;

[0129] Step 443: Generate an index data structure for multi-field combination queries on the message sending time field, user ID field, session ID field, keyword tag, and file media type field through a composite index structure and a precomputed index table, which is the index data.

[0130] In the embodiment of the present invention, first connect a distributed relational database and a distributed file storage system. In the database, by executing a data query statement, read data row by row from the message record table. For each message record, accurately locate the message sending time field, which may be stored in the form of a timestamp, and convert it into a date and time format that is easy for people to understand (such as "December 31, 2024 15:20:00"); the user ID field is the unique identifier of the user, whether it is a number, a combination of letters and numbers, or a string in a specific format, and it is directly extracted; the session ID field is used to distinguish different chat conversations, and it is also completely extracted.

[0131] For the keyword tags in the message, using natural language processing technology, first perform word segmentation on the message text, splitting long sentences into individual words. For example, split "Project discussion will be held in the conference room this afternoon" into words such as "this afternoon", "conference room", "project", "discussion", etc. Then, remove the stop words that have no practical meaning, such as "the", "of", "in", "is", etc. The remaining words such as "conference room", "project", "discussion", etc. are extracted as keyword tags. If the message contains a file attachment, the system will go to the distributed file storage system, find the corresponding file metadata according to the storage path of the file, and obtain the file media type field in the metadata. For example, "image / jpeg" indicates a JPEG format picture file, "application / pdf" indicates a PDF format document file, etc. Finally, integrate all the above-extracted fields together to construct a structured metadata set for each message data. For example, [Message sending time: December 31, 2024 15:20:00, User ID: user001, Session ID: chat003, Keyword tags: [conference room, project, discussion], File media type: application / pdf].

[0132] In an Elasticsearch cluster environment, we begin building a composite index structure by creating a time range filter. This filter comprehensively analyzes all extracted message sending times. It divides time into appropriate granularity based on business needs and data volume. For example, it can be divided by day, with each of the 365 days of a year corresponding to 365 time intervals, or by hour, with each 24-hour day having 24 intervals. For each time interval, a corresponding index entry is created. When new message data arrives, the send time is used to determine which interval it belongs to and assign it to the corresponding time index interval. For example, a message sent at "December 31, 2024, 10:15:00" will be assigned to the time index interval "December 31, 2024, 00:00:00 - December 31, 2024, 23:59:59."

[0133] Then build a user session aggregator, traverse all structured metadata collections, and for each user ID, search for all session IDs in which the user participates. For example, if user "user001" participates in three sessions "chat001", "chat003" and "chat005", all message metadata related to "user001" in these three sessions will be aggregated together. During the aggregation process, an association relationship is established, so that all message data of the user in different sessions can be quickly found by user ID, and can also be found by session ID. Find all user message data in a specific conversation, and finally establish a keyword analyzer. For the extracted keyword tags, use the word segmentation algorithm for further processing. For example, for the keyword "project progress report", it will be split into three independent terms: "project", "progress", and "report". Then, establish an inverted index for each term. The principle of the inverted index is to associate each term with all message data containing the term. For example, the term "project" will record the location or identifier of all message data containing "project" in the keyword tag. In this way, when the user searches for the keyword "project", all related message metadata can be quickly found through the inverted index.

[0134] Continuously collect user query logs, which record the specific information of each user's query in detail, including the query conditions (such as by time, user ID, keyword, etc.), the time when the query was initiated, the number of results returned by the query, and the query time. Through in-depth analysis and statistics of these query logs, the frequency of each index field in all queries is calculated. For example, in the past week, a total of 1,000 query operations were performed, of which 300 were queries by keyword tags, then the query frequency of the keyword tag field is 30%; there were 250 queries by user ID, and the user ID field was 20%. The query frequency of the segment is 25%; the query by session ID is 150 times, with a frequency of 15%; the query by message sending time is 150 times, with a frequency of 15%; the query by file media type is 100 times, with a frequency of 10%. Based on the calculated query frequency, a weight is dynamically assigned to each index field. The more frequently queried a field is, the greater the weight it is assigned to reflect its importance in the query. Suppose the weight of the keyword tag is set to 0.4 because it has the highest query frequency; the weight of the user ID is set to 0.3; the weight of the session ID is set to 0.15; the weight of the message sending time is set to 0.1; and the weight of the file media type is set to 0.05.

[0135] At the same time, we carefully analyze the query logs to identify frequently occurring combination query conditions. For example, we find that the combination query "user ID is user001 and the keyword tag contains project" appeared 50 times in the past week, which is a high-frequency combination query. For this type of high-frequency combination query, we pre-calculate it. During the pre-calculation, we filter out all message metadata that meets the combination query conditions in advance based on the existing index data, and organize and store these result sets in the pre-calculated index table. In this way, when a user initiates the same combination query again, there is no need to re-perform complex query calculations. The results can be directly obtained from the pre-calculated index table and returned to the user, improving query efficiency.

[0136] The constructed composite index structure is combined with the generated pre-calculated index table to form a complete index data structure. When a user initiates a multi-field combination query request, such as "query all messages of user user002 during November 2024 that contain the keyword "conference" and the file media type is application / pdf", the pre-calculated index table will be checked first to see if there is a result set that meets the query conditions; if it exists, the result set will be directly formatted and returned to the user; if it does not exist, the composite index structure will be used for querying; first, the time range filter will be used to find the time index interval corresponding to "November 2024" to filter out all message elements within this time period. data; then, through the user session aggregator, find the relevant message metadata of the user ID "user002" from the filtered results; then, using the inverted index of the keyword analyzer, further filter out the message metadata containing the keyword tag "meeting"; finally, based on the file media type field, filter out the message metadata with the file media type of "application / pdf"; these message metadata after multiple layers of filtering are organized and stored in a specific format to form the final index data structure, that is, indexable data that meets the user's multi-condition combination query requirements. This indexable data can be quickly retrieved and accessed, thereby responding to the user's query requests in a timely manner and returning accurate query results.

[0137] The construction of a multi-dimensional indexing mechanism and a composite index structure eliminates the need for full-table scan queries when faced with massive amounts of instant messaging data. Through the collaborative operation of components such as time range filters, user session aggregators, and keyword analyzers, data that meets the query conditions can be quickly located, greatly shortening query response time. This allows users to quickly obtain the required message data, improving the user experience of the instant messaging platform. By analyzing user query logs and dynamically adjusting index field weights, resources can be allocated to high-frequency query fields, avoiding over-indexing and resource waste on low-frequency query fields. At the same time, the pre-calculated index table processes high-frequency combination queries in advance, reducing the workload of repeated calculations, reducing computing resource consumption and storage pressure, and improving overall resource utilization efficiency. Indexable data supports combined retrieval based on multiple conditions such as time, user, session, keyword, and file type. Whether searching for chat records between a specific user and a specific object within a certain time period, or searching for specific types of file messages containing specific keywords, accurate results can be returned, fully meeting the diverse query needs of users in different scenarios. It continuously collects user query logs and dynamically adjusts index strategies based on query behaviors, and can make timely adjustments as user usage habits and business needs change. Regardless of how user query preferences change, it can maintain efficient retrieval performance by adjusting the index structure and weight distribution, thereby enhancing the system's adaptability and flexibility to different usage scenarios and business changes. The structured metadata collection and clear index data structure facilitate data management and analysis, allowing data managers to more easily perform statistics, monitoring, and maintenance operations on data. Data analysts can also quickly conduct data mining and in-depth analysis based on index data, thereby providing more valuable data support and reference for the company's business decisions.

[0138] In a preferred embodiment of the present invention, step 5 above, implementing fine-grained permission management based on indexable data, setting permission levels through role-based access control, and generating permission-controlled data, may include:

[0139] Step 550: In the data management device, a role-based access control role set is defined, including an administrator role, a department head role, and a common member role, and role definition information is generated;

[0140] Step 551: Based on the role definition information, an access control role identifier that allows access to the data entity is associated with each data entity in the indexable data to form a role access tag, thereby generating an indexable data entity set with the role access tag.

[0141] Step 552: Deploy a permission interception function at the query interface layer and input the indexable data entity set with the role access tag into the permission interception function. When a user initiates a data query request through the query interface layer, the permission interception function identifies the current user's access control role and analyzes the user behavior log in real time to calculate the permission deviation degree.

[0142] In step 553, the permission interception function filters the data entities that meet the query conditions in the indexable data entity set with role access tags according to the identified current user access control role, returns the data entities retained after filtering as query results, and generates data subject to permission control.

[0143] In an embodiment of the present invention, in a data management device, an administrator begins to define a role-based access control role set based on the organizational structure and business needs. Taking the enterprise instant messaging scenario as an example, the core role types are first determined, such as the administrator role, the department head role, and the ordinary member role; for the administrator role, its authority scope is determined to cover the data management operations of the entire instant messaging platform. The administrator can create, modify, and delete user accounts, adjust the authority configuration of other roles, manage the chat records and file storage of all users, and even view the system logs to troubleshoot security issues. These permissions are recorded in detail to form the definition information of the administrator role, such as "the administrator role has user management, authority configuration, full platform data viewing and deletion, system log access, etc."

[0144] The permissions of the department manager role are set based on the department's business scope. The department manager can view and manage the chat records of department members, approve file sharing requests from department members, and perform limited management operations on the accounts of department members, such as resetting passwords and adjusting some permissions. Similarly, these permissions will be recorded to generate department manager role definition information, such as "The department manager role can access the department member data, perform partial management of member accounts, and approve file sharing."

[0145] The Ordinary Member role has relatively limited permissions, allowing users to access only their own chat history and authorized files. They can modify their personal account information, but cannot manipulate other users' data or permissions. This is recorded as "Ordinary Member roles can only access personal data and authorized files, and can modify personal information." By combining these different roles and their corresponding permissions, a complete role definition is generated.

[0146] Based on the role definition information generated in step 550, each data entity in the indexable data is processed. Taking a chat message data entity as an example, the attributes and business scenarios of the message are first determined. If it is a common internal department chat message and the users involved all belong to the same department, the department head role ID will be associated with the message based on the department head role's access rights to the department's data. For full-platform notification messages created by administrators, since administrators have full-platform data access rights, the administrator role ID will be associated with them. If it is a private chat message between ordinary members, it only involves personal data access of the message sender and receiver. The message will be associated with an ordinary member role identifier, and it will be noted that only the ordinary member roles corresponding to the message sender and receiver have access rights. All indexable data entities will be traversed, and the access control role identifiers of the data entities that are allowed to be accessed will be associated with them one by one according to the above rules to form a role access label. For example, the role access label of a message may be [Access allowed role: ordinary member (sender ID: user001, receiver ID: user002), department head (department ID: dept001)]. By combining all data entities with role access labels, a set of indexable data entities with role access labels is generated.

[0147] At the query interface layer, a permission interception module is deployed. This module acts as a "data gatekeeper," inspecting all data query requests initiated by users through the query interface layer. When the permission interception function receives a user's data query request, it first identifies the current user's access control role from the user's login information or the authentication information accompanying the request. For example, the token information carried by user "user003" when logging in is verified, confirming that their role is department manager. Simultaneously, the permission interception function collects and analyzes user behavior logs in real time. User behavior logs record various user operations in the system, including query time, query conditions, and accessed data types. By analyzing these logs, the permission deviation degree is calculated. For example, a department manager role normally primarily queries data of department members. If this user frequently attempts to query data from other departments or queries data types that are beyond their permission scope (such as attempting to query system configuration data reserved for administrators), a permission deviation value is calculated based on preset rules, taking into account factors such as query frequency and the sensitivity of the queried data. Assuming that the permission deviation threshold for normal query behavior is 0.3, if the calculated permission deviation degree of a query by the department manager reaches 0.5, it indicates that their behavior is abnormal.

[0148] The permission interception function filters the indexable data entity set with role access tags based on the identified current user access control role. Taking the department manager "user003" as an example, when he initiates a query request "Query all chat records of this department in December 2024", the permission interception function will traverse the indexable data entity set and check the role access tag of each data entity; for those chat record data entities whose role access tags contain the department manager role identifier, the message sending time is December 2024, and they belong to this department, the permission interception function will retain them; for data entities that do not meet these conditions, such as chat records of other departments, records not in December 2024, or data entities that do not contain the department manager role access tag, they will be filtered out; the data entities retained after filtering are sorted and returned to the user as query results according to the format requirements of the user query request. In this way, the data that the user finally obtains is permission-controlled and only contains permission-controlled data that he has access to, ensuring that the user cannot obtain data beyond his authority.

[0149] Role-based access control allows for fine-grained data access permissions, allowing different roles to access only data within their authorized scope. This effectively prevents unauthorized access to sensitive information by internal personnel, mitigates data leakage risks, and protects the core data assets of an enterprise or organization. Role definition and permission configuration streamline and standardize data management, allowing administrators to easily adjust role permissions based on business needs without having to individually configure permissions for each user. Department heads can also efficiently manage their department's data access, reducing management costs and operational complexity. Only relevant personnel can access personal privacy information, such as private chat data of ordinary members. Even if other roles have access to some data, they cannot view others' private content. This fully protects users' data privacy and enhances their trust in the instant messaging platform. Data access is controlled based on role permissions, ensuring that critical resources are only used by authorized roles, preventing resource abuse. Detailed role access tags and user behavior logs provide a comprehensive basis for data access audits. When data security issues or permission disputes arise, query logs and role permission configurations can quickly trace data access behavior, clarify responsibilities, and facilitate problem investigation and resolution.

[0150] In a preferred embodiment of the present invention, step 6 above executes a periodic data archiving strategy based on the data under permission control, automatically migrates overdue data to a cloud storage device, and sets three fixed checkpoints arranged in sequence during the archiving process. Dynamic check values ​​are generated based on the feedback results of each checkpoint. Based on the dynamic check values, the deployment parameters of the microservice architecture are adjusted to achieve data traceability, which may include:

[0151] Step 660: Based on the predefined lifecycle configuration table of data subject to permission control, when it is detected that the preset archiving trigger condition is met, a data archiving task is automatically initiated;

[0152] Step 661: In the data archiving task execution process, checkpoints are preset for three key operation stages. These include a data extraction stage checkpoint, which is set after the permission-controlled data extraction operation from the source storage device is completed; an encrypted transmission stage checkpoint, which is set after the data encryption is completed and the transmission operation to the cloud is started; and a cloud storage stage checkpoint, which is set after the data is successfully written to the cloud persistent storage device.

[0153] Step 662: During the execution of the data archiving task, when each set checkpoint is reached, a corresponding check operation is performed and check parameters are generated, including a compression ratio deviation value, a transmission hash value, and a number of storage copies;

[0154] Step 663: The compression ratio deviation value, the transmission hash value, and the number of stored copies are used to form a dynamic check matrix that reflects the multi-stage integrity of the archiving task. The dynamic check matrix is ​​analyzed, and a matrix anomaly index that reflects the potential risk of the archiving process is calculated according to predefined rules.

[0155] Step 664 , based on the value of the matrix anomaly indicator, dynamically increase the number of running instances of the log audit service responsible for recording data operation details, and achieve data traceability through the increased running instances of the log audit service.

[0156] In an embodiment of the present invention, data attributes (such as creation time, update time, and access frequency) in a lifecycle configuration table are quantized, and each attribute is assigned a different weight. The weights are determined using a particle swarm optimization algorithm, with the weight range serving as the search space for particles. Each particle represents a weight combination, and an objective function is set to optimize the accuracy and timeliness of data archiving. For example, the objective function might minimize the sum of the false positive rate and false negative rate for data that meets the archiving criteria. The particle swarm continuously iterates within the search space, updating the particle's speed and position based on feedback from the objective function, ultimately finding the final weight combination. For each data record, an archiving trigger index is calculated based on the determined weights and the quantized attribute values. The archiving trigger index is the sum of the product of each attribute value and the corresponding weight. When this index reaches or exceeds a preset archiving trigger threshold, the archiving trigger condition is considered met, and the data archiving task is automatically initiated.

[0157] When determining the locations of checkpoints, we introduce a particle swarm optimization algorithm, dividing the data archiving task execution process into multiple sub-stages, with each sub-stage corresponding to a particle in the swarm. The objective function is to quickly locate the problem stage when errors occur during the data archiving process, while also considering the impact of checkpoint settings on system performance (such as the additional overhead caused by the checkpoint operation). Particles iterate in the search space, adjusting their positions based on the objective function. Through continuous optimization, they identify three key operational stages: data extraction, encrypted transmission, and cloud storage. This allows for effective monitoring of the data archiving process while minimizing performance losses.

[0158] After data extraction is complete, a particle swarm optimization algorithm is used to optimize the parameters of the data compression algorithm. The compression algorithm parameters (such as compression level and dictionary size) serve as the particle search space, with the objective function set to maximize the compression ratio and minimize compression time while ensuring data integrity. The particle swarm iterates through the search space, adjusting its speed and position based on the objective function to find the final compression algorithm parameter combination. This combination is used to compress the extracted data, and the actual compression ratio is calculated. This is then compared with the average compression ratio of similar historical data or a preset standard compression ratio to determine the compression ratio deviation. After data encryption and transmission begins, the particle swarm optimization algorithm is used to optimize the transmission path and protocol parameters. The particle search space is filled with possible transmission paths and protocol parameters (such as transmission rate and error correction mechanism parameters). The objective function is set to minimize the data error rate and maximize transmission efficiency during transmission. The particle swarm iterates and finds the final transmission path and protocol parameter combination. During transmission, a hash value of the data, known as the transmission hash value, is calculated in real time to verify the integrity of the data during transmission. After the data is successfully written to the cloud persistent storage device, the particle swarm optimization algorithm is used to determine the number of storage copies. The possible value range of the number of storage copies is used as the search space of the particles. The objective function comprehensively considers data security (for example, the more copies, the safer the data) and storage cost (the more copies, the higher the storage cost). For example, it can be to minimize the weighted sum of data loss risk and storage cost. The particle swarm finds the final number of storage copies through iteration.

[0159] The compression ratio deviation, transfer hash value, and number of stored copies are normalized to the same numerical range. These three values ​​are then arranged sequentially to form a dynamic checksum matrix. The analysis rules for the dynamic checksum matrix are optimized using a particle swarm optimization algorithm, with the parameters of the analysis rules (such as the weights of each indicator and the anomaly threshold) serving as the particle search space. The objective function is set to ensure that the matrix anomaly indicator most accurately reflects the potential risks of the archiving process—that is, it correctly identifies situations that pose real risks while reducing false positives. The particle swarm algorithm iterates to find the final combination of analysis rule parameters. Based on this combination and predefined rules, the matrix anomaly indicator is calculated. For example, this can be achieved by weighting each indicator and comparing it with a threshold to determine the value of the anomaly indicator.

[0160] A mapping relationship is established between the matrix anomaly indicator values ​​and the number of running instances of the log audit service. The parameters of this mapping relationship are optimized using a particle swarm optimization algorithm. The mapping parameters (such as scaling coefficients and offsets) serve as the particle search space. The objective function is set to minimize the resource consumption of the log audit service while ensuring data traceability accuracy. The particle swarm finds the final mapping relationship parameters through an iterative search. Based on the calculated matrix anomaly indicator values ​​and the optimized mapping relationship, the number of running instances of the log audit service is dynamically determined and increased, thereby enhancing the recording and auditing capabilities of data operations and achieving data traceability.

[0161] Using a particle swarm optimization algorithm to determine the weights of data attributes and trigger thresholds for archiving, the system can more accurately determine whether data meets archiving requirements, reduce false positives and missed data, ensure that permission-controlled data is archived at the appropriate time, and improve the accuracy and timeliness of data management. Determining the location of checkpoints allows for more efficient distribution of checkpoints at key stages of data archiving, effectively monitoring the data archiving process and identifying issues promptly. At each checkpoint, the particle swarm optimization algorithm optimizes the parameters of relevant operations (such as compression, transmission, and storage), improving data processing efficiency and quality, generating more reliable checksum parameters, and safeguarding data integrity. Dynamic check matrix analysis rules more accurately calculate matrix anomaly indicators, effectively identifying potential risks during the data archiving process, providing a reliable basis for the system to take appropriate measures, and reducing the probability of data archiving failures. Determining the mapping between the number of running log audit service instances and matrix anomaly indicators allows for dynamic adjustment of log audit service resources based on the actual risk of the archiving process. This ensures data traceability while avoiding resource waste, improving resource utilization, and reducing operating costs.

[0162] like Figure 2 As shown, an embodiment of the present invention further provides a data management system based on an instant messaging platform, comprising:

[0163] The architecture deployment module is used to build a communication infrastructure platform to achieve distributed deployment of instant messaging device functional services and dynamic node registration and discovery;

[0164] A communication verification module is used to establish two-way communication based on the communication infrastructure platform through the web socket protocol and perform version and integrity verification on structured messages to generate verified message packets;

[0165] The data storage module is used to build a hierarchical storage system based on verified message packets, write communication data into a distributed relational database in real time and store attachments through distributed file devices, realizing unified storage of structured and unstructured data;

[0166] An index building module, which is used to create composite indexes for message and file metadata based on uniformly stored data, generating indexable data that can be searched based on multiple conditions such as time, user, session, keyword, and file type.

[0167] The permission control module is used to implement fine-grained permission management based on indexable data, set permission levels through role-based access control, and generate permission-controlled data;

[0168] The data archiving module is used to execute periodic data archiving strategies based on permission-controlled data, migrate expired data to cloud storage, and generate dynamic checksums based on feedback from checkpoints during the archiving process to adjust microservice architecture deployment parameters and achieve data traceability.

[0169] It should be noted that this system is a system corresponding to the above method, and all implementation methods in the above method embodiment are applicable to this embodiment and can achieve the same technical effects.

[0170] An embodiment of the present invention further provides a computing device comprising: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the above-described method. All implementations in the above-described method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0171] The embodiment of the present invention further provides a computer-readable storage medium storing instructions, which, when executed on a computer, causes the computer to execute the above-described method. All implementations in the above-described method embodiment are applicable to this embodiment and can achieve the same technical effects.

[0172] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A data management method based on an instant messaging platform, characterized in that: The method comprises: Step 1: Using a microservices architecture, the functional services of the instant messaging device are distributed and deployed in a container environment. Dynamic node registration and discovery are implemented through service governance components to build a communication infrastructure platform. Step 2: Based on the communication infrastructure platform, establish two-way communication through the web socket protocol, and perform version verification and integrity verification on the structured messages exchanged between the client and the server to generate a verified message package. Specifically, it includes: establishing a persistent two-way communication link between the client and the API gateway through the web socket protocol based on the API gateway unified access point provided by the communication infrastructure platform; receiving the structured messages sent by the client and the response messages returned by the server through the persistent two-way communication link; at the API gateway layer, attaching a version identifier and a digital signature generated based on the message content to each structured message of the interaction In the server-side message processing component, the version identifier is extracted and compared with the preset valid version range to generate a version verification result. Based on the version verification result, the digital signature is decrypted using the verification key corresponding to the signature algorithm to generate a message digest hash value. At the same time, the comparison hash value is calculated in real time based on the current message content. When the message digest hash value matches the comparison hash value calculated in real time, an abnormal confidence calculation is performed. If the confidence is lower than the threshold, the message is determined to be complete and has not been tampered with. The structured message that has passed the dual verification of version verification and integrity verification is generated as a verified message package carrying a verification status mark. Step 3: Based on the verified message packets, a hierarchical storage system is constructed to write the communication data into a distributed relational database in real time. Attachments are stored in a distributed file device, achieving unified storage of structured and unstructured data. Step 4: Based on the uniformly stored data, a multi-dimensional indexing mechanism is established. Elasticsearch is used to create a composite index for the metadata of messages and files, generating indexable data that can be searched based on multiple conditions such as time, user, session, keyword, and file type. Step 5: Implement fine-grained permission management based on indexable data, set permission levels through role-based access control, and generate permission-controlled data; Step 6: Based on the permission-controlled data, execute the periodic data archiving strategy, automatically migrate the expired data to the cloud storage device, and set three fixed detection points in sequence during the archiving process. Generate dynamic verification values ​​based on the feedback results of each detection point. Based on the dynamic verification values, adjust the microservice architecture deployment parameters to achieve data traceability.

2. The data management method based on the instant messaging platform according to claim 1, characterized in that: Using a microservices architecture, the functional services of instant messaging devices are distributed and deployed in a container environment. Dynamic node registration and discovery are achieved through service governance components to build a basic communication platform, including: Based on the functional requirements of instant messaging devices, the functions are divided into independent microservice units, including user authentication service, message routing service, user status management service and contact management service; Encapsulate microservice units in independent container images and deploy them in a distributed container cluster environment using container orchestration tools; Deploy service governance components in a container cluster environment, including a service registry and service discovery mechanism; Each microservice unit automatically registers its node with the service registration center when it starts, reporting the service identifier, network address, health status information, and past call frequency and response time data; The service discovery mechanism dynamically monitors the status changes of microservice nodes in the service registration center, calculates the load trend of the reported call frequency and response time through a time series prediction algorithm, and generates future load forecast values; Based on the future load forecast, an API gateway is deployed at the entrance of the container cluster environment to perform unified routing, load balancing, and access control on the service access endpoints provided by the microservice units. A communication infrastructure platform is built through the distributed deployment of microservice units, dynamic node registration and discovery of service governance components, and unified access management of API gateways.

3. The data management method based on the instant messaging platform according to claim 2, characterized in that: Based on verified message packets, a hierarchical storage system is built to write communication data into a distributed relational database in real time. Attachments are stored through distributed file devices, achieving unified storage of structured and unstructured data, including: Write the text content, timestamp field, and user relationship field in the verified message package into the distributed relational database shard to generate a message record including a primary key; Based on the message record, extract the file attachment from the corresponding verified message package and upload the file attachment to the object storage service to generate a unique storage path; Bind the message record primary key to the storage path, create a primary key-path mapping table in the relational database, and form a unified storage structure in which text data and file attachments can be associated and accessed.

4. The data management method based on the instant messaging platform according to claim 3, characterized in that: Based on the unified stored data, a multi-dimensional indexing mechanism is established. Elastic search is used to create composite indexes for message and file metadata. This generates indexable data that can be searched based on multiple conditions such as time, user, session, keyword, and file type. This includes: Extract the message sending time field, user ID field, session ID field, keyword tag, and file media type field from the unified storage to form a structured metadata set; Based on the structured metadata collection, build a composite index structure in the Elasticsearch cluster, including time range filters, user session aggregators, and keyword analyzers; Continuously collect user query logs, dynamically adjust index field weights based on query frequency statistics, and generate pre-calculated index tables for high-frequency combined queries; By using a composite index structure and a pre-calculated index table, an index data structure is generated for performing multi-field combined queries on the message sending time field, user ID field, session ID field, keyword tag, and file media type field, thereby indexing the data.

5. The data management method based on the instant messaging platform according to claim 4, characterized in that: Implement fine-grained permission management based on indexable data, set permission levels through role-based access control, and generate permission-controlled data, including: In the data management device, a role-based access control role set is defined, including an administrator role, a department head role, and a common member role, and role definition information is generated; Based on the role definition information, an access control role identifier that allows access to the data entity is associated with each data entity in the indexable data to form a role access tag, so as to generate a set of indexable data entities with role access tags; Deploy a permission interception function at the query interface layer and input a collection of indexable data entities with role access tags into the permission interception function. When a user initiates a data query request through the query interface layer, the permission interception function identifies the current user's access control role and analyzes the user behavior log in real time to calculate the permission deviation degree. The permission interception function filters the data entities that meet the query conditions in the indexable data entity set with role access tags based on the identified current user access control role, returns the data entities retained after filtering as query results, and generates data controlled by permissions.

6. The data management method based on the instant messaging platform according to claim 5, characterized in that: Based on permission-controlled data, a periodic data archiving strategy is implemented to automatically migrate overdue data to cloud storage. During the archiving process, three fixed checkpoints are set in sequence. Dynamic checkpoint values ​​are generated based on the feedback from each checkpoint. Based on the dynamic checkpoint values, the deployment parameters of the microservice architecture are adjusted to achieve data traceability, including: Based on the predefined lifecycle configuration table of permission-controlled data, the data archiving task is automatically initiated when the preset archiving trigger conditions are detected. In the data archiving task execution process, checkpoints are preset for three key operation stages, including the data extraction stage checkpoint, which is set after the permission-controlled data extraction operation is completed from the source storage device; the encrypted transmission stage checkpoint, which is set after the data encryption is completed and the transmission operation to the cloud is started; and the cloud storage stage checkpoint, which is set after the data is successfully written to the cloud persistent storage device. During the execution of the data archiving task, when each set checkpoint is reached, the corresponding check operation is performed and the check parameters are generated, including the compression ratio deviation value, the transmission hash value and the number of storage copies; The compression ratio deviation, transmission hash value, and number of stored copies are used to form a dynamic check matrix that reflects the multi-stage integrity of the archiving task. The dynamic check matrix is ​​analyzed and matrix anomaly indicators reflecting the potential risks of the archiving process are calculated according to predefined rules. Based on the value of the matrix anomaly indicator, the number of running instances of the log audit service responsible for recording data operation details is dynamically increased, and data traceability is achieved through the increased running instances of the log audit service.

7. A data management system based on an instant messaging platform, the system implementing the method according to any one of claims 1 to 6, characterized in that: include: The architecture deployment module is used to build a communication infrastructure platform to achieve distributed deployment of instant messaging device functional services and dynamic node registration and discovery; A communication verification module is used to establish two-way communication based on the communication infrastructure platform through the web socket protocol and perform version and integrity verification on structured messages to generate verified message packets; The data storage module is used to build a hierarchical storage system based on verified message packets, write communication data into a distributed relational database in real time and store attachments through distributed file devices, realizing unified storage of structured and unstructured data; An index building module, which is used to create composite indexes for message and file metadata based on uniformly stored data, generating indexable data that can be searched based on multiple conditions such as time, user, session, keyword, and file type. The permission control module is used to implement fine-grained permission management based on indexable data, set permission levels through role-based access control, and generate permission-controlled data; The data archiving module is used to execute periodic data archiving strategies based on permission-controlled data, migrate expired data to cloud storage, and generate dynamic checksums based on feedback from checkpoints during the archiving process to adjust microservice architecture deployment parameters and achieve data traceability.

8. A computing device, characterized in that include: one or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images