A container-based service deployment solution system

Abstract

The application discloses a kind of service deployment solution systems based on containerization, belong to project deployment technical field, including service registry, Data data layer, MQ message module and file storage;Service registry carries out centralized management to MYSQL database, ECS host information and Rabbitmq configuration information, then host information is dynamically read in nginx layer, and flow distribution is realized by IP hash mode;MQ message module is used to process time-consuming tasks, to realize asynchronous waiting;File storage is used for file management;Local environment can be packaged into image file, uploaded to cloud and run, to realize cross-cloud platform support;Visual operation console, project one-key deployment, based on the environment provided by container guarantees service consistency and standardization, version control can be realized to application running environment;The splitting of service architecture solves the single-point problem existing in previous system architecture through the cluster between services.

Description

Technical Field

[0001] This invention belongs to the field of project deployment technology, specifically a service deployment solution system based on containerization. Background Technology

[0002] Docker container service deployment is a popular, easy-to-use, and continuously efficient project deployment solution. Often, due to differences between traditional development and cloud environments, coupled with complex deployment processes, uploaded applications encounter various errors and fail to run. By integrating Docker container technology with GitLab, this pain point is perfectly solved, while also optimizing and improving the system architecture and enhancing system high availability.

[0003] Traditional project deployment solutions have many problems and drawbacks:

[0004] 1. The versions of development and production environment services (nginx, php, mysql) are always inconsistent.

[0005] 2. The versions of the dependencies or extension libraries required by the runtime environment are not consistent.

[0006] 3. The configuration parameters of the operating environment were not kept consistent after optimization.

[0007] 4. The project version iteration and update is cumbersome and lacks good version control. When deployment errors occur, it is not possible to revert to the previous version in a timely manner.

[0008] 5. The system has poor horizontal scaling and upgrade capabilities when facing peak traffic. Summary of the Invention

[0009] To address the problems of the above solutions, this invention provides a service deployment solution based on containerization.

[0010] The objective of this invention can be achieved through the following technical solutions:

[0011] A containerized service deployment solution system includes a service registry, a data layer, an MQ message module, and file storage;

[0012] The service registry centrally manages the MySQL database, ECS host information, and RabbitMQ configuration information, and then dynamically reads the host information at the nginx layer to distribute traffic using IP hashing.

[0013] The tasks of the Data layer include:

[0014] Step SA1: Store the data entered in the background into a MySQL database for persistence;

[0015] Step SA2: Real-time synchronization and storage of hot data from the site to the Redis database;

[0016] Step SA3: For list search data, first connect to Elasticsearch. Before creating the index, perform word segmentation on relevant columns or words when creating the document based on the displayed data and search conditions. Then, write the corresponding document based on the displayed data.

[0017] Step SA4: By writing a search interface, the interface uses the keywords entered by the user to perform a matching search in Elasticsearch, and then outputs the results to the page for display.

[0018] The MQ message module is used to handle time-consuming tasks and implement asynchronous waiting;

[0019] File storage is used for file management.

[0020] Furthermore, the service registry monitors services via heartbeats. When a service encounters an anomaly, the abnormal host information is removed from the service list. Once the service is repaired, it is rejoined to the cluster, achieving dynamic scaling.

[0021] Furthermore, the working method of the MQ message module includes:

[0022] Step SB1: Create the corresponding switches and queues;

[0023] Step SB2: Add the corresponding task to the created queue;

[0024] Step SB3: After receiving the message, the business processing module retrieves the message according to the corresponding function design and processes the corresponding business logic; after successful consumption, the message is removed.

[0025] Furthermore, the working methods of file storage include:

[0026] Step SC1: Activate different account information according to the project;

[0027] Step SC2: After the account is activated, create the corresponding file storage directory according to the project name and set the restrictions on uploaded files;

[0028] Step SC3: Perform the corresponding file operations according to the corresponding interface provided by the file server.

[0029] Furthermore, specific methods for identifying hotspot data include:

[0030] Based on big data analysis, several target items are obtained. The obtained target items are summarized to establish corresponding initial statistical tables. Data items within the enterprise are identified. The obtained data items are matched with the initial statistical tables. Data items in the initial statistical tables are marked as matching items. The obtained matching items are summarized to obtain a matching table. Hotspot data are determined based on the obtained matching table.

[0031] Furthermore, methods for determining hotspot data based on the obtained matching table include:

[0032] The data in the site is identified in real time based on the established matching table, and data belonging to the matching table is marked as hot data.

[0033] Furthermore, methods for determining hotspot data based on the obtained matching table include:

[0034] The process involves identifying the update interval of data corresponding to each matching item in the matching table, obtaining several historical update times, analyzing these historical update times to obtain the corresponding initial time and amplification coefficient set; identifying the data items and update times corresponding to the data in the site, calculating the similarity between the data items and each matching item in the matching table, identifying the matching item with the highest similarity and marking it as a similar item, obtaining the corresponding amplification coefficient based on the obtained similar items and marking the obtained amplification coefficient as FD, marking the obtained initial time as SC, matching the corresponding correction coefficient based on the corresponding similarity and marking the obtained correction coefficient as β; calculating the corresponding marking time according to the formula SW=SC×FD×β+Sd, where Sd is the dynamic time, marking the update time and marking time corresponding to the data item, and marking the data item corresponding to the data item as hot data when the update time is greater than the marking time.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] The local environment can be packaged into an image file and uploaded to the cloud for operation, enabling cross-cloud platform support; a visual operation console allows for one-click project deployment, continuous CI deployment and testing; the container-based environment ensures service consistency and standardization, and version control can be implemented for the application runtime environment; compared to traditional applications, container services start up quickly, consume less memory, and can run more application examples under the same conditions, making fuller use of system resources; the service architecture splitting, through the clustering of services, solves the single point of failure problem that existed in the previous system architecture. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a block diagram illustrating the principle of the present invention. Detailed Implementation

[0039] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0040] like Figure 1 As shown, this is a containerized service deployment solution system. This system can be implemented as a Docker container service environment cluster using multiple or a single ECS instance. Functional services are split, reducing coupling between services. Environment deployment is achieved by uploading a pre-packaged Docker image to the cloud and running it directly. No human intervention is required during the environment deployment process. After the script execution is complete, code push deployment can be achieved by configuring a Git repository.

[0041] Specifically, it includes: service registry, data layer, MQ message module, and file storage;

[0042] The service registry centrally manages the MySQL database, ECS host information, and RabbitMQ configuration information (Consul registry), and then dynamically reads the host information at the Nginx layer, distributing traffic through IP hashing.

[0043] The service registry listens for services via heartbeats. When a service encounters an anomaly, the abnormal host information is removed from the service list. Once the service is repaired, it is rejoined to the cluster, enabling dynamic scaling.

[0044] The tasks of the Data layer include:

[0045] Step SA1: Store the data entered in the background into a MySQL database for persistence;

[0046] Step SA2: Real-time synchronization and storage of hot data from the site to the Redis database; this helps accelerate data query and rendering.

[0047] Step SA3: For list search data (keyword search), first connect to Elasticsearch. Before creating the index (table), you need to perform word segmentation on the relevant columns or words when creating the document based on the displayed data and search conditions. Then, write the corresponding document (record) based on the displayed data.

[0048] Word segmentation is performed by using existing word segmentation technologies or built-in word segmentation systems.

[0049] Step SA4: By writing a search interface, the interface uses the keywords entered by the user to perform a matching search in Elasticsearch and then outputs the results to the page for display.

[0050] Hotspot data refers to data that is not frequently updated or updated infrequently. Specific methods for determining hotspot data include:

[0051] In one embodiment, based on big data analysis, several target items are obtained, namely, by using current big data analysis to identify which data items are not frequently updated, such as personal information, article categories, and industry information; the obtained target items are summarized to establish a corresponding initial statistical table, the data items within the enterprise are identified, the obtained data items are matched with the initial statistical table, the data items in the initial statistical table are marked as matching items, and the obtained matching items are summarized to obtain a matching table;

[0052] The data in the site is identified in real time based on the established matching table, and data belonging to the matching table is marked as hot data.

[0053] In another embodiment, because direct matching through a matching table has certain limitations—it can only determine the data corresponding to the matching items in the matching table, and cannot mark certain data types that have not been updated for a long time within the enterprise—this embodiment is provided to solve this problem. The specific method includes:

[0054] The process involves identifying the update interval of data corresponding to each matching item in the matching table (referring to the update interval of data corresponding to each matching item within the enterprise), obtaining several historical update times, analyzing these historical update times to obtain the corresponding initial time and amplification coefficient set; identifying the data items and update times corresponding to the data in the site, calculating the similarity between the data items and each matching item in the matching table, and performing similarity calculations based on the data items, which can be done using existing similarity algorithms; identifying the matching item with the highest similarity and marking it as a similar item, obtaining the corresponding amplification coefficient based on the obtained similar item, marking the obtained amplification coefficient as FD, marking the obtained initial time as SC, matching the corresponding correction coefficient based on the corresponding similarity, and marking the obtained correction coefficient as β; calculating the corresponding marking time according to the formula SW=SC×FD×β+Sd, where Sd is the dynamic time, marking the update time and marking time corresponding to the data item, and marking the data item corresponding to the data item when the update time is greater than the marking time.

[0055] The method for analyzing the obtained historical update times is as follows: a corresponding time analysis model is established based on a CNN or DNN network. The corresponding training set is set manually for training. The historical update times are analyzed using the successfully trained time analysis model to obtain the corresponding initial time and amplification coefficient set. The specific amplification coefficient is determined by the corresponding matching item. If the data item to be analyzed has the highest similarity with the user information, then the amplification coefficient is the amplification coefficient corresponding to the user information, which is used to amplify the initial time. The initial time is a representative time obtained through the analysis of historical update times, because the corresponding historical update times may vary greatly for different matching items.

[0056] The corresponding correction coefficient is matched according to the corresponding similarity to obtain the possible similarity and form the corresponding similarity interval. The similarity interval is divided into several smaller intervals by manual means, and a corresponding correction coefficient is set for each smaller interval. Matching is then performed based on the corresponding similarity value.

[0057] Methods for determining dynamic time: Dynamic time is mainly related to the historical update time of the data item. If the data item is frequently updated within a certain period, but does not update for a long time in the current period, it cannot be identified as hot data. Specifically, a corresponding dynamic analysis model is built based on a CNN or DNN network. The corresponding training set is built manually for training. The dynamic analysis model is analyzed after successful training to obtain the corresponding dynamic time. If it cannot be analyzed, the dynamic time is determined to be a dynamic fixed value, which is set by the expert group through discussion.

[0058] The MQ (Message Queue) module is primarily used to handle time-consuming tasks and implement asynchronous waiting. This prevents excessive site traffic from causing process (or thread) blocking. It is mainly used in scenarios such as users sending SMS messages, message notifications, data synchronization, and order status modifications.

[0059] Specific working methods include:

[0060] Step SB1: Create the corresponding switches and queues;

[0061] Step SB2: Deploy the corresponding task (production message) to the created queue;

[0062] Step SB3: After receiving the message, the business processing module (message consumer) retrieves the message according to the corresponding functional design and processes the corresponding business logic; after successful consumption, it notifies the upstream to remove the message. At this point, the business processing of the message is complete.

[0063] The business processing module is the original system's built-in processing module, i.e., the existing module, used to perform corresponding business processing.

[0064] This enables asynchronous business logic processing, which not only reduces the pressure on downstream interfaces,

[0065] This also avoids traffic congestion and program crashes caused by excessive upstream user access.

[0066] File storage is used for file management, and specific methods include:

[0067] Step SC1: Activate different account information according to the project;

[0068] Step SC2: After the account is activated, create the corresponding file storage directory according to the project name and set the restrictions on uploaded files, such as size limits and other configurations. These settings should be made manually according to the actual situation.

[0069] Step SC3: Perform the corresponding file operations according to the corresponding interface provided by the file server; such as uploading, deleting, etc.

[0070] By using OSS file storage service, the original file storage machines can be replaced, achieving separation between business and static resources. Centralized management of site files through OSS improves management efficiency, and CDN enables fast access to static resources.

[0071] The project environment is easy to deploy; simply upload the corresponding Docker image to the server and run it to complete the deployment.

[0072] Setting up a PHP environment requires installing PHP, Php-fpm, Nginx, MySQL, Redis, and related PHP extension packages in sequence. After containerization with Docker, all services and dependencies are packaged into a container image using scripts. When deploying the PHP environment, the image is simply pulled to the server and run, achieving one-click deployment.

[0073] The MQ messaging module utilizes a cluster of multiple RabbitMQ instances to ensure high availability for SMS and email sending, message push, and Nginx log collection services. After integrating the MQ messaging system, users only need to manually trigger the sending of emails or verification codes; the business logic of whether the sending was successful is offloaded to the downstream system. Upstream users simply wait for the result. This design transforms synchronous waiting into asynchronous waiting, reducing system load and enhancing the user experience.

[0074] Files are uploaded to a dedicated file server, replacing traditional file upload methods, which effectively improves image access speed. Meanwhile, information search replaces traditional SQL fuzzy matching searches by deploying Elasticsearch (ES), accelerating query speed and significantly improving the customer search experience.

[0075] Simultaneously, it integrates with the ELK (Elasticsearch, Logstash, and Kibana) log collection system, providing a visual platform for service monitoring and troubleshooting. Server-side log information is displayed in the integrated log system, replacing raw file logs. This provides visual monitoring of server system stability and performance. Specific problems can be located promptly and intuitively, thereby improving operational efficiency and work efficiency.

[0076] Git+SSH+Rsync enables one-click code deployment, replacing the traditional FTP code update method, reducing the error rate of code. At the same time, Git's version control allows for quick rollback of code versions when project deployment errors occur.

[0077] The Web Service layer implements a cluster of multiple ECS instances, which effectively improves the website's concurrency performance compared to a traditional monolithic architecture.

[0078] The above formulas are all numerical calculations after removing dimensions. The formulas are obtained by software simulation based on a large amount of data and are closest to the real situation. The preset parameters and preset thresholds in the formulas are set by those skilled in the art according to the actual situation or obtained by simulation based on a large amount of data.

[0079] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A container-based service deployment solution system, characterized by, It comprises a service registry center, a Data data layer, an MQ message module and a file storage; The service registry center centrally manages MYSQL database, ECS host information and Rabbitmq configuration information, and then dynamically reads the host information at the nginx layer to realize traffic distribution through IP hash method; The working content of the Data data layer comprises: Step SA1: storing the data input in the background into the MySQL database for persistence; Step SA2: storing the hot data in the site into the Redis database in real time; Step SA3: for the data of list search, first connect to ES, and before creating an index, according to the displayed data and search conditions, perform word segmentation on the related columns or words when creating a document, and then write the corresponding document according to the displayed data; Step SA4: by writing a search interface, the interface user inputs a keyword, and the corresponding data in ES is searched and matched, and then output to the page for display; The MQ message module is used for processing time-consuming tasks and realizing asynchronous waiting; The file storage is used for file management; The method for specifically determining the hot data comprises: Based on big data analysis, a plurality of target items are obtained, the obtained target items are summarized to establish a corresponding initial statistical table, data items possessed by the data in the enterprise are identified, the obtained data items are matched with the initial statistical table, the data items possessed by the initial statistical table are marked as matching items, the obtained matching items are summarized to obtain a matching table, and the hot data is determined according to the obtained matching table; The method for determining the hot data according to the obtained matching table comprises: The update interval time of the data corresponding to each matching item in the matching table is identified, a plurality of historical update times are obtained, the obtained historical update times are analyzed to obtain a corresponding initial time and a set of amplification coefficients; the data items and the update time corresponding to the data in the site are identified, the similarity between the data items and each matching item in the matching table is calculated, the matching item with the highest similarity is identified and marked as a similar item, the corresponding amplification coefficient is obtained according to the similar item, the obtained amplification coefficient is marked as FD, the obtained initial time is marked as SC, the corresponding correction coefficient is obtained according to the similarity, the obtained correction coefficient is marked as β, and the corresponding marking time is calculated according to the formula SW=SC×FD×β+Sd, wherein Sd is a dynamic time, the update time corresponding to the data item is marked with the marking time, and when the update time is greater than the marking time, the data corresponding to the data item is marked as hot data; The initial time is a representative time analyzed through the historical update time; The dynamic time is mainly related to the historical update time of the data item, the data corresponding to the data item is frequently updated in a period of time, but in the current period, it is not updated for a long time, so it cannot be determined as hot data, specifically, a dynamic analysis model is established based on a CNN network or a DNN network, a training set is established in an artificial manner for training, the dynamic analysis model after successful training is used for analysis, and the corresponding dynamic time is obtained, if the analysis fails, the dynamic time is determined as a dynamic fixed value, and a group of experts discusses and sets the dynamic fixed value.

2. The container-based service deployment solution system according to claim 1, wherein, The service registration center listens to the service in a heartbeat mode, removes abnormal host information from the service list when the service is abnormal, and re-joins the cluster after being repaired, so that dynamic expansion and contraction are realized.

3. The container-based service deployment solution system of claim 1, wherein, The working method of the MQ message module includes: Step SB1: creating an exchange and a queue corresponding to a function; Step SB2: putting a corresponding task into the created queue; Step SB3: after the business processing module receives the message, the message is taken out according to the corresponding function design, the corresponding business logic is processed, and the message is removed after successful consumption.

4. The container-based service deployment solution system of claim 1, wherein, The working method of the file storage includes: Step SC1: creating different account information according to a project; Step SC2: after the account is opened, a corresponding file storage directory is created according to the project name, and a limit condition for uploading files is set; Step SC3: corresponding file operations are performed according to the corresponding interface provided by the file server.

5. The container-based service deployment solution system of claim 1, wherein, The method for determining hot data according to the obtained matching table includes: Real-time identification is performed on the data in the site according to the established matching table, and the data classified into the matching table is marked as hot data.

Images