Data storage and reading method, device, equipment and medium

Abstract

The present invention discloses a method for storing and reading data, comprising the steps of: classifying article data, obtaining first information and second information of the article data; adding the first information to a Kafka message queue, and storing the first information in a first server after the Kafka message queue consumes the first information; storing the second information in different data tables of a second server respectively, and generating third information and storing it in a third server after adding style serialization data to the second information; completing consistency comparison of the data stored in the second server, the first server, and the third server respectively according to a set time threshold; obtaining the first information from the first server, or obtaining the second information from the second server, or obtaining the third information from the third server according to different application program interfaces. The present application can realize the safe storage of article data and provide efficient article data query.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a method, device, electronic device and storage medium for data storage and reading. Background Art

[0002] As more and more articles are published on the Internet, and their contents are becoming richer and richer, relevant servers are required to store article data stably and provide article information to search users quickly and accurately. Servers in the prior art generally store Internet articles directly in a database, provide data ports to users, and allow users to directly read corresponding article data from the database. This method can cope with small traffic, but it is easy to cause server downtime when a large number of users access the site at the same time.

[0003] For this problem, the existing technology generally adopts different database types to improve data access and data storage security, but this method can only provide limited room for improvement and cannot fundamentally change the server downtime problem. Another technology is to store part of the information in the cache to improve the efficiency of user access to article data, but this method has poor time efficiency. As the number of Internet articles increases, the memory usage will increase, which will also affect the computer's processing speed.

[0004] Therefore, a method is needed that can change the storage method of article data to ensure the stability of article data storage, while improving the fluency of users' rapid access and retrieval of article information without downtime. Summary of the invention

[0005] To this end, the present invention provides a method, device, electronic device and storage medium for storing and reading data, in an effort to solve or at least alleviate at least one of the above problems.

[0006] According to one aspect of the present invention, a method for storing and reading data is provided, wherein the method stores summary information and full text information of article data in different servers respectively, thereby achieving stable storage and fast reading of article data. The method comprises the steps of:

[0007] The article data is classified to obtain first information and second information of the article data, wherein the first information is summary information of the article data, and the second information is full-text information of the article data; according to the first information, the first information is added to the Kafka message queue, and after the Kafka message queue consumes the first information, the first information is stored in the first server, which is the search server ElasticSearch; according to different uses of the second information, the second information is respectively stored in different data tables of the second server, and after adding style serialization data to the second information in different data tables of the second server, the third information is generated and stored in the third server, the second server is the data server SqlServer cluster of the active-active cluster, and the third server is the data server HBase; according to the set time threshold, the consistency comparison of the data stored in the first server and the second server, and the second server and the third server is respectively completed; according to different application program interfaces, the first information is obtained from the first server, or the second information is obtained from the second server, or the third information is obtained from the third server.

[0008] Optionally, the step of classifying the article data and obtaining the first information and the second information of the article data includes: obtaining the article data published by the merchant, the article data including the article title, summary text information, content text information, picture information, article serial number, merchant serial number, and article data validity time information; based on the article data published by the merchant, obtaining the first information and the second information in the article data, the first information is the summary information of the article data, the summary information includes: article serial number, summary text information, the first picture in the picture information, the merchant serial number, and article data validity time information, and the second information is the full-text information of the article data, the full-text information includes: article title, content text information, picture information, article serial number, merchant serial number, and article data validity time information.

[0009] Optionally, the step of adding the first information to the Kafka message queue based on the first information, and storing the first information in the first server after the Kafka message queue consumes the first information includes: adding the first information to the Kafka message queue, and obtaining the first information process list of the Kafka message queue; obtaining the monitoring information consumed by the first information process list of the Kafka message queue based on the first information process list of the Kafka message queue; obtaining the data format of the consumed first information based on the monitoring information consumed by the first information process list of the Kafka message queue, converting it into the data storage format of the first server, and storing the consumed first information in the first server, the first server being the search server ElasticSearch.

[0010] Optionally, the step of adding the first information to the Kafka message queue based on the first information, and storing the first information in the first server after the Kafka message queue consumes the first information also includes: adding one or more first information stored in the first server to the Kafka message queue to obtain a first information process list of the Kafka message queue; obtaining monitoring information consumed by the first information process list of the Kafka message queue based on the first information process list of the Kafka message queue; and updating the consumed first information in the first server based on the monitoring information consumed by the first information process list of the Kafka message queue.

[0011] Optionally, according to the different uses of the second information, the second information is stored in different data tables of the second server respectively, and after adding style serialization data to the second information in the different data tables of the second server, the step of generating third data and storing it in the third server includes: obtaining all application interfaces connected to the second information and the data table of each application interface in the second server; according to all application interfaces connected to the second information and the data table of each application interface in the second server, the second information is stored in the data table of the second server corresponding to the application interface; according to the second information stored in the data table of each application interface in the second server, style serialization data is added to the second information to obtain third information, and the third information is compatible with the data format of the corresponding application interface; and the third information is stored in the third server corresponding to the application interface.

[0012] Optionally, the second server is a data server SqlServer cluster of an active-active cluster, and the data server SqlServer cluster of the active-active cluster is two sets of data server SqlServer clusters deployed in two regions respectively, and the two sets of data server SqlServer clusters store the second information simultaneously. Among the two sets of data server SqlServer clusters, the first set of data server SqlServer cluster is a production cluster, used to provide data reading and writing services, and the second set of data server SqlServer cluster is a backup cluster, used to take over the data reading service of the first set of data server SqlServer cluster when the first set of data server SqlServer cluster fails; the two sets of data server SqlServer clusters are in a master-slave mode, including a master server and multiple slave servers, the master server provides a write data operation, used to write the second information to the second server, and the slave server provides a read data operation, used to read the second information from the second server.

[0013] Optionally, the step of comparing the consistency of data stored in the first server and the second server includes: updating the first information stored in the first server and the second information stored in the second server; obtaining the first information from the first server and the second information from the second server respectively; determining whether there is corresponding first information for the second information; if not, writing or deleting the first information that does not correspond to the second information in the first server.

[0014] Optionally, the step of comparing the consistency of the data stored in the second server and the third server includes: updating the second information stored in the second server and the third information stored in the third server; setting a timestamp, obtaining the second information in a set number of second servers within a timestamp range, and the third information in the third server; judging whether the second information has corresponding third information; if not, writing or deleting the third information that does not correspond to the second information in the third server.

[0015] According to another aspect of the present invention, a device for storing and reading data is disclosed. The device realizes stable storage and fast reading of article data by storing summary information and full-text information of article data in different servers respectively. The device comprises: an article data processing module, which is used to classify article data and obtain first information and second information of article data, wherein the first information is summary information of the article data and the second information is full-text information of the article data; an information storage module, which is used to add the first information to a Kafka message queue based on the first information, and store the first information to a first server after the Kafka message queue consumes the first information, wherein the first server is a search server ElasticSearch; According to the different uses of the second information, the second information is stored in different data tables of the second server respectively, and after adding style serialization data to the second information in the different data tables of the second server, the third information is generated and stored in the third server, the second server is a data server SqlServer cluster of the active-active cluster, and the third server is a data server HBase; the information comparison module is used to complete the consistency comparison of the data stored in the first server and the second server, and the second server and the third server respectively according to the set time threshold; the information reading module is used to obtain the first information from the first server according to different application program interfaces, or obtain the second information from the second server, or obtain the third information from the third server.

[0016] According to another aspect of the present invention, a computing device is provided, comprising: one or more processors; and a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the methods for storing and reading data as described above.

[0017] According to another aspect of the present invention, there is provided a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, enable the computing device to perform any of the methods for storing and reading data as described above.

[0018] According to the data storage and reading scheme of the present invention, the first information and the second information of the article data are obtained by classifying the article data; the first information is added to the Kafka message queue, and after the Kafka message queue consumes the first information, the first information is stored in the first server; the second information is stored in different data tables of the second server respectively, and after adding style serialization data to the second information in different data tables of the second server, the third information is generated and stored in the third server; according to the set time threshold, the consistency comparison of the data stored in the first server and the second server, and the second server and the third server is completed respectively; according to different application program interfaces, the first information is obtained from the first server, or the second information is obtained from the second server, or the third information is obtained from the third server. The present application can realize the safe storage of article data and provide efficient article data query. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] To achieve the above and related purposes, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings, which indicate various ways in which the principles disclosed herein can be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. The above and other purposes, features and advantages of the present disclosure will become more apparent by reading the following detailed description in conjunction with the accompanying drawings. Throughout the present disclosure, the same reference numerals generally refer to the same parts or elements.

[0020] Figure 1 A schematic diagram showing the structure of a computing device 100 according to an embodiment of the present invention; and

[0021] Figure 2 A flowchart showing a method 200 for storing and reading data according to an embodiment of the present invention; and

[0022] Figure 3 A schematic structural diagram of a data storage and reading device 300 according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0023] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0024] Figure 11 is a block diagram of an example computing device 100. In a basic configuration 102, computing device 100 typically includes a system memory 106 and one or more processors 104. A memory bus 108 may be used for communication between processor 104 and system memory 106.

[0025] Depending on the desired configuration, the processor 104 can be any type of processor, including but not limited to: a microprocessor (μP), a microcontroller (μC), a digital information processor (DSP), or any combination thereof. The processor 104 can include one or more levels of cache such as a first-level cache 110 and a second-level cache 112, a processor core 114, and registers 116. An example processor core 114 can include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. An example memory controller 118 can be used with the processor 104, or in some implementations, the memory controller 118 can be an internal part of the processor 104.

[0026] Depending on the desired configuration, the system memory 106 can be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. The system memory 106 may include an operating system 120, one or more applications 122, and program data 124. In some embodiments, the application 122 can be arranged to operate on the operating system using the program data 124. In some embodiments, the computing device 100 is configured to perform a method 200 for storing and reading data, which can achieve stable storage and fast reading of article data by storing summary information and full-text information of article data in different servers respectively, and the program data 124 includes instructions for executing the method 200.

[0027] The computing device 100 may also include an interface bus 140 that facilitates communication from various interface devices (e.g., output devices 142, peripheral interfaces 144, and communication devices 146) to the basic configuration 102 via the bus / interface controller 130. Example output devices 142 include a graphics processing unit 148 and an audio processing unit 150. They can be configured to facilitate communication with various external devices such as a display or speakers via one or more A / V ports 152. Example peripheral interfaces 144 may include a serial interface controller 154 and a parallel interface controller 156, which may be configured to facilitate communication with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices) or other peripherals (e.g., printers, scanners, etc.) via one or more I / O ports 158. Example communication devices 146 may include a network controller 160, which may be arranged to facilitate communication with one or more other computing devices 162 via a network communication link via one or more communication ports 164.

[0028] A network communication link can be an example of a communication medium. Communication media can generally be embodied as computer-readable instructions, data structures, program modules in modulated data signals such as carrier waves or other transmission mechanisms, and can include any information delivery medium. A "modulated data signal" can be a signal in which one or more of its data sets or its changes can be performed in a manner that encodes information in the signal. As a non-limiting example, communication media can include wired media such as wired networks or dedicated networks, and various wireless media such as sound, radio frequency (RF), microwaves, infrared (IR) or other wireless media. The term computer-readable media used here can include both storage media and communication media. In some embodiments, one or more programs are stored in a computer-readable medium, and the one or more programs include instructions for executing certain methods, such as according to an embodiment of the present invention, a computing device 100 executes a method 200 for data storage and reading by the instructions.

[0029] The computing device 100 may be implemented as part of a small-sized portable (or mobile) electronic device, such as a cellular phone, a personal digital assistant (PDA), a personal media player device, a wireless network browsing device, a personal head-mounted device, an application-specific device, or a hybrid device that may include any of the above functions. The computing device 100 may also be implemented as a personal computer including desktop and notebook computer configurations.

[0030] Figure 2 FIG. 2 is a flow chart of a method 200 for storing and reading data according to an embodiment of the present invention. Figure 2As shown, the method 200 achieves stable storage and fast reading of article data by storing summary information and full-text information of article data in different servers respectively. The method 200 starts at step S210, classifies the article data, and obtains first information and second information of the article data, wherein the first information is the summary information of the article data, and the second information is the full-text information of the article data.

[0031] Specifically, article data refers to articles about automobile sales, automobile product introductions and other automobile information published on the Internet by automobile manufacturers or sellers. The article data contains a large amount of text and pictures. Each article data contains the article title and the article summary text. The network automatically generates the article serial number, the merchant serial number and the set article data effective time information. Therefore, for an article data, the article data can be divided into two parts. One part is to simply obtain the theme of the article, that is, the summary information of the article, and the other part is the full-text data of the article, which is used for users to understand the full-text information of the article data. By dividing the article data into summary information and full-text information, it helps to improve the user's search efficiency. For example, if a user wants to retrieve a car discount article data, then he can obtain the article data he wants by retrieving the summary information without obtaining the full-text information of the article data. In this way, on the one hand, the amount of data retrieved will be greatly reduced, and on the other hand, it is convenient for users to quickly obtain the target article.

[0032] Specifically, in one embodiment of the present application, the step of classifying the article data and obtaining the first information and the second information of the article data includes:

[0033] Get the article data published by the merchant, the article data includes the article title, abstract text information, content text information, picture information, article serial number, merchant serial number, and article data validity period information; specifically, when the merchant publishes data on the Internet through the backend management system, it is necessary to complete the production of the following article data content: article title, which is displayed to users at the top of the corresponding web page, and the article title is generally a high-level summary of the article content; abstract text information, which is used to summarize the main text content of the article data in brief language, generally dozens of words; content text information, all text information of the article data, describing in detail the entire content of the article published by the merchant; picture information, which may include one or more pictures A piece of text used to assist in introducing the articles published by merchants; an article serial number, when a merchant enters the backend management system to publish an article, the backend management system automatically generates an article serial number; a merchant serial number, when a merchant completes the registration of the corresponding content in the backend management system of the Internet platform, it automatically obtains a unique serial number corresponding to the merchant; article data effective time information, in order to improve the timeliness of merchant article releases, for example, a certain car is sold at a limited time discount, then the article data effective time information can be set from the release time to the discount end time. After the article data effective time information is reached, the article is automatically deleted or the corresponding content is no longer displayed to Internet customers.

[0034] According to the article data published by the merchant, the first information and the second information in the article data are obtained, the first information is the summary information of the article data, the summary information includes: article serial number, summary text information, the first picture in the picture information, the merchant serial number, and the effective time information of the article data, and the second information is the full text information of the article data, and the full text information includes: article title, content text information, picture information, article serial number, merchant serial number, and effective time information of the article data. Specifically, after the merchant publishes the article data on the background management system of the Internet platform, the article data can be divided into the first information and the second information according to the corresponding data module of the article data and stored separately, the first information is the summary information of the article data, that is, the core content of the article information can be roughly understood through the first information, but the detailed content cannot be understood. At the same time, because the first information contains relatively little data, it is convenient for Internet users to quickly search and obtain the core content of the article data. The second information is the full text information of the article data, which is the detailed content of the article data. After the Internet user obtains the second information, the full content of the article data can be understood. In a specific implementation, for example, after a user searches for a keyword, the search system can quickly obtain a first information list containing the keyword in the first information. Since the first information contains relatively little data, the search time for each first information is relatively short. Therefore, when searching for the same number of first information lists and article data, the efficiency of searching for the first information will be greatly improved. Internet users can obtain summary information of interest by searching the first information lists, and can obtain the second information corresponding to the first information by clicking on a certain first information list, that is, obtaining the full-text information of the article data. Since the data link is highly directional, the speed of obtaining the second information is also relatively fast, thereby more quickly achieving the purpose of Internet users obtaining target article data.

[0035] Through step S220, according to the first information, the first information is added to the Kafka message queue, and after the Kafka message queue consumes the first information, the first information is stored in the first server, and the first server is the search server ElasticSearch.

[0036] Specifically, in order to improve the stability of data storage, the first information is added to the Kafka message queue, and the first information to be stored is queued through the Kafka message queue. After the first information in each Kafka message queue is consumed, it is stored in the first server. Since the first information in the Kafka message queue is consumed one by one in a queue manner, when multiple merchants publish the first information of multiple article data at the same time, the Kafka message queue will add these first information to the list, so that the first information is stored one by one according to the list when storing, and there will be no storage confusion. The first server is the search server ElasticSearch. The search server ElasticSearch is a search server based on Lucene, which provides a distributed multi-user full-text search engine based on the RESTful web interface. The search server ElasticSearch is released as open source under the Apache license terms and is a popular enterprise-level search engine.

[0037] Specifically, in an embodiment of the present application, the step of adding the first information to a Kafka message queue based on the first information, and storing the first information in a first server after the Kafka message queue consumes the first information includes:

[0038] Add the first information to the Kafka message queue and obtain the first information process list of the Kafka message queue; specifically, after adding the first information to the Kafka message queue, the first information process list of the Kafka message queue after the current first information is added can be obtained. When multiple first information are added to the Kafka message queue at the same time, the Kafka message queue will generate a first information process list according to the set rules. The function of the first information process list is to queue multiple first information that need to be stored, so that the first information waits to be processed in the order of the list.

[0039] According to the first information process list of the Kafka message queue, the monitoring information of the first information process list being consumed of the Kafka message queue is obtained; specifically, the first information process list being consumed means that a certain first information in the first information process list is processed, and the processing at this time is to notify the first message to perform a storage operation. Because the first server does not know which first message in the Kafka message queue is consumed, it is necessary to set a process to monitor in real time which first information in the first information process list is consumed. After the first server receives the monitoring information that the first information process list of the Kafka message queue is consumed, the first server obtains the consumed first information through the information that the first information in the first information process list is consumed.

[0040] According to the monitoring information consumed by the first information process list of the Kafka message queue, the data format of the first information consumed is obtained and converted into the data storage format of the first server, and the first information consumed is stored in the first server, which is the search server ElasticSearch. Specifically, after the first server obtains the first information consumed by the first information process list of the Kafka message queue, it obtains the content of the first information and converts the data format of the first information into the data format stored by the first server. After completing the format conversion, the consumed first information can be stored in the first server. At this time, the first server continues to wait for the monitoring information of the first information process list of the Kafka message queue sent by the monitoring process, and continues to store the next consumed first information.

[0041] Specifically, in one embodiment of the present application, the step of adding the first information to a Kafka message queue based on the first information, and storing the first information in a first server after the Kafka message queue consumes the first information further includes:

[0042] Add one or more first information stored in the first server to a Kafka message queue, and obtain a first information process list of the Kafka message queue;

[0043] According to the first information process list of the Kafka message queue, obtaining the monitoring information consumed by the first information process list of the Kafka message queue;

[0044] According to the monitoring information consumed by the first information process list of the Kafka message queue, the first server updates the consumed first information.

[0045] Specifically, the above steps are operations for the merchant to modify or delete the published article data. The merchant sends the first information that needs to be modified or deleted to the Kafka message queue. Since the article data has a corresponding article serial number, when an article data needs to be modified, it is only necessary to obtain the article serial number. After sending the modified or deleted first information to the Kafka message queue, the first information in the first information process list is consumed, and then the first information is re-stored in the first server, completing the update of the first information with the same article serial number. If it is a deletion instruction for the first information, the first information corresponding to the article serial number stored in the first server is deleted.

[0046] Through step S230, according to different uses of the second information, the second information is respectively stored in different data tables of the second server, and after adding style serialization data to the second information in the different data tables of the second server, the third information is generated and stored in the third server, the second server is the data server SqlServer cluster of the active-active cluster, and the third server is the data server HBase.

[0047] Specifically, since different terminal user interfaces require different data styles, for example, the data styles that can be received by Windows operating system, Andiron operating system and IOS operating system are different, and some special terminal devices are also included, for each type of terminal, the second server sets a corresponding data table, and the second server stores the second data in different data tables according to different terminal applications. In the second information of each data table, the second information is added with the style serialization data according to the corresponding terminal format to generate the third information. At this time, the third information is the data style that can be directly read by the corresponding terminal device. The third information is stored in the third server again. The second server is a data server SqlServer cluster of a dual-active cluster, that is, the first data server SqlServer cluster is set to store the second information, and then the second data server SqlServer cluster is set in a different place as a backup of the first data server SqlServer cluster. When the first data server SqlServer cluster fails and cannot be used, the second data server SqlServer cluster takes over the first data server SqlServer cluster to provide data services. The third server is a data server HBase, which is a distributed, column-oriented open source database.

[0048] Specifically, in one embodiment of the present application, according to different uses of the second information, the second information is stored in different data tables of the second server respectively, and after adding style serialization data to the second information in the different data tables of the second server, the step of generating third data and storing it in the third server includes:

[0049] Obtain all application program interfaces connected to the second information and the data table of each application program interface in the second server; specifically, since each application program interface corresponds to a terminal device, the second server provides a data table for each application program interface according to the difference of the application program interfaces. That is to say, when the second server is connected to different application program interfaces, since the terminal devices connected to the application program interfaces are different, the data formats that can be received by different terminal devices may be different, but the data formats that can be received by the corresponding application program interfaces are the same. Therefore, according to the style of the application program interface connected to the second server, data tables with the same number of application program interface styles are set, and each data table corresponds to an application program interface of one style, so that the application program interface can obtain the second information in a data format suitable for it.

[0050] According to all the application interfaces connected to the second information and the data table of each application interface in the second server, the second information is stored in the data table of the second server corresponding to the application interface; specifically, since the second server sets up a data table according to the style of each application interface, and each application interface is connected to a terminal device, when a terminal device needs to obtain the second information in the second server, the second information needs to be stored in the data table of the second server corresponding to the application interface connected to the terminal device. Since the application interface corresponds to the data table of the second server one by one, when the second information is stored, the second information needs to be stored in the data table of the second server corresponding to the application interface.

[0051] Based on the second information stored in the data table of the second server by each application interface, the second information is added with style serialization data to obtain third information, and the third information is adapted to the data format of the corresponding application interface; specifically, after the second information is classified and stored in the data table of the second server, a corresponding storage area of ​​the third server is set for the data table of each second server, and the second information stored in the data table of each second server is added with style serialization data to generate third information. The purpose of adding style serialization data at this time is to adapt the data format of the third information to the data format of the corresponding application interface, so that the terminal device connected to the application interface program can directly read the third information corresponding to the application interface program.

[0052] The third information is stored in the third server corresponding to the application interface. Specifically, similar to the second server, the third server also provides a storage space for each style of application interface, and the third information is stored in the third server corresponding to the application interface, that is, the third information is stored in the storage space corresponding to the application interface program in the third server, so that the terminal device of the Internet user can obtain the corresponding third information, that is, the article data, according to the corresponding application interface program. It should be noted that the data content between the second information and the third information has not changed. The second information and the third information are both full-text information of the article data. The difference between the second information and the third information is the different data format. The second information only stores the full-text information of the article data in the data table of the second server corresponding to the application interface program, while the third information is to adaptively change the data of the second information stored in the data table of the second server to the data format of the corresponding application interface program.

[0053] Specifically, in one embodiment of the present application, the second server is a data server SqlServer cluster of an active-active cluster, the data server SqlServer cluster of the active-active cluster is two sets of data server SqlServer clusters deployed in two regions respectively, the two sets of data server SqlServer clusters store the second information simultaneously, and of the two sets of data server SqlServer clusters, the first set of data server SqlServer cluster is a production cluster, used to provide data reading and writing services, and the second set of data server SqlServer cluster is a backup cluster, used to take over the data reading service of the first set of data server SqlServer cluster when the first set of data server SqlServer cluster fails;

[0054] The two sets of data server SqlServer clusters are in master-slave mode, including a master server and multiple slave servers. The master server provides a write data operation for writing the second information to the second server, and the slave server provides a read data operation for reading the second information from the second server.

[0055] Correspondingly, the third server also includes two sets, and the two sets of third servers correspond to the two sets of second servers respectively. Specifically, for example, when a merchant publishes article data, the second information of the article data is written to the main server; when an Internet user reads the second information through an application interface program, the data is read from the slave server. In specific implementation, the number of slave servers can be set according to the size of the business volume, and the request to read data is sent to a certain slave server through a set allocation method.

[0056] Through step S240, according to the set time threshold, the consistency comparison of the data stored in the first server and the second server, and the consistency comparison of the data stored in the second server and the third server is completed respectively.

[0057] Specifically, the purpose of consistency comparison is to ensure that the first information stored in the first server is synchronized with the second information stored in the second server, and the second information stored in the second server is synchronized with the third information stored in the third server, so as to achieve synchronization of the article information stored in the first server, the second server, and the third server. In particular, consistency comparison does not need to be performed in real time, because if real-time comparison is implemented, the computer content overhead will be very large, affecting data processing efficiency. Moreover, the need for consistency comparison is not necessary. For example, even if consistency comparison is not performed, the article data stored in the first server, the second server, and the third server are out of sync, which may cause Internet users to be unable to obtain part of the article data content in a certain server, and will not affect the article data itself or the server itself. However, through consistency comparison, Internet users can obtain a better service experience and can obtain all article data.

[0058] Specifically, in one embodiment of the present application, the step of comparing the consistency of data stored in the first server and the second server includes:

[0059] updating the first information stored in the first server and the second information stored in the second server;

[0060] respectively obtaining first information from the first server and obtaining second information from the second server;

[0061] Determining whether the second information corresponds to the first information;

[0062] If not, the first information that does not correspond to the second information is written or deleted in the first server.

[0063] Specifically, in one embodiment of the present application, the step of comparing the consistency of data stored in the second server and the third server includes:

[0064] updating the second information stored in the second server and the third information stored in the third server;

[0065] A timestamp is set to obtain a set number of second information in the second server and the third information in the third server within the timestamp range; specifically, the purpose of setting the timestamp is to improve the performance of the server storing data. If the timestamp is not set, the second information stored in the second server and the third information stored in the third server are constantly updated, and since the third information stored in the third server is after the second information is stored in the second server, the third information is generated after the second information is added with the style serialization data. In this way, there is a problem that the third information stored in the third server does not correspond to the second information stored in the second server. After setting the timestamp, by comparing the correspondence between a certain number of second information and third information between the timestamps, the above problem can be overcome well, and the illusion of data inconsistency caused by the merchant publishing new article data during the execution of the comparison task can be avoided.

[0066] Determining whether the second information has corresponding third information;

[0067] If not, third information that does not correspond to the second information is written or deleted in the third server.

[0068] Specifically, through consistency comparison, the first information in the first server, the third information in the third server and the second information in the second server are made consistent, so that it can be ensured that after the Internet user retrieves the first information, the third information can be obtained.

[0069] Through step S250, first information is obtained from the first server according to different application programming interfaces, or second information is obtained from the second server, or third information is obtained from the third server.

[0070] Specifically, different application programming interfaces read article data from different servers. For example, if the application programming interface is an article list, then the first information, that is, the summary of the article data, can be obtained from the first server. If it is an application programming interface for the full text of the article, then the third information, that is, the full text information of the article data, can be obtained from the third server.

[0071] According to the data storage and reading scheme of the present invention, the first information and the second information of the article data are obtained by classifying the article data; the first information is added to the Kafka message queue, and after the Kafka message queue consumes the first information, the first information is stored in the first server; the second information is respectively stored in different data tables of the second server, and after adding style serialization data to the second information in the different data tables of the second server, the third information is generated and stored in the third server; according to the set time threshold, the consistency comparison of the data stored in the first server and the second server, and the second server and the third server is completed respectively; according to different application program interfaces, the first information is obtained from the first server, or the second information is obtained from the second server, or the third information is obtained from the third server. This application can realize the safe storage of article data and provide efficient article data query. It divides article data into summary information and full-text information, and stores them in different servers respectively, making full use of the advantages of different servers to achieve the purpose of safe storage and quick access. Through consistency comparison between servers, it can ensure the synchronization of data in each server. The configuration of the data server SqlServer cluster of the active-active cluster can ensure that when a data server SqlServer cluster fails, another data server SqlServer cluster can take over the work, ensuring the security of data storage.

[0072] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0073] In one embodiment, Figure 3 As shown, a data storage and reading device 300 is provided, and the device 300 realizes stable storage and fast reading of article data by storing summary information and full-text information of article data in different servers respectively. The device 300 includes: an article data processing module, an information storage module, an information comparison module, and an information reading module.

[0074] An article data processing module, used to classify the article data, and obtain first information and second information of the article data, wherein the first information is summary information of the article data, and the second information is full text information of the article data;

[0075] An information storage module is used to add the first information to a Kafka message queue according to the first information, and store the first information in a first server after the Kafka message queue consumes the first information, wherein the first server is a search server ElasticSearch; according to different uses of the second information, the second information is respectively stored in different data tables of a second server, and after adding style serialization data to the second information in different data tables of the second server, third information is generated and stored in a third server, wherein the second server is a data server SqlServer cluster of a dual-active cluster, and the third server is a data server HBase;

[0076] An information comparison module, used to complete the consistency comparison of the data stored in the first server and the second server, and the second server and the third server respectively according to a set time threshold;

[0077] The information reading module is used to obtain the first information from the first server, or obtain the second information from the second server, or obtain the third information from the third server according to different application program interfaces.

[0078] Specifically, in another embodiment of the present application, the article data processing module is used to obtain article data published by a merchant, the article data including article title, summary text information, content text information, picture information, article serial number, merchant serial number, and article data validity time information; based on the article data published by the merchant, obtain first information and second information in the article data, the first information is summary information of the article data, the summary information includes: article serial number, summary text information, the first picture in the picture information, merchant serial number, and article data validity time information, the second information is full-text information of the article data, the full-text information includes: article title, content text information, picture information, article serial number, merchant serial number, and article data validity time information.

[0079] Specifically, in another embodiment of the present application, the information storage module is used to add the first information to a Kafka message queue, and obtain a first information process list of the Kafka message queue; based on the first information process list of the Kafka message queue, obtain monitoring information consumed by the first information process list of the Kafka message queue; based on the monitoring information consumed by the first information process list of the Kafka message queue, obtain the data format of the consumed first information and transform it into the data storage format of the first server, and store the consumed first information in the first server, the first server being the search server ElasticSearch.

[0080] Specifically, in another embodiment of the present application, the information storage module is used to add one or more first information stored in the first server to a Kafka message queue, and obtain a first information process list of the Kafka message queue; based on the first information process list of the Kafka message queue, obtain monitoring information consumed by the first information process list of the Kafka message queue; based on the monitoring information consumed by the first information process list of the Kafka message queue, update the consumed first information in the first server.

[0081] Specifically, in another embodiment of the present application, the information storage module is used to obtain all application interfaces connected to the second information and the data table of each application interface in the second server; based on all application interfaces connected to the second information and the data table of each application interface in the second server, the second information is respectively stored in the data table of the second server corresponding to the application interface; based on the second information stored in the data table of each application interface in the second server, the second information is added with style serialization data to obtain third information, and the third information is compatible with the data format of the corresponding application interface; the third information is stored in the third server corresponding to the application interface.

[0082] Specifically, in another embodiment of the present application, the second server of the information storage module is a data server SqlServer cluster of an active-active cluster, and the data server SqlServer cluster of the active-active cluster is two sets of data server SqlServer clusters deployed in two regions respectively, and the two sets of data server SqlServer clusters store the second information simultaneously. Among the two sets of data server SqlServer clusters, the first set of data server SqlServer cluster is a production cluster, used to provide data reading and writing services, and the second set of data server SqlServer cluster is a backup cluster, used to take over the data reading service of the first set of data server SqlServer cluster when the first set of data server SqlServer cluster fails; the two sets of data server SqlServer clusters are in a master-slave mode, including a master server and multiple slave servers, the master server provides a write data operation, used to write the second information to the second server, and the slave server provides a read data operation, used to read the second information from the second server.

[0083] Specifically, in another embodiment of the present application, the information comparison module is used to update the first information stored in the first server and the second information stored in the second server; obtain the first information from the first server and the second information from the second server respectively; determine whether the second information has corresponding first information; if not, write or delete the first information that does not correspond to the second information in the first server.

[0084] Specifically, in another embodiment of the present application, the information comparison module is used to update the second information stored in the second server and the third information stored in the third server; set a timestamp, obtain the second information in a set number of second servers within a timestamp range, and the third information in the third server; determine whether there is corresponding third information for the second information; if not, write or delete the third information that does not correspond to the second information in the third server.

[0085] According to the data storage and reading scheme of the present invention, the article data is classified through the article data processing module to obtain the first information and the second information of the article data; the first information is added to the Kafka message queue according to the first information through the information storage module, and the first information is stored in the first server after the Kafka message queue consumes the first information; according to different uses of the second information, the second information is respectively stored in different data tables of the second server, and after adding style serialization data to the second information in the different data tables of the second server, the third information is generated and stored in the third server; the consistency comparison of the data stored in the first server and the second server, and the second server and the third server is respectively completed according to the set time threshold through the information comparison module; the first information is obtained from the first server according to different application program interfaces, or the second information is obtained from the second server, or the third information is obtained from the third server. This application can realize the safe storage of article data and provide efficient article data query. It divides article data into summary information and full-text information, and stores them in different servers respectively, making full use of the advantages of different servers to achieve the purpose of safe storage and quick access. Through consistency comparison between servers, it can ensure the synchronization of data in each server. The configuration of the data server SqlServer cluster of the active-active cluster can ensure that when a data server SqlServer cluster fails, another data server SqlServer cluster can take over the work, ensuring the security of data storage.

[0086] A8. A method as described in A1, wherein the step of consistency comparison of data stored in the second server and the third server includes: updating the second information stored in the second server and the third information stored in the third server; setting a timestamp, obtaining a set number of second information in the second servers within a timestamp range, and the third information in the third server; determining whether there is corresponding third information for the second information; if not, writing or deleting third information that does not correspond to the second information in the third server.

[0087] It should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: that the claimed invention requires more features than those explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0088] Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in the devices described in the embodiment, or alternatively may be located in one or more devices different from the devices in the examples. The modules in the foregoing examples may be combined into one module or may be divided into multiple submodules.

[0089] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0090] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0091] In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the functions. Therefore, a processor with necessary instructions for implementing the method or method elements forms a device for implementing the method or method elements. In addition, the elements described herein of the device embodiments are examples of devices for implementing the functions performed by the elements for the purpose of implementing the invention.

[0092] As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are involved, and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

[0093] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.

Claims

1. A method for storing and reading data, wherein the method stores summary information and full text information of article data in different servers respectively, thereby achieving stable storage and fast reading of article data, and the method comprises the steps of: Classify the article data to obtain first information and second information of the article data, wherein the first information is summary information of the article data and the second information is full text information of the article data; According to the first information, the first information is added to a Kafka message queue, and after the Kafka message queue consumes the first information, the first information is stored in a first server, where the first server is a search server ElasticSearch; According to different uses of the second information, the second information is stored in different data tables of the second server respectively, and after adding style serialization data to the second information in different data tables of the second server, the third information is generated and stored in the third server, the second server is a data server SqlServer cluster of the active-active cluster, and the third server is a data server HBase, and the style serialization data is used to adapt the data format of the third information to the data format of the corresponding application program interface, so that the terminal device connected to the application interface program can directly read the corresponding third information; According to the set time threshold, respectively complete the consistency comparison of the data stored in the first server and the second server, and the second server and the third server; Obtaining first information from the first server, or obtaining second information from the second server, or obtaining third information from the third server according to different application program interfaces, wherein: Different application programming interfaces read information from different servers. With respect to obtaining the second information from the second server, data tables having the same number as the application programming interface styles are set according to the style of the application programming interface connected to the second server. Each data table corresponds to an application programming interface of one style, so that the application programming interface can obtain the second information in a data format that is suitable for it.

2. The method of claim 1, wherein: The step of classifying the article data and obtaining the first information and the second information of the article data comprises: Obtain article data published by a merchant, the article data including article title, abstract text information, content text information, picture information, article serial number, merchant serial number, and article data validity period information; Based on the article data published by the merchant, first information and second information in the article data are obtained, wherein the first information is summary information of the article data, and the summary information includes: article serial number, summary text information, the first picture in the picture information, merchant serial number, and article data validity time information; the second information is full-text information of the article data, and the full-text information includes: article title, content text information, picture information, article serial number, merchant serial number, and article data validity time information.

3. The method of claim 1, wherein: The step of adding the first information to a Kafka message queue according to the first information, and storing the first information in a first server after the Kafka message queue consumes the first information includes: Add the first information to the Kafka message queue, and obtain a first information process list of the Kafka message queue; According to the first information process list of the Kafka message queue, obtaining the monitoring information consumed by the first information process list of the Kafka message queue; According to the first information process list of the Kafka message queue, the data format of the first information consumed is obtained and converted into the data storage format of the first server, and the first information consumed is stored in the first server, which is the search server ElasticSearch.

4. The method of claim 3, wherein: The step of adding the first information to a Kafka message queue according to the first information, and storing the first information in a first server after the Kafka message queue consumes the first information, further includes: Add one or more first information stored in the first server to a Kafka message queue, and obtain a first information process list of the Kafka message queue; According to the first information process list of the Kafka message queue, obtaining the monitoring information consumed by the first information process list of the Kafka message queue; According to the monitoring information consumed by the first information process list of the Kafka message queue, the first server updates the consumed first information.

5. The method of claim 1, wherein: The step of storing the second information in different data tables of the second server according to different uses of the second information, and generating third data and storing it in the third server after adding style serialization data to the second information in the different data tables of the second server includes: Acquire all application program interfaces connected to the second information and a data table of each application program interface on the second server; According to all the application program interfaces connected to the second information and the data table of each application program interface on the second server, the second information is stored in the data table of the second server corresponding to the application program interface; According to the second information stored in the data table of the second server for each application program interface, the second information is added with the style serialization data to obtain the third information, wherein the third information is adapted to the data format of the corresponding application program interface; The third information is stored in a third server corresponding to the application program interface.

6. The method of claim 1, wherein: The second server is a data server SqlServer cluster of an active-active cluster, and the data server SqlServer cluster of the active-active cluster is two sets of data server SqlServer clusters deployed in two regions respectively, and the two sets of data server SqlServer clusters store the second information simultaneously. Among the two sets of data server SqlServer clusters, the first set of data server SqlServer cluster is a production cluster, which is used to provide data reading and writing services, and the second set of data server SqlServer cluster is a backup cluster, which is used to take over the data reading service of the first set of data server SqlServer cluster when the first set of data server SqlServer cluster fails; The two sets of data server SqlServer clusters are in master-slave mode, including a master server and multiple slave servers. The master server provides a write data operation for writing the second information to the second server, and the slave server provides a read data operation for reading the second information from the second server.

7. The method of claim 1, wherein: The step of comparing the consistency of the data stored in the first server and the second server comprises: updating the first information stored in the first server and the second information stored in the second server; respectively obtaining first information from the first server and obtaining second information from the second server; Determining whether the second information corresponds to the first information; If not, the first information that does not correspond to the second information is written or deleted in the first server.

8. The method of claim 1, wherein: The steps of consistency comparison of data stored in the second server and the third server include: updating the second information stored in the second server and the third information stored in the third server; setting a timestamp, obtaining a set number of second information in the second servers within a timestamp range, and the third information in the third server; determining whether there is corresponding third information for the second information; if not, writing or deleting third information that does not correspond to the second information in the third server.

9. A device for storing and reading data, wherein the device stores summary information and full text information of article data in different servers respectively, thereby realizing stable storage and fast reading of article data, the device comprising: An article data processing module, used to classify the article data, and obtain first information and second information of the article data, wherein the first information is summary information of the article data, and the second information is full text information of the article data; An information storage module is used to add the first information to a Kafka message queue based on the first information, and store the first information in a first server after the Kafka message queue consumes the first information, wherein the first server is a search server ElasticSearch; according to different uses of the second information, the second information is respectively stored in different data tables of a second server, and after adding style serialization data to the second information in different data tables of the second server, third information is generated and stored in a third server, wherein the second server is a data server SqlServer cluster of a dual-active cluster, and the third server is a data server HBase, wherein the style serialization data is used to adapt the data format of the third information to the data format of the corresponding application program interface, so that a terminal device connected to the application interface program can directly read the corresponding third information; An information comparison module, used to complete the consistency comparison of the data stored in the first server and the second server, and the second server and the third server respectively according to a set time threshold; An information reading module is used to obtain the first information from the first server, or to obtain the second information from the second server, or to obtain the third information from the third server according to different application interfaces, wherein different application interfaces read information from different servers, and for obtaining the second information from the second server, according to the style of the application interface connected to the second server, set data tables with the same number as the application interface style, and each data table corresponds to an application interface of one style, so that the application interface can obtain the second information in a data format suitable for it.

10. An electronic device comprising: one or more processors; and Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for executing any of the methods according to claims 1-8.

11. A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when a computing device executes the instructions, the computing device performs any one of the methods according to claims 1-8.

Images