Data storage method, device and equipment and computer readable storage medium

Abstract

The invention provides a data storage method, device and equipment and a computer readable storage medium. The method comprises the steps that multiple pieces of to-be-stored data are obtained, and each piece of to-be-stored data comprises answer content corresponding to multiple questions; according to the question type of the question contained in each piece of to-be-stored data, performing encoding processing on each answer content to obtain each piece of encoded data; compressing the coded data of the plurality of answer contents corresponding to the same question to obtain compressed data; and storing the compressed data. Through the application, the data storage cost can be reduced, and the data compression efficiency can be improved.

Description

Technical Field

[0001] The present application relates to information processing technology, and more particularly to a data storage method, device, equipment and computer-readable storage medium. Background Art

[0002] In real life, questionnaire survey is a method of collecting data and information. Questionnaire survey has the advantages of collecting opinions and feedback, quantifying data, saving costs, being efficient and fast, monitoring and evaluating, achieving important data collection and analysis purposes, and providing a scientific basis for research, decision-making and management.

[0003] Before analyzing the data, the first thing to face is how to store the collected data. In the related technologies, there are storage solutions represented by either relational databases such as the MySQL version of the structured query language (MySQL) or open source full-text search engines (such as elastic search). However, no matter which solution in the related technologies is adopted, the storage cost is high due to the huge amount of stored data. Summary of the invention

[0004] The embodiments of the present application provide a data storage method, apparatus, computer equipment, computer-readable storage medium, and computer program product, which can reduce data storage costs and improve data compression efficiency.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present invention provides a data storage method, the method comprising:

[0007] Acquire multiple pieces of data to be stored, wherein each piece of data to be stored includes answers corresponding to multiple questions;

[0008] According to the question type of each question contained in each piece of data to be stored, each answer content is encoded to obtain each encoded data;

[0009] Compressing the coded data of the multiple answers corresponding to the same question to obtain compressed data;

[0010] The compressed data is stored.

[0011] An embodiment of the present application provides a data storage device, the device comprising:

[0012] A first acquisition module is used to acquire multiple pieces of data to be stored, wherein each piece of data to be stored includes answers corresponding to multiple questions;

[0013] The encoding module is used to encode the content of each answer according to the question type of each question contained in each piece of data to be stored, so as to obtain each encoded data;

[0014] A compression module, used for compressing the coded data of multiple answers corresponding to the same question to obtain compressed data;

[0015] A storage module is used to store the compressed data.

[0016] An embodiment of the present application provides a computer device, including:

[0017] A memory for storing computer executable instructions;

[0018] The processor is used to implement the data storage method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the data storage method provided in the embodiment of the present application when the instructions are executed by a processor.

[0020] An embodiment of the present application provides a computer program product, including a computer program or a computer executable instruction. When the computer program or the computer executable instruction is executed by a processor, the data storage method provided in the embodiment of the present application is implemented.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] In an embodiment of the present application, firstly, a plurality of data to be stored are obtained, each of which includes the answer content corresponding to a plurality of questions; then, according to the question type of the question contained in each piece of data to be stored, each answer content is encoded to obtain each encoded data, so that the answer content can be converted into encoded data that is easy to process and store through encoding; then, the encoded data of the multiple answer contents corresponding to the same question are compressed to obtain compressed data, that is, column compression is achieved according to the question. Since the multiple answer contents corresponding to the same question are similar, the compression efficiency can be improved when the column compression is performed, and efficient compression can be achieved to obtain compressed data with small storage space requirements; finally, the compressed data is stored. In this way, when storing the above compressed data, the demand for storage space can be reduced while ensuring the integrity and accuracy of the data, that is, the data storage cost can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic diagram of a network architecture of a data storage system 100 provided in an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the structure of the server 400 provided in an embodiment of the present application;

[0025] Figure 3 This is a schematic diagram of an implementation flow of the data storage method provided in an embodiment of the present application;

[0026] Figure 4 This is a schematic diagram of an implementation flow of caching multiple pieces of data to be stored provided in an embodiment of the present application;

[0027] Figure 5 This is a schematic diagram of an implementation flow of the encoding method provided in an embodiment of the present application;

[0028] Figure 6 A schematic diagram of an implementation flow of the compression method provided in an embodiment of the present application;

[0029] Figure 7 A schematic diagram of an implementation flow of determining a compression algorithm provided in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of an implementation flow of a data cleaning method provided in an embodiment of the present application;

[0031] Fig. 9 A schematic diagram of another implementation flow of the data storage method provided in an embodiment of the present application;

[0032] Fig.10 A schematic diagram of a data storage framework provided in an embodiment of the present application;

[0033] Fig.11 A schematic diagram of a framework for team processing provided in an embodiment of the present application;

[0034] Fig.12 A schematic diagram of a framework of a mapping encoding method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0036] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0037] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0038] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0039] The embodiments of the present application provide a data storage method, apparatus, computer equipment, computer-readable storage medium, and computer program product, which can solve the problem of high storage cost and reduced data analysis performance. The following describes an exemplary application of the computer equipment provided by the embodiments of the present application. The computer equipment provided by the embodiments of the present application can be implemented as various types of user terminals such as laptops, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals. The following describes an exemplary application when the computer equipment is implemented as a server.

[0040] See also Figure 1 , Figure 1 It is a network architecture diagram of the data storage system 100 provided in an embodiment of the present application. To support a data storage application, the terminal is connected to the server 400 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0041] exist Figure 1 1, 200-2 and 200-3 are shown as examples. In practical applications, the data storage system 100 may include multiple terminals. The terminal 200-1, the terminal 200-2 and the terminal 200-3 collect a piece of data to be stored respectively, wherein each piece of data to be stored includes the answer content corresponding to multiple questions; when the terminal 200-1, the terminal 200-2 and the terminal 200-3 receive the collection end instruction, the data collection is completed, and the terminal 200-1, the terminal 200-2 and the terminal 200-3 send the multiple pieces of data to be stored to the server 400. The server 400 is used to encode each answer content according to the question type of the question contained in each piece of data to be stored to obtain each encoded data; the server 400 is used to compress the encoded data of multiple answer contents corresponding to the same question to obtain compressed data; the server 400 is used to store the compressed data.

[0042] In an embodiment of the present application, the server 400 can convert the answer content into coded data that is easy to process and store through encoding processing. And it is compressed by column according to the question. Since the contents of multiple answers corresponding to the same question are similar, the compression efficiency can be improved when it is compressed by column, and efficient compression can be achieved to obtain compressed data with small storage space requirements. In this way, when storing the above compressed data, under the premise of ensuring data integrity and accuracy, the demand for storage space can be reduced, that is, the data storage cost is reduced. In addition, since the compression is performed by column according to the same question, when performing data analysis, the compressed data corresponding to a certain question can be quickly found, so that the compressed data corresponding to the question can be analyzed and processed, that is, the data analysis performance is improved.

[0043] In some embodiments, the server 400 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car-mounted smart terminal, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0044] See also Figure 2 , Figure 2 A schematic diagram of the structure of the server 400 provided in the embodiment of the present application, Figure 2 The server 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the server 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0045] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0046] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0047] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0048] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0049] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0050] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0051] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0052] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0053] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0054] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The data storage device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a first acquisition module 4551, an encoding module 4552, a compression module 4553 and a storage module 4554. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0055] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data storage method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (Application Specific Integ rated Circuit, ASIC), DSP, programmable logic device (Programmable Logic Device, PLD), complex programmable logic device (Complex Programmable Logic Device, CPLD), field programmable gate array (Field-Programmable Gate Array, FPGA) or other electronic components.

[0056] The data storage method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0057] The following describes the data storage method provided by the embodiment of the present application. As mentioned above, the computer device implementing the data storage method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution subject of each step will not be repeatedly described below.

[0058] It should be noted that in the example of the data storage method below, questionnaire data is used as an example. Based on the understanding of the following text, those skilled in the art can apply the data storage method provided in the embodiment of the present application to the storage of other data such as test answers including the answer content corresponding to multiple questions.

[0059] See also Figure 3 , Figure 3 This is a schematic diagram of an implementation flow of a data storage method provided in an embodiment of the present application. The method is applied to a computer device. Figure 3 The steps shown are used to illustrate the data storage method provided in the embodiment of the present application. Figure 3 The execution subject is the server.

[0060] In step S101, a plurality of pieces of data to be stored are obtained.

[0061] In the embodiment of the present application, each piece of data to be stored includes answers corresponding to multiple questions.

[0062] In some embodiments, the data to be stored may be questionnaire data, activity registration data, test evaluation data, etc., wherein the data to be stored may be data obtained through an online system or data obtained through a terminal application (application, APP). Exemplarily, the data to be stored may be satisfaction survey data, holiday gift selection data, sports meeting registration data, grade test data, etc.

[0063] In some embodiments, due to the problem that the generation time of each piece of data to be stored is different, before storing multiple pieces of data to be stored, the obtained data to be stored can be cached in real time. In some embodiments, a queue can be used to cache multiple pieces of data to be stored in real time, see Figure 4 , the multiple pieces of data to be stored can be cached in real time through the following steps S01 to S04.

[0064] In step S01, the generation time of multiple pieces of data to be stored is obtained.

[0065] In some embodiments, for a piece of data to be stored, the generation time of the data to be stored refers to the time when the data to be stored is generated. For example, the generation time of a piece of data to be stored may be 8:00.

[0066] In some embodiments, the time when the submission instruction is received can be determined as the generation time of the corresponding data to be stored. The time when the submission instruction is received refers to the time when the corresponding data to be stored is submitted to the server.

[0067] In step S02, multiple pieces of data to be stored are cached in a queue in the order of generation time from earliest to latest, thereby obtaining a data queue.

[0068] In some embodiments, the data to be stored acquired within the preset period can be cached into a queue according to a preset period. When at least two data to be stored are acquired within the preset period, the at least two data to be stored are sorted in order from early to late according to the generation time of the at least two data to be stored; and the sorted data to be stored are cached into the queue in sequence, thereby obtaining a data queue. Then, the data to be stored in the next period is cached in the same way. In some embodiments, in order to improve the cache efficiency, the acquired data to be stored can also be cached in real time.

[0069] In some embodiments, in the first preset cycle, the data to be stored is cached in the initial queue to obtain a first data queue, and the initial queue is empty. From the second preset cycle onwards, based on the data queue obtained in the previous cycle, the data to be stored in the current cycle continues to be cached until the last preset cycle to obtain a data queue. The preset cycle can be a value set in advance. Exemplarily, the preset cycle can be 10 minutes, 20 minutes, 1 hour, etc. Among them, the number of preset cycles is related to the preset cycle and the total duration of data collection. Exemplarily, assuming that the preset cycle is 1 hour and the total duration of data collection is 1 week, the number of preset cycles is 168, that is, there are 168 preset cycles.

[0070] In step S03, new data to be stored is obtained.

[0071] In some embodiments, after obtaining a plurality of pieces of data to be stored, new data to be stored may continue to be obtained, and the method for obtaining the new data to be stored may be the same as the method for obtaining the data to be stored described above.

[0072] In step S04, the new data to be stored is cached at the end of the data queue to obtain an updated data queue.

[0073] In some embodiments, similar to the implementation process of step S02 above, new data to be stored is cached at the end of the data queue, thereby obtaining an updated data queue.

[0074] In some embodiments, when there are multiple new data to be stored, the generation time of each new data to be stored is obtained, and each new data to be stored is sorted in order from early to late according to the generation time of each new data to be stored; then, the first new data to be stored is cached to the end of the data queue to obtain the first new data queue; then, the second new data to be stored is cached at the end of the new data queue to obtain the second new data queue, until the caching of all new data to be stored is completed to obtain an updated data queue.

[0075] Through the above steps S01 to S04, after obtaining multiple data to be stored and before storing the multiple data to be stored, the multiple data to be stored will be cached in the form of a queue. Among them, a queue is a linear data structure that can be used to store and operate elements. The queue has the advantages of first-in first-out (FIFO), low-complexity insertion and deletion operations, control of concurrent access, buffering and breadth-first search. Therefore, when caching the data to be stored in a queue, it can not only ensure the orderliness of the data to be stored, but also flexibly and dynamically adjust the queue length, which is convenient for caching new data to be stored and can also prevent the accumulation of data to be stored.

[0076] Continue to see Figure 3 , and the description continues with step S101 above.

[0077] In step S102, according to the question type of the question contained in each piece of data to be stored, each answer content is encoded to obtain each encoded data.

[0078] In some embodiments, the question type of each question may be obtained first; and then, according to the encoding method corresponding to each question type, the answer content corresponding to each question type may be encoded to obtain various encoded data.

[0079] In some embodiments, since question types may include single-choice type, multiple-choice type, scoring type, ranking type, text type, etc., and different question types correspond to different forms of answer content, based on this, in order to convert the answer content into a format that is easy to process, it is necessary to use different encoding methods for different forms of answer content, that is, encode the answer content corresponding to each question type according to the encoding method corresponding to each question type, so as to obtain encoded data in a format that is easy to process.

[0080] In some embodiments, see Figure 5 The above-mentioned "encoding the answer content corresponding to each question type according to the encoding method corresponding to each question type to obtain each encoded data" can be achieved through the following steps S1021 to S10211, which are explained in detail below.

[0081] In step S1021, it is determined whether the question type is a single-choice type.

[0082] In some embodiments, the single-choice type means that the answer content corresponding to the question is to select one option from multiple options. Different question types may correspond to different reference type codes, and the question type and the reference type code are one-to-one corresponding. For example, the reference type code corresponding to the single-choice type may be 001, the reference type code of the multiple-choice type may be 002, the reference type code of the scoring type may be 003, the reference type code of the sorting type may be 004, and the reference type code of the text type may be 005. Based on this, the type code of the question type may be obtained first, and then the target reference type with the same type code may be determined from each reference type code, and finally the target question type corresponding to the target reference type may be determined as the question type.

[0083] In the embodiment of the present application, following the above example, if the target reference type is 001, it indicates that the question type is a single-choice type, and the process proceeds to step S1022; if the target reference type is one of 002, 003, 004, and 005, it indicates that the question type is not a single-choice type, and the process proceeds to step S1023.

[0084] In step S1022, a reference mapping value of the answer content corresponding to the question type is obtained, and the reference mapping value is determined as the coded data of the answer content, and the process ends.

[0085] In some embodiments, the answer content corresponding to the single-choice type is a character, and each character corresponds to a reference mapping value. Exemplarily, assuming that the answer content is one of A, B, C, and D, the reference mapping value corresponding to A can be 1, the reference mapping value corresponding to B can be 2, the reference mapping value corresponding to C can be 3, and the reference mapping value corresponding to D can be 4. Based on this, the reference mapping value can be determined as the encoding data of the answer content. Continuing with the above example, 1 can be determined as the encoding data of A, 2 can be determined as the encoding data of B, 3 can be determined as the encoding data of C, and 4 can be determined as the encoding data of D.

[0086] In step S1023, it is determined whether the question type is a multiple choice type.

[0087] In the embodiment of the present application, if the question type is not a single-choice type, it is determined whether the question type is a multiple-choice type, wherein the multiple-choice type means that the answer content corresponding to the question is to select at least two options from multiple options. Continuing with the above example, if the target reference type is 002, it indicates that the question type is a multiple-choice type, and then proceeds to step S1024; if the target reference type is one of 003, 004, and 005, it indicates that the question type is neither a single-choice type nor a multiple-choice type, then it is continued to determine whether the question type is other types except the above two types, and then proceeds to step S1025.

[0088] In step S1024, the reference binary codes of the options in the answer content corresponding to the question type are obtained, and the reference binary codes are ORed to obtain the coded data of the answer content. The process ends.

[0089] In some embodiments, the answer content corresponding to the multiple choice type is at least two characters, and each character corresponds to a reference binary code. Exemplarily, the reference binary code corresponding to A can be 0001, the reference binary code corresponding to B can be 0010, the reference binary code corresponding to C can be 0100, and the reference binary code corresponding to D can be 1000. Based on this, if the answer content is ACD, 0001, 0100 and 1000 are ORed to obtain 1101, and 1101 is determined as the coded data of the answer content.

[0090] In step S1025, it is determined whether the question type is a scoring type.

[0091] In the embodiment of the present application, if the question type is neither a single-choice type nor a multiple-choice type, it is determined whether the question type is a scoring type, wherein the scoring type means that the answer content corresponding to the question is a scoring result. Continuing with the above example, if the target reference type is 003, it indicates that the question type is a scoring type, and then proceeds to step S1026; if the target reference type is 004 or 005, it indicates that the question type is neither a single-choice type, nor a multiple-choice type, nor a scoring type, then it is determined whether the question type is other than the above three types, and then proceeds to step S1027.

[0092] In step S1026, the answer content corresponding to the question type is determined as the coded data of the answer content, and the process ends.

[0093] In some embodiments, the answer content corresponding to the scoring type question is a number, for example, the answer content expressed in a sentence is 5. Based on this, the answer content can be directly determined as the encoding data of the answer content. Exemplarily, if the scoring result (i.e., the answer content) is 5, 5 is determined as the encoding result of the answer content.

[0094] In step S1027, it is determined whether the question type is a ranking type.

[0095] In the embodiment of the present application, if the question type is neither a single-choice type, nor a multiple-choice type, nor a scoring type, then it is determined whether the question type is a sorting type, wherein the sorting type refers to the answer content corresponding to the question being the sorting result of the options. Continuing with the above example, if the target reference type is 004, then the characterization question type is a sorting type, then the process proceeds to step S1028; if the target reference type is 005 or other codes not listed above, then the characterization question type is neither a single-choice type, nor a multiple-choice type, a scoring type, nor a sorting type, then it is continued to determine whether the question type is other types other than the above four types, then the process proceeds to step S1029.

[0096] In step S1028, the reference mapping values ​​of the sorting items in the answer content corresponding to the question type are obtained, and the reference mapping values ​​of the sorting items are sorted according to the arrangement order of the sorting items to obtain the encoded data of the answer content. The process ends.

[0097] In some embodiments, the answer content corresponding to the sorting type includes multiple characters, and each character corresponds to a reference mapping value. Exemplarily, the reference mapping value corresponding to A can be 1, the reference mapping value corresponding to B can be 2, the reference mapping value corresponding to C can be 3, and the reference mapping value corresponding to D can be 4. Based on this, if the answer content is BADC, the above reference mapping values ​​are sorted according to the order of the answer content, and 2143 is obtained after sorting, then 2143 is determined as the encoding data of the answer content (BADC).

[0098] In step S1029, it is determined whether the title type is a text type.

[0099] In the embodiment of the present application, if the question type is neither a single-choice type, nor a multiple-choice type, a scoring type, nor a sorting type, then it is determined whether the question type is a text type, wherein the text type means that the answer content corresponding to the question is a string. Continuing with the above example, if the target reference type is 005, then the characterization question type is a text type, then the process proceeds to step S10210; if the target reference type is another code not listed above, then the characterization question type is neither a single-choice type, nor a multiple-choice type, a scoring type, nor a sorting type, then the process ends, that is, proceeds to step S10211.

[0100] In step S10210, the answer content corresponding to the question type is segmented to obtain the segmentation result of the reply content, and the segmentation result is determined as the coded data of the reply content. The process ends.

[0101] In some embodiments, the answer content corresponding to the text type is a character string. To facilitate subsequent matching and processing of the answer content, the answer content is segmented to obtain a segmentation result, and the segmentation result is determined as the encoded data of the reply content.

[0102] Step S10211, end the process.

[0103] Through the above step S102, the answers corresponding to each question type are encoded according to the encoding method corresponding to the question type, so as to improve the encoding flexibility, that is, the encoding efficiency, and obtain each encoded data through encoding, that is, convert the answer content into encoded data, wherein the format of the encoded data is a format that is convenient for processing or storage. This provides a data basis for subsequent efficient compression and storage.

[0104] Continue to see Figure 3 , and the description continues with step S102 above.

[0105] In step S103, the encoded data of the multiple answers corresponding to the same question are compressed to obtain compressed data.

[0106] In some embodiments, see Figure 6 Step S103 can be implemented by following the steps S1031 to S1033, which are described in detail below.

[0107] In step S1031, the coded data of each answer content corresponding to the i-th question is obtained.

[0108] In the embodiment of the present application, i is a positive integer less than or equal to N, and N is the total number of questions. For example, N can be 100, 500, 1000, 10000, etc.

[0109] In some embodiments, the coded data of each answer content corresponding to each question is obtained according to the question. Taking the first question as an example, the coded content of each answer content corresponding to the first question is obtained. If there are 100 pieces of data to be stored, the coded content of the 100 answers content corresponding to the first question is obtained.

[0110] In step S1032, the i-th compression algorithm for the encoded data of each answer content corresponding to the i-th question is determined.

[0111] In some embodiments, see Figure 7 Step S1032 can be implemented by following the steps S321 to S324, which are described in detail below.

[0112] In step S321, the i-th data type of the encoded data corresponding to the i-th question is obtained.

[0113] In the embodiment of the present application, the i-th data type of the coded data corresponding to the i-th question is continuously obtained. Continuing with the above example, if the question type of the i-th question is a single-choice type, then the coded data corresponding to the i-th question can be 1, 2, 3 or 4, that is, the i-th data type is a numeric type. If the question type of the i-th question is a text type, then the coded data corresponding to the i-th question can be a word segmentation result, that is, the i-th data type is a string type.

[0114] In step S322, it is determined whether the i-th data type is a numeric type.

[0115] In the embodiment of the present application, if the i-th data type is a numeric type, then the process proceeds to step S323; if the i-th data type is not a numeric type, then it indicates that the i-th data type is a string type, then the process proceeds to step S324.

[0116] In step S323, the integer compression algorithm is determined as the i-th compression algorithm.

[0117] In some embodiments, the integer compression algorithm compresses a set of integer sequences into a shorter sequence to more efficiently store and transmit data, wherein the integer compression algorithm may include a Huffman coding algorithm, a geometric sequence compression algorithm, a Delta coding algorithm, an integer interval coding algorithm, and the like.

[0118] In step S324, the character string compression algorithm is determined as the i-th compression algorithm.

[0119] In some embodiments, the string compression algorithm includes Huffman coding, LZ77 algorithm, LZ78 algorithm, Lipman-Ziv-Welch (LZW) algorithm, arithmetic coding, etc.

[0120] Through the above steps S321 to S324, a coding algorithm matching the data type of the coded data of different data types can be determined.

[0121] Continue to see Figure 6 , and the description continues with step S1032 above.

[0122] In step S1033, the encoded data of each answer content corresponding to the i-th question is compressed based on the i-th compression algorithm to obtain the i-th compressed data corresponding to the i-th question.

[0123] In some embodiments, the encoded data of each answer content corresponding to the first question can be compressed based on a first compression algorithm to obtain first compressed data; the encoded data of each answer content corresponding to the second question can be compressed based on a second compression algorithm to obtain second compressed data; and so on, until the encoded data of each answer content corresponding to the Nth question is compressed based on an Nth compression algorithm to obtain Nth compressed data.

[0124] In some embodiments, in order to improve compression efficiency, the encoded data of the answers corresponding to multiple questions can be compressed simultaneously in a parallel manner, so that the compressed data corresponding to multiple questions can be obtained at one time, thereby improving the compression speed.

[0125] Through the above step S103, on the one hand, the coded data of each answer content corresponding to the same question is compressed, and since the coded data of each answer content corresponding to the same question has the same data type, the convenience of compression can be improved. On the other hand, when compressing, different coded data use different compression algorithms, so that the flexibility of compression can be improved, the compression time can be shortened, and the compression efficiency can be improved.

[0126] Continue to see Figure 3 , and the description continues with step S103 above.

[0127] In step S104, the compressed data is stored.

[0128] In some embodiments, in response to a received data storage instruction, a storage path corresponding to the compressed data may be determined; and then the compressed data may be stored in a storage space corresponding to the storage path according to a preset file format.

[0129] In some embodiments, the data storage instruction may be a click instruction for a control, or an instruction in the form of voice; after receiving the data storage instruction, the data size of the compressed data may be obtained, and a storage path where the available remaining storage space is greater than the data size may be determined; finally, the compressed data may be stored in the storage space corresponding to the storage path according to a preset file format. The preset file format refers to a file format with high compression and fast query characteristics. For example, the preset file format may be a columnar storage format (Parquet) or an optimized row columnar format (Optimized Row Columnar, ORC).

[0130] The columnar storage format stores data in columns, which means that data of the same type is stored together, which can reduce I / O operations and improve the efficiency of subsequent queries. The columnar storage format also has advantages such as good compression performance, strong scalability, and rich data types. The optimized row-column format is an efficient columnar storage format, which is usually used in data warehouses and data analysis applications in big data environments. The optimized row-column format has better compression rate, query performance, and reading speed.

[0131] In some embodiments, when storing compressed data, a data structure may be determined according to the data type of the compressed data corresponding to different questions, and the determined data structure may be used to store the compressed data corresponding to the question. Exemplarily, when the data type of the compressed data corresponding to a certain question is a numeric type, a list is used to store the compressed data corresponding to the question; when the data type of the compressed data corresponding to a certain question is a string type, a linked list is used to store the compressed data corresponding to the question.

[0132] Through the above step S104, since the compressed data itself is compressed according to the title, that is, compressed by column, the storage space required for the compressed data is reduced, and the purpose of saving storage cost is achieved; the compressed data is also stored in a preset file format, which further improves the compression rate and achieves efficient compression. At the same time, it can also provide a data basis for subsequent efficient query and fast reading.

[0133] Through the above steps S101 to S104, firstly, a plurality of data to be stored are obtained, each of which includes the answer content corresponding to a plurality of questions; then, according to the question type of the question contained in each piece of data to be stored, each answer content is encoded to obtain each encoded data, so that the answer content can be converted into encoded data that is easy to process and store through encoding; then, the encoded data of the multiple answer contents corresponding to the same question are compressed to obtain compressed data, that is, column compression according to the question is realized. Since the multiple answer contents corresponding to the same question have similarities, the compression efficiency can be improved when the column compression is performed, and efficient compression is achieved, and compressed data with small storage space requirements is obtained; finally, the compressed data obtained is stored. In this way, when storing the above compressed data, the demand for storage space can be reduced while ensuring the integrity and accuracy of the data, that is, the data storage cost can be reduced. In addition, since the column compression is performed according to the same question, the compressed data corresponding to a certain question can be quickly found when performing data analysis, so that the compressed data corresponding to the question can be analyzed and processed, that is, the data analysis performance is improved.

[0134] In some embodiments, in order to improve the data quality of the data to be stored, before the above step S102 "encoding the content of each answer according to the question type of each question contained in each piece of data to be stored to obtain each encoded data", see Figure 8 , the following steps S001 to S003 can also be performed to clean the data to be stored so as to improve the data quality, which is described in detail below.

[0135] In step S001, abnormal data is determined from a plurality of pieces of data to be stored.

[0136] In an embodiment of the present application, the abnormal data includes at least one of duplicate data, data of a different format from a reference, data of a different type from a reference, and null value data.

[0137] In step S002, data cleaning is performed on the abnormal data to obtain cleaned data.

[0138] In some embodiments, during the collection and processing of the data to be stored, duplicate data to be stored may appear, that is, the abnormal data is duplicate data, and the duplicate data can be removed (that is, deleted) to achieve data cleaning.

[0139] In some embodiments, if there are data with a different format from the reference or abnormal data with a different type from the reference in the plurality of data to be stored, the abnormal data is standardized to obtain cleaned data so that the cleaned data has the same format and type as the reference.

[0140] In some embodiments, when the abnormal data is null value data, the abnormal data is deleted to achieve data cleaning.

[0141] In step S003, the abnormal data in the plurality of data to be stored are replaced with the corresponding cleaned data to obtain the updated plurality of data to be stored.

[0142] In some embodiments, the cleaned data is used to replace abnormal data in the multiple pieces of data to be stored to obtain updated multiple pieces of data to be stored, so that there is no abnormal data in the updated multiple pieces of data to be stored to avoid affecting the encoding and compression effects.

[0143] In some embodiments, the above steps S001 to S003 may be executed after the above step S101, that is, the execution timing of steps S001 to S003 is after executing step S101 and before executing step S102.

[0144] Through the above steps S001 to S003, through data cleaning processing, it is possible to ensure that the updated multiple data to be stored are accurate, consistent, complete and reliable, thereby achieving the purpose of reducing errors and deviations and improving the quality of the updated multiple data to be stored.

[0145] In some embodiments, see Fig. 9 After the above step S104 "storing compressed data", the following steps S105 to S113 may be performed to achieve a quick query of the query question and to achieve the purpose of efficiently analyzing the target answer content of the query question, as described in detail below.

[0146] In step S105, in response to the received query instruction, the topic identifier of the topic to be queried is obtained.

[0147] In some embodiments, the query instruction may be a click instruction for a control, or may be an instruction in the form of sound, and the query instruction carries the title identifier of the title to be queried. Based on this, the title identifier of the title to be queried may be obtained by parsing the query instruction.

[0148] In step S106, the target compressed data corresponding to the topic identifier is determined from the storage space.

[0149] In some embodiments, since the compressed data includes compressed data corresponding to a question, the target compressed data corresponding to the question identifier can be determined from the storage space based on the question identifier. For example, assuming that the question identifier is the question identifier of the 5th question, the compressed data corresponding to the question identifier of the 5th question is determined from the storage space, and the compressed data corresponding to the question identifier of the 5th question is the target compressed data. In this way, a quick query of the target compressed data is achieved.

[0150] In step S107, it is determined whether a correctness statistics instruction for the target compressed data is received.

[0151] In the embodiment of the present application, if no accuracy statistics instruction is received, step S108 is entered; if an accuracy statistics instruction for the target compressed data is received, the accuracy of the target compressed data corresponding to the data to be stored is obtained through data processing, and step S109 is entered.

[0152] In some embodiments, the above-mentioned accuracy statistics instruction is merely an exemplary description, and the instruction may also be a real-time analysis instruction for the data to be stored, such as a question answering time distribution statistics instruction and a data recovery analysis instruction.

[0153] In step S108, the target compressed data is output and the process ends.

[0154] In the embodiment of the present application, if no correctness statistics instruction is received, it means that there is no need to continue analyzing and processing the target compressed data, and the target compressed data can be output through the display module.

[0155] In step S109, the target compressed data is decompressed and decoded in sequence to obtain the target answer content of the question to be queried.

[0156] In the embodiment of the present application, when the correct rate statistics instruction is received, the target compression algorithm used for the target compressed data obtained in the above step S1032 is obtained based on the question identifier, and the target decompression algorithm corresponding to the target compression algorithm is determined; the corresponding target encoding method is also obtained based on the question identifier, and the target decoding method corresponding to the target encoding method is determined. Based on this, the target compressed data is first decompressed using the target decompression algorithm to obtain the decompressed data; and then the decompressed data is decoded using the target decoding method, so as to quickly and accurately obtain the target answer content of the question to be queried.

[0157] In step S110, the reference answer content of the question to be queried is obtained.

[0158] In some embodiments, the question to be queried has a reference answer content corresponding to it. For example, the reference answer for the fifth multiple-choice question is C.

[0159] In step S111, the correct answer content that is consistent with the reference answer content is determined from the target answer content.

[0160] In some embodiments, each target answer content is compared with the reference answer content one by one to determine whether each target answer content is consistent with the reference answer content, thereby obtaining a comparison result; and then based on the comparison result, the correct answer content that is consistent with the reference answer content is determined.

[0161] In step S112, the ratio of the correct number of correct answer contents to the total number of target answer contents is determined as the correct answer rate of the question to be queried.

[0162] In an embodiment of the present application, the correct number of correct answer contents is obtained, and the total number of target answer contents is also obtained, and then the ratio of the correct number to the total number is determined as the answer accuracy rate of the question to be queried.

[0163] Exemplarily, assuming that for the fifth multiple-choice question, the number of correct answers is 800 and the total number is 1000, the correct answer rate for the fifth multiple-choice question is 80%.

[0164] In step S113, the correct answer rate of the question to be queried is output.

[0165] In some embodiments, the correct rate of answers to the questions to be queried can be output through icons, texts, etc., to achieve real-time analysis of the target answer content of the questions to be queried.

[0166] In some embodiments, the average response time of the questions to be queried can also be determined by first obtaining the response time of each target answer in the questions to be answered, and then performing weighted averaging on each response time to obtain the average response time of the questions to be queried.

[0167] In some embodiments, the answer rate of the question to be queried can also be determined by determining from each target answer the answer content that is a null value or the empty answer content that represents the answer content as a null value; and then determining the ratio of the number of empty values ​​in the empty answer content to the total number of target answer contents as the answer rate of the question to be queried.

[0168] Through the above steps S105 to S113, it is possible to quickly and accurately query the compressed data of the query question, and to perform real-time and efficient analysis on the target answer content of the query question.

[0169] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0170] The embodiment of the present application proposes a data storage method. Taking questionnaire data as an example, the embodiment of the present application provides a dedicated database for storing questionnaire return data. The amount of questionnaire return data is huge, the storage cost is high, and the main purpose is data analysis. The embodiment of the present application achieves ultra-low-cost storage of return data and high-performance return data analysis capabilities by mapping codes, storing and compressing by topic.

[0171] In some embodiments, in a questionnaire, users often answer from a limited number of options. From the perspective of the question dimension, user answers are highly similar. At the same time, data analysis of recovered data is also carried out based on the questions. Therefore, choosing appropriate mapping encoding and column storage format can greatly reduce storage costs while having higher data analysis performance.

[0172] In the embodiment of the present application, mapping encoding is performed according to the topic dimension and storage is compressed by column, which can greatly reduce the storage cost of the recovered data and improve the analysis performance of the data.

[0173] The embodiment of the present application provides a technical implementation method for a questionnaire recovery database, which mainly uses a queue to collect data and compresses and stores the data in columns according to the answers to the questions. The method can improve the processing speed of questionnaire data, reduce the required storage space, and ensure the integrity and accuracy of the data. In addition, the method can also support real-time data analysis and adapt to various types of questionnaires and answer sheet data formats.

[0174] In some embodiments, see Fig.10 The data storage method provided in the embodiment of the present application includes questionnaire data input 1001, data cleaning 1002, queue processing 1003, data conversion and mapping 1004, column-based compressed storage 1005, disk file generation 1006, and real-time data analysis 1007, which are described in detail below.

[0175] Regarding questionnaire data input 1001, in the embodiment of the present application, the questionnaire data can be input in a variety of ways, such as an online questionnaire system, a mobile phone application software (application, APP), an application program (Application Programming Interface, API) interface, etc. To ensure the integrity and accuracy of the data, the input data should include key information such as the questionnaire number, question number, option number, and user account (Identity Document, ID).

[0176] Regarding data cleaning 1002, data cleaning mainly includes operations such as removing null values, correcting erroneous values, and removing duplicate values ​​to ensure data quality and prevent attackers from submitting data values ​​that do not meet the requirements of the topic. Data cleaning is an important step in data preprocessing, including deleting duplicate values, filling missing values, handling abnormal values, data type conversion, invalid data deletion, merging data sets, and format unification.

[0177] For queue processing 1003, questionnaire data is collected in the order in which the questionnaires are processed. Fig.11 , the questionnaire data 32 submitted by multiple respondents 31 are cached in a queue until all the questionnaire data are cached; then, the answers to the same questions in the same questionnaire are stored in columns, see Fig.11 , including 5 questionnaires, namely Questionnaire 1, Questionnaire 2, Questionnaire 3, Questionnaire 4 and Questionnaire 5. Among them, taking the multiple-choice question in Questionnaire 1 as an example, this question can correspond to N answers 33, corresponding to Fig.11 Option 1, Option 2, ... Option N; Taking the Chinese question in Questionnaire 1 as an example, there are also N answers 44, and these N answers are of indefinite length, which is reflected in Fig.11 The lengths of the rectangular frames contained in 44 are different.

[0178] Queue processing is used to improve the processing speed of questionnaire data. When new questionnaire data is input, the system will automatically add it to the queue for processing. The queue adopts the first-in-first-out (FIFO) principle to ensure the orderliness of data. At the same time, the queue processing module can also dynamically adjust the queue length according to system resources and processing capabilities to prevent data accumulation.

[0179] For data conversion and mapping 1004, see Fig.12 The questionnaire question types may include multiple-choice questions 1201, drop-down questions 1202, linkage questions 1203, matrix questions 1204, scale questions 1205, and text questions 1206. Among them, the answers to the multiple-choice questions 1201, drop-down questions 1202, linkage questions 1203, and matrix questions 1204 correspond to options 1207, the answers to the scale questions 1205 correspond to numerical values ​​1208, and the answers to the text questions 1206 correspond to texts 1209.

[0180] In some embodiments, data mapping is the process of converting raw questionnaire data into a format that is easy to process and store. Different data mapping schemes can be used depending on the type of question or the form of answer. The following are data mapping schemes for several typical questionnaire question types.

[0181] For single-choice questions, that is, the answer is one option, there are usually a fixed number of options, and each option can be mapped to a unique numerical value. For example, if the options are A, B, C, and D, they can be mapped to 1, 2, 3, and 4 respectively. In this way, the option identifiers in the original questionnaire data can be converted into corresponding numerical values ​​for subsequent processing and storage.

[0182] For multiple-choice questions, that is, the answer is multiple options, the answer to the multiple-choice question usually includes multiple options. In the data mapping process, the answer to the multiple-choice question can be converted into a binary code. Exemplarily, if the options are A, B, C and D, option A can be mapped to 0001, option B can be mapped to 0010, option C can be mapped to 0100, and option D can be mapped to 1000. When the respondent selects multiple options, the corresponding binary codes can be bitwise ORed to obtain a unique code. For example, when the answer is ABD, the answer can be mapped to 1011. This method can effectively represent the answer combination of multiple-choice questions while reducing storage space.

[0183] For scale questions, the answers are numerical. Scale questions usually require respondents to rate a question or statement. In the data mapping process, the rating value can be directly used as the mapping result. For example, if the rating range of the scale question is 1 to 5, the rating given by the user can be directly used as the result of data mapping. This method is simple and intuitive, and is convenient for subsequent processing and analysis.

[0184] For ranking questions, the ranking questions require the respondents to rank the options. In the data mapping process, the ranking position of each option can be mapped to a numerical value. For example, if the options are A, B, C, and D, and the respondents rank them as B, A, D, C, option A can be mapped to 2, option B to 1, option C to 4, and option D to 3. Based on this, the answer BADC can be mapped to 2143.

[0185] For open questions, the answers are usually free text. In the data mapping process, word segmentation and keyword extraction can be performed to convert the text answers into a keyword list. In addition, keywords can be classified and labeled according to actual needs.

[0186] In other embodiments, natural language processing techniques may be used to perform semantic analysis, topic model analysis, etc. on text data to extract more valuable information.

[0187] In some embodiments, for data containing geographic information, the geographic location (such as longitude and latitude, address, etc.) can be mapped to geographic coordinates or region codes to facilitate subsequent geographic information analysis.

[0188] In some embodiments, for time series data, the time data may be mapped into a unified timestamp format to facilitate subsequent time series analysis.

[0189] In some embodiments, when processing open questions, operations such as keyword weight calculation and text clustering can be performed as needed to further mine potential information in text data.

[0190] Continue to see Fig.10 , with respect to column-based compression storage 1005, in the embodiment of the present application, column-based compression is a data compression technology for column-based storage. Compared with row-based compression, column-based compression is more efficient when processing data columns with similar values. The main steps and methods of column-based compression include column segmentation, data type identification, determining compression algorithms corresponding to various topic data types, performing compression, and storing compression results.

[0191] In some embodiments, column segmentation means that before compression by column, the title recovery data needs to be segmented by column. This means that each column is stored separately in a data structure, such as an array or a list. This method can improve the locality of the data, thereby improving the compression effect.

[0192] In some embodiments, data type identification means that when performing column compression, the type of each question as column data needs to be identified. For example, the data may be an integer (option, scale, sorting, etc.), a string (open-ended question), or a date. The corresponding compression algorithm is used to identify the data type.

[0193] In some embodiments, the compression algorithm corresponding to each topic data type is determined by selecting a suitable compression algorithm for each column based on the characteristics of the data type and data distribution. For example, the following are commonly used compression algorithms for different data types:

[0194] Integer data types (options, scales, sorting, etc.) can be compressed using integer encodings (such as variable-length encoding, run-length encoding, etc.). These encoding methods can effectively compress consecutive integer values.

[0195] String data types (open questions) can be compressed using dictionary encoding or prefix encoding (such as LZ77, LZ78, etc.). These encoding methods can effectively compress recurring strings.

[0196] The date data type can convert date data into a timestamp and compress it using integer encoding.

[0197] In some embodiments, performing compression refers to performing compression, applying a selected compression algorithm to each column of data. The compression process can be performed in parallel on a single column to increase compression speed.

[0198] In some embodiments, the compression result storage is to store the compressed data in an appropriate data structure, such as an array or a list. In addition, it is necessary to record the compression information (such as compression algorithm, dictionary size, etc.) of each column of data so as to use it during decompression.

[0199] Through the above process, column-based compression of questionnaire data can be achieved. The advantage of column-based compression is that it improves data storage and query efficiency while reducing storage space requirements. When processing large amounts of questionnaire data, this compression method has good performance and scalability. In future development, the column-based compression algorithm and implementation method can be further optimized according to actual needs to meet the growing data processing needs. Through the above optimization, the storage cost can be optimized by more than 60 times.

[0200] Continue to see Fig.10 , for generating disk files 1006, the compressed data is generated into disk files according to a certain file format (such as Parquet, ORC, etc.). These file formats have the characteristics of high compression and fast query, which is conducive to improving data storage and query efficiency.

[0201] Continue to see Fig.10,For real-time data analysis 1007, the real-time data analysis module takes advantage of column storage to efficiently perform data query, aggregation and calculation. The real-time data analysis function supports a variety of ,analysis scenarios, such as real-time statistics of questionnaire collection, report generation, answer time distribution, answer accuracy rate, etc.

[0202] The data storage method provided by the present application can reduce storage costs and achieve efficient analysis of data, that is, it can reduce the storage cost of recovered data and improve the performance of data analysis.

[0203] It can be understood that in the embodiments of the present application, the collection, use and processing of data related to data to be stored, question type, data type of encoded data, generation time, new data to be stored, question identifier, reference answer content, etc. need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0204] The following is a description of an exemplary structure of the data storage device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the data storage device 455 of the memory 450 may include:

[0205] The first acquisition module 4551 is used to acquire multiple pieces of data to be stored, wherein each piece of data to be stored includes answers corresponding to multiple questions;

[0206] The encoding module 4552 is used to encode the content of each answer according to the question type of each question contained in each piece of data to be stored, so as to obtain each piece of encoded data;

[0207] The compression module 4553 is used to compress the coded data of the multiple answers corresponding to the same question to obtain compressed data;

[0208] The storage module 4554 is used to store the compressed data.

[0209] In some embodiments, the encoding module 4552 is further used to:

[0210] Get the question type of each question;

[0211] According to the encoding method corresponding to each question type, the answer content corresponding to each question type is encoded to obtain the respective encoded data.

[0212] In some embodiments, the encoding module 4552 is further used to:

[0213] When the question type is a single-choice type, obtaining a reference mapping value of the answer content corresponding to the question type, and determining the reference mapping value as the encoded data of the answer content;

[0214] When the question type is a multiple-choice type, obtaining reference binary codes of each option in the answer content corresponding to the question type, performing an OR operation on each reference binary code, and obtaining coded data of the answer content;

[0215] When the question type is a scoring type, determining the answer content corresponding to the question type as the coded data of the answer content;

[0216] When the question type is a sorting type, obtaining reference mapping values ​​of various sorting items in the answer content corresponding to the question type, sorting the reference mapping values ​​of various sorting items according to the arrangement order of the various sorting items, and obtaining the encoded data of the answer content;

[0217] When the question type is a text type, word segmentation processing is performed on the answer content corresponding to the question type to obtain a word segmentation result of the reply content, and the word segmentation result is determined as the encoded data of the reply content.

[0218] In some embodiments, the compression module 4553 is further used to:

[0219] Obtain the coded data of each answer content corresponding to the i-th question, where i is a positive integer less than or equal to N, and N is the total number of questions;

[0220] Determine an i-th compression algorithm for encoding data of each answer content corresponding to the i-th question;

[0221] Based on the i-th compression algorithm, the encoded data of each answer content corresponding to the i-th question is compressed to obtain the i-th compressed data corresponding to the i-th question.

[0222] In some embodiments, the compression module 4553 is further used to:

[0223] Obtaining the i-th data type of the coded data corresponding to the i-th question;

[0224] When the i-th data type is a digital type, determining an integer compression algorithm as the i-th compression algorithm;

[0225] When the i-th data type is a character string type, the character string compression algorithm is determined to be the i-th compression algorithm.

[0226] In some embodiments, the data storage device 455 further includes:

[0227] A second acquisition module is used to acquire the generation time of the plurality of pieces of data to be stored;

[0228] A queue storage module, used to cache the plurality of data to be stored in a queue in the order of generation time from earliest to latest, to obtain a data queue;

[0229] A third acquisition module is used to acquire new data to be stored;

[0230] The updating module is used to cache the new data to be stored at the end of the data queue to obtain an updated data queue.

[0231] In some embodiments, the data storage device 455 further includes:

[0232] A first determining module, configured to determine abnormal data from the plurality of pieces of data to be stored, wherein the abnormal data includes at least one of duplicate data, data of a different format from a reference, data of a different type from a reference, and null value data;

[0233] A cleaning module, used for performing data cleaning on the abnormal data to obtain cleaned data;

[0234] The replacement module is used to replace the abnormal data in the multiple pieces of data to be stored with the corresponding cleaned data to obtain the updated multiple pieces of data to be stored.

[0235] In some embodiments, the storage module 4554 is further used to:

[0236] In response to the received data storage instruction, determining a storage path corresponding to the compressed data;

[0237] According to a preset file format, the compressed data is stored in the storage space corresponding to the storage path.

[0238] In some embodiments, the data storage device 455 further includes:

[0239] A first response module, configured to obtain a topic identifier of a topic to be queried in response to a received query instruction;

[0240] A second determination module, configured to determine the target compressed data corresponding to the topic identifier from the storage space;

[0241] The first output module is used to output the target compressed data.

[0242] In some embodiments, the data storage device 455 further includes:

[0243] A second response module is used to respond to the received correctness statistics instruction for the target compressed data, decompress and decode the target compressed data in sequence, and obtain the target answer content of the question to be queried;

[0244] The third acquisition module is used to obtain the reference answer content of the question to be queried;

[0245] A third determination module is used to determine the correct answer content that is consistent with the reference answer content from the target answer content;

[0246] A fourth determination module is used to determine the correct answer rate of the question to be queried by taking the ratio of the correct number of the correct answer contents to the total number of the target answer contents;

[0247] The second output module is used to output the correct answer rate of the question to be queried.

[0248] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the data storage method described above in the embodiment of the present application.

[0249] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the data storage method provided by the present application embodiment, for example, Figure 3 , Fig. 9 and Fig.10 The data storage method is shown.

[0250] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0251] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0252] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0253] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0254] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data storage method, characterized in that: The method comprises: Acquire multiple pieces of data to be stored, wherein each piece of data to be stored includes answers corresponding to multiple questions; According to the question type of each question contained in each piece of data to be stored, each answer content is encoded to obtain each encoded data; Compressing the coded data of the multiple answers corresponding to the same question to obtain compressed data; The compressed data is stored.

2. The method according to claim 1, characterized in that The method of encoding each answer content according to the type of the question contained in each piece of data to be stored to obtain each encoded data includes: Get the question type of each question; According to the encoding method corresponding to each question type, the answer content corresponding to each question type is encoded to obtain the respective encoded data.

3. The method according to claim 2, characterized in that The encoding method corresponding to each question type is used to encode the answer content corresponding to each question type to obtain the encoded data, including: When the question type is a single-choice type, obtaining a reference mapping value of the answer content corresponding to the question type, and determining the reference mapping value as the encoded data of the answer content; When the question type is a multiple-choice type, obtaining reference binary codes of each option in the answer content corresponding to the question type, performing an OR operation on each reference binary code, and obtaining coded data of the answer content; When the question type is a scoring type, determining the answer content corresponding to the question type as the coded data of the answer content; When the question type is a sorting type, obtaining reference mapping values ​​of various sorting items in the answer content corresponding to the question type, sorting the reference mapping values ​​of various sorting items according to the arrangement order of the various sorting items, and obtaining the encoded data of the answer content; When the question type is a text type, word segmentation processing is performed on the answer content corresponding to the question type to obtain a word segmentation result of the reply content, and the word segmentation result is determined as the encoded data of the reply content.

4. The method according to claim 1, characterized in that The step of compressing the coded data of the multiple answers corresponding to the same question to obtain compressed data includes: Obtain the coded data of each answer content corresponding to the i-th question, where i is a positive integer less than or equal to N, and N is the total number of questions; Determine an i-th compression algorithm for encoding data of each answer content corresponding to the i-th question; Based on the i-th compression algorithm, the encoded data of each answer content corresponding to the i-th question is compressed to obtain the i-th compressed data corresponding to the i-th question.

5. The method according to claim 4, characterized in that The i-th compression algorithm for determining the coded data of the content of each answer corresponding to the i-th question includes: Obtaining the i-th data type of the coded data corresponding to the i-th question; When the i-th data type is a digital type, determining an integer compression algorithm as the i-th compression algorithm; When the i-th data type is a character string type, the character string compression algorithm is determined to be the i-th compression algorithm.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Obtaining generation time of the plurality of pieces of data to be stored; Cache the plurality of data to be stored in a queue in the order of generation time from earliest to latest to obtain a data queue; Get new data to be stored; The new data to be stored is cached at the end of the data queue to obtain an updated data queue.

7. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Determining abnormal data from the plurality of pieces of data to be stored, wherein the abnormal data includes at least one of duplicate data, data of a different format from a reference, data of a different type from a reference, and null value data; Performing data cleaning on the abnormal data to obtain cleaned data; The abnormal data in the plurality of pieces of data to be stored are replaced with the corresponding cleaned data to obtain the updated plurality of pieces of data to be stored.

8. The method according to any one of claims 1 to 5, characterized in that The storing of the compressed data comprises: In response to the received data storage instruction, determining a storage path corresponding to the compressed data; According to a preset file format, the compressed data is stored in the storage space corresponding to the storage path.

9. The method according to claim 8, characterized in that The method further comprises: In response to the received query instruction, obtaining a topic identifier of the topic to be queried; Determining the target compressed data corresponding to the topic identifier from the storage space; Output the target compressed data.

10. The method according to claim 9, characterized in that The method further comprises: In response to the received correctness statistics instruction for the target compressed data, the target compressed data is sequentially decompressed and decoded to obtain the target answer content of the question to be queried; Obtaining the reference answer content of the question to be queried; Determine the correct answer content that is consistent with the reference answer content from the target answer content; The ratio of the correct number of the correct answer contents to the total number of the target answer contents is determined as the correct answer rate of the question to be queried; Output the correct answer rate of the question to be queried.

11. A data storage device, characterized in that: The device comprises: A first acquisition module is used to acquire multiple pieces of data to be stored, wherein each piece of data to be stored includes answers corresponding to multiple questions; The encoding module is used to encode the content of each answer according to the question type of each question contained in each piece of data to be stored, so as to obtain each encoded data; A compression module, used for compressing the coded data of multiple answers corresponding to the same question to obtain compressed data; A storage module is used to store the compressed data.

12. A computer device, characterized in that: The computer device comprises: A memory for storing computer executable instructions; A processor, configured to implement the data storage method according to any one of claims 1 to 10 when executing the computer executable instructions stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the data storage method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program or computer executable instructions, characterized in that: When the computer program or computer executable instructions are executed by a processor, the data storage method according to any one of claims 1 to 10 is implemented.

Images