Data extraction method and device, electronic equipment and storage medium

Abstract

The present application relates to data processing technology for the financial field, and discloses a data extraction method, comprising: data encoding of business vector data according to the data format of the business vector data, obtaining a plurality of encoded data and storing into an in-memory database, request type analysis of a data extraction request, obtaining a type analysis result, calling a synchronous connector to connect a sender and the in-memory database when the type analysis result meets a first preset condition, calling an asynchronous connector to connect the sender and the in-memory database when the type analysis result meets a second preset condition, data decoding of the extracted data according to the memory return value of the in-memory database when the reference connector is connected with the in-memory database, and obtaining standard extraction data. The present application also provides a data extraction device, an electronic device and a storage medium. The present application can improve the efficiency of data extraction in the financial field such as insurance transactions.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular to a data extraction method and device, electronic equipment and storage medium. BACKGROUND

[0002] A large amount of transaction data is usually generated in different business transaction scenarios in the financial field, for example: in the insurance business, the policy transaction data is generally stored in the database for subsequent data reading and data processing of the read data. The database can be a Redis database, and Redis is an in-memory non-relational data storage service.

[0003] Since the transaction data such as policy transaction data may be data converted by an algorithm, and the vector data used by various algorithms often occupies a large amount of memory, which can cause serious database blocking, thereby affecting the size of the data storage space and the efficiency of data extraction. Therefore, a data extraction method is proposed. SUMMARY

[0004] The present application provides a data extraction method, device, electronic equipment and storage medium, which mainly aims to improve the efficiency of data extraction.

[0005] To achieve the above purpose, the present application provides a data extraction method, comprising:

[0006] Obtain a plurality of business vector data, respectively identify the data format of the plurality of business vector data, and encode the business vector data according to the data format to obtain a plurality of encoded data, and store the plurality of encoded data in a preset in-memory database;

[0007] Receive the data extraction request sent by the sender, analyze the request type of the data extraction request, and obtain a type analysis result;

[0008] When the type analysis result meets the first preset condition, a synchronous connector in the reference connector is called to connect the sender and the in-memory database, and when the type analysis result meets the second preset condition, an asynchronous connector in the reference connector is called to connect the sender and the in-memory database;

[0009] When the reference connector is connected with the in-memory database, the extracted data is decoded according to the memory return value of the in-memory database to obtain standard extraction data.

[0010] Optionally, the data format of the plurality of business vector data is identified respectively, comprising:

[0011] An acquisition type judgment function and a plurality of reference types are combined with the service vector data to obtain a plurality of input combinations.

[0012] The input combinations are judged according to the type judgment function, and when the judgment is passed, the reference type in the input combination is taken as the data format of the service vector data.

[0013] Optionally, the service vector data is encoded according to the data format to obtain a plurality of encoded data, comprising:

[0014] The data memory of the service vector data is identified to obtain a data memory result.

[0015] When the data memory result is a first memory data, the service vector data is encoded according to a preset type encoding algorithm and the data format to obtain a plurality of encoded data.

[0016] When the data memory result is a second memory data, the service vector data is format-converted to obtain a plurality of encoded data.

[0017] Optionally, the service vector data is encoded according to a preset type encoding algorithm and the data format to obtain a plurality of encoded data, comprising:

[0018] When the data format of the service vector data is a list type, the service vector data is twice format-converted, and the twice format-converted data is compressed to obtain encoded data.

[0019] When the data format of the service vector data is a dictionary type, the service vector data is serialized, and the serialized data is compressed to obtain encoded data.

[0020] When the data format of the service vector data is a string type, the data format of the service vector data is converted to a byte format, and the byte format data is compressed to obtain encoded data.

[0021] Optionally, the data extraction request is analyzed to obtain a type analysis result, comprising:

[0022] The data extraction request is parsed to obtain a request content.

[0023] If the request content contains a requirement for more than or equal to a preset number of data, the type analysis result is output as a first type.

[0024] If the request content contains a requirement of less than a preset number of data, the type analysis result is output as a second type.

[0025] Optionally, the data decoding of the extracted data according to the memory return value of the memory database comprises:

[0026] The memory return value is compared with a reference value, and a type corresponding to the reference value consistent with the comparison is taken as a return type of the memory return value.

[0027] The extracted data is decoded according to different return types to obtain standard extracted data.

[0028] Optionally, the decoding of the extracted data according to different return types to obtain standard extracted data comprises:

[0029] When the return type is a list type, the memory return value is decompressed, and the decompressed result is format-converted to obtain standard extracted data.

[0030] When the return type is a dictionary type, the memory return value is deserialized, and the deserialized data is converted into a dictionary, and the dictionary is taken as standard extracted data.

[0031] When the return type is a string type, the memory return value is decompressed to obtain standard extracted data.

[0032] To solve the above problems, the application further provides a data extraction device, which comprises:

[0033] A data encoding module is configured to acquire a plurality of service vector data, identify data formats of the plurality of service vector data respectively, encode the service vector data according to the data formats, obtain a plurality of encoded data, and store the plurality of encoded data in a preset memory database.

[0034] A type analysis module is configured to receive a data extraction request sent by a sender, analyze a request type of the data extraction request, and obtain a type analysis result.

[0035] A database connection module is configured to connect the sender and the memory database by calling a synchronous connector in a reference connector when the type analysis result meets a first preset condition, and connect the sender and the memory database by calling an asynchronous connector in the reference connector when the type analysis result meets a second preset condition.

[0036] The data decoding module is configured to decode the extracted data according to a memory return value of the in-memory database to obtain standard extracted data when the reference connector is connected to the in-memory database.

[0037] To solve the above problems, the present application also provides an electronic device, which comprises:

[0038] at least one processor; and

[0039] a memory in communication with the at least one processor; wherein

[0040] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the data extraction method.

[0041] To solve the above problems, the present application also provides a storage medium, which stores at least one computer program executable by a processor in an electronic device to implement the data extraction method.

[0042] In the embodiment of the present application, the data format of the service vector data is identified, and the service vector data is encoded according to the data format to obtain a plurality of pieces of encoded data, and the plurality of pieces of encoded data are stored in the in-memory database. The data encoding process can reduce the memory occupied by the data and improve the environmental adaptability. The request type of the data extraction request is analyzed to obtain a type analysis result, and a synchronous connector or an asynchronous connector is selected according to the type analysis result to improve the access efficiency, thereby improving the data extraction efficiency. Therefore, the data extraction method, device, electronic device and storage medium provided by the present application can solve the problem of low efficiency of policy transaction data extraction. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A flowchart of a data extraction method provided by an embodiment of the present application is shown in the figure.

[0044] Figure 2 A detailed implementation flowchart of one of the steps in the figure. Figure 1

[0045] Figure 3 A functional module diagram of a data extraction device provided by an embodiment of the present application is shown in the figure.

[0046] Figure 4 A structural diagram of an electronic device for implementing the data extraction method provided by an embodiment of the present application is shown in the figure.

[0047] ​The objectives, functional features and advantages of the present application will be further illustrated in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

[0049] Embodiments of the present application provide a data extraction method. The execution subject of the data extraction method includes but is not limited to at least one of electronic devices such as a server and a terminal, which can be configured to execute the method provided by the embodiments of the present application. In other words, the data extraction method can be executed by software or hardware installed in a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to a single server, a server cluster, a cloud server, or a cloud server cluster, etc. The server can be a stand-alone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and big data and artificial intelligence platforms, etc. basic cloud computing services.

[0050] Reference Figure 1 As shown in the figure, the flowchart of the data extraction method provided by an embodiment of the present application.

[0051] In the present embodiment, the data extraction method includes the following steps S1-S4:

[0052] S1, obtain a plurality of service vector data, respectively identify the data format of the plurality of service vector data, and encode the service vector data according to the data format to obtain a plurality of encoded data, and store the plurality of encoded data in a preset memory database.

[0053] In the embodiment of the present application, the plurality of service vector data refers to the data generated in the transaction process in the field of financial technology, for example, the transaction data generated by the policy transaction in the insurance business. Since the transaction data is data converted by an algorithm, it is vector data. Different service vector data has different data formats, and when data encoding processing is performed, corresponding data encoding processing can be performed according to different format data, thereby improving the accuracy of data encoding.

[0054] Specifically, the data format of the plurality of service vector data is identified respectively, including:

[0055] Obtain a type judgment function and a plurality of reference types, randomly combine the service vector data with the plurality of reference types to obtain a plurality of input combinations;

[0056] According to the type judgment function, the input combination is judged, and when the judgment is passed, the reference type in the input combination is taken as the data format of the business vector data.

[0057] In detail, the type judgment function is the isinstance() function, and isinstance() is a built-in function in Python, which is used to judge the variable type of an object. The reference type can be a list type, a dictionary type, and a string type.

[0058] For example, the business vector data is a policy transaction data A, and the policy transaction data A is randomly combined with multiple reference types such as a list type, a dictionary type, and a string type to obtain multiple input combinations (policy transaction data A, list type), (policy transaction data A, dictionary type), and (policy transaction data A, string type). According to the type judgment function, the input combination is judged, that is, isinstance(policetransaction data A, list type), isinstance(policetransaction data A, dictionary type), and isinstance(policetransaction data A, string type) are constructed, and when the judgment is passed, that is, True is output, the reference type in the input combination is taken as the data format of the business vector data.

[0059] Further, referring to Figure 2 According to the data format, the business vector data is data encoded to obtain multiple pieces of encoded data, including the following steps S21-S23:

[0060] S21, identifying the data memory of the business vector data to obtain a data memory result;

[0061] S22, when the data memory result is first memory data, the business vector data is encoded according to a preset type encoding algorithm and the data format to obtain multiple pieces of encoded data;

[0062] S23, when the data memory result is second memory data, the business vector data is format-converted to obtain multiple pieces of encoded data.

[0063] In detail, the first memory data is large memory data, and the second memory data is small memory data.

[0064] Specifically, the business vector data is encoded according to the preset type encoding algorithm and the data format to obtain multiple pieces of encoded data, including:

[0065] When the data format of the service vector data is a list type, the service vector data is subjected to secondary format conversion, and the data subjected to the secondary format conversion is subjected to compression processing to obtain encoded data.

[0066] When the data format of the service vector data is a dictionary type, the service vector data is subjected to serialization processing, and the data subjected to the serialization processing is subjected to compression processing to obtain encoded data.

[0067] When the data format of the service vector data is a string type, the data format of the service vector data is converted to a byte format, and the data in the byte format is subjected to compression processing to obtain encoded data.

[0068] In detail, the secondary format conversion refers to converting the service vector data into a numpyarray format and then into a byte format, and then performing compression.

[0069] Preferably, when the data memory result is second memory data, the service vector data is subjected to format conversion, and the format conversion method is consistent with that of the large memory data, but the compression processing is excluded.

[0070] The encoding of the policy transaction data not only optimizes the space, but also optimizes the network transmission, because large memory reading and writing also cause network transmission blocking.

[0071] Specifically, the plurality of encoded data is stored in a preset memory database, and the memory database is Redis. Redis is a database, but unlike traditional databases, its data exists in memory, so the reading and writing speed is fast.

[0072] S2, receiving the data extraction request sent by the sender, performing request type analysis on the data extraction request to obtain a type analysis result.

[0073] In the embodiment of the application, the data extraction request sent by the sender refers to the request data demand sent by the program.

[0074] Specifically, the request type analysis on the data extraction request to obtain a type analysis result comprises:

[0075] performing request analysis on the data extraction request to obtain request content;

[0076] If the request content contains a requirement for more than or equal to a preset number of data, the type analysis result is output as a first type;

[0077] If the request content contains a requirement of less than a preset number of data, the type analysis result is output as a second type.

[0078] In detail, the request content containing a requirement of more than or equal to a preset number of data refers to that the program requests a large amount of policy transaction data from Redis at a time, and the request content containing a requirement of less than a preset number of data refers to that the program requests a small amount of policy transaction data from Redis multiple times.

[0079] The request type analysis on the data extraction request can facilitate subsequent provision of different connectors for data extraction, thereby improving the efficiency of policy transaction data extraction.

[0080] S3, when the type analysis result meets a first preset condition, a synchronous connector in a reference connector is called to connect the sender and the memory database, and when the type analysis result meets a second preset condition, an asynchronous connector in the reference connector is called to connect the sender and the memory database.

[0081] In the embodiment of the application, the first preset condition refers to a first type, and the second preset condition refers to a second type. The reference connector can be a synchronous connector or an asynchronous connector. The synchronous connector is used when the type analysis result meets the first preset condition.

[0082] Specifically, when the type analysis result meets the first preset condition, i.e., the program needs to request a large amount of data from Redis at a time, a synchronous connector is used because pipeline mode needs to be used at this time. The pipeline mode is a pipeline mode, which processes the input policy transaction data through a series of stages set in advance. The characteristic of this mode is that Redis will compress multiple request commands into one command for execution. According to this characteristic, the use of a synchronous connector at this time can reduce the time consumption caused by frequent network transmission.

[0083] Further, when the type analysis result meets the second preset condition, i.e., the program needs to request a small amount of policy transaction data from Redis multiple times, an asynchronous connector is used because discrete zadd mode needs to be used multiple times at this time to add elements to the set in order, hmset mode is used to set field values to the hash table, sadd mode is used to add one or more elements to the set, and set mode is used to automatically exclude duplicate elements. Frequent network waiting will become a bottleneck, so the use of an asynchronous connector can avoid a large amount of network waiting time.

[0084] S4, when the reference connector is connected with the memory database, the extracted data is decoded according to the memory return value of the memory database to obtain standard extraction data.

[0085] The embodiment of the application comprises the following steps:

[0086] The memory return value is compared with a reference value, and a type corresponding to a reference value consistent with the comparison is taken as a return type of the memory return value;

[0087] The extracted data is decoded according to different return types to obtain standard extracted data.

[0088] In detail, inverse operations are performed according to which encoding type the Redis return value belongs to.

[0089] Specifically, the decoding of the extracted data according to different return types to obtain standard extracted data comprises the following steps:

[0090] When the return type is a list type, the memory return value is decompressed, and the decompressed result is format-converted to obtain standard extracted data;

[0091] When the return type is a dictionary type, the memory return value is deserialized, and the deserialized data is converted into a dictionary, and the dictionary is taken as standard extracted data;

[0092] When the return type is a string type, the memory return value is decompressed to obtain standard extracted data.

[0093] In detail, inverse operations are performed according to which encoding type the Redis return value belongs to, for example, when the return value is large memory list data, byte is first converted into a multi-dimensional array numpyarray after decompression; when the return value is large memory dictionary, byte is first converted into a dictionary after pickle deserialization, wherein pickle is a stack-based programming language.

[0094] Further, after the standard extracted data is obtained, the method further comprises the following steps:

[0095] The standard extracted data is transmitted to the sending end.

[0096] In the embodiment of the present application, the data format of the service vector data is identified, the service vector data is encoded according to the data format, a plurality of encoded data is obtained, and the plurality of encoded data is stored in the memory database. The data encoding process can reduce the memory occupied by the data and improve the environmental adaptability. The request type analysis is performed on the data extraction request to obtain a type analysis result, and the synchronous connector or the asynchronous connector is selected for connection according to the type analysis result, thereby improving the access efficiency and further improving the data extraction efficiency. Therefore, the data extraction method provided by the present application can improve the low efficiency of the policy transaction data extraction.

[0097] As shown in Figure 3 Fig. 1 is a functional module diagram of a data extraction device provided by an embodiment of the present application.

[0098] The data extraction device 100 can be installed in an electronic device. According to the implemented functions, the data extraction device 100 can include a data encoding module 101, a type analysis module 102, a database connection module 103, and a data decoding module 104. The modules of the present application can also be referred to as units, which refer to a series of computer program segments that can be executed by an electronic device processor and can complete a fixed function, and are stored in the memory of the electronic device.

[0099] In the embodiment, the functions of each module / unit are as follows:

[0100] The data encoding module 101 is configured to obtain a plurality of service vector data, identify the data format of each of the plurality of service vector data, encode the service vector data according to the data format, obtain a plurality of encoded data, and store the plurality of encoded data in a preset memory database.

[0101] The type analysis module 102 is configured to receive a data extraction request sent by a sender, analyze the request type of the data extraction request, and obtain a type analysis result.

[0102] The database connection module 103 is configured to connect the sender and the memory database by calling a synchronous connector in a reference connector when the type analysis result meets a first preset condition, and connect the sender and the memory database by calling an asynchronous connector in the reference connector when the type analysis result meets a second preset condition.

[0103] The data decoding module 104 is configured to decode the extracted data according to the memory return value of the memory database when the reference connector is connected to the memory database, and obtain standard extraction data.

[0104] In detail, the specific implementation of each module of the data extraction device 100 is as follows:

[0105] Step one, obtaining a plurality of service vector data, respectively identifying the data format of the plurality of service vector data, and according to the data format, data encoding is performed on the service vector data to obtain a plurality of encoded data, and the plurality of encoded data is stored in a preset memory database.

[0106] In the embodiment of the application, the plurality of service vector data refers to data generated in the transaction process in the field of financial technology, for example, transaction data generated by a policy transaction in an insurance business. Since the transaction data is data converted by an algorithm, it is vector data. Different service vector data has different data formats, and when data encoding processing is performed, corresponding data encoding processing can be performed according to different format data, thereby improving the accuracy of data encoding.

[0107] Specifically, the data format of the plurality of service vector data is identified, including:

[0108] Obtaining a type judgment function and a plurality of reference types, randomly combining the service vector data with the plurality of reference types to obtain a plurality of input combinations;

[0109] According to the type judgment function, the plurality of input combinations are respectively judged, and when the judgment processing is judged to be passed, the reference type in the input combination is taken as the data format of the service vector data.

[0110] In detail, the type judgment function is an isinstance() function, and isinstance() is a built-in function in Python, which is used to judge the variable type of an object. The plurality of reference types can be list type, dictionary type and string type.

[0111] For example, the service vector data is policy transaction data A, the policy transaction data A is randomly combined with a plurality of reference types such as list type, dictionary type and string type to obtain a plurality of input combinations (policy transaction data A, list type), (policy transaction data A, dictionary type) and (policy transaction data A, string type), and according to the type judgment function, the plurality of input combinations are respectively judged, that is, isinstance (policy transaction data A, list type), isinstance (policy transaction data A, dictionary type) and isinstance (policy transaction data A, string type) are constructed, and when the judgment processing is judged to be passed, that is, True is output, the reference type in the input combination is taken as the data format of the service vector data.

[0112] Further, the data encoding of the service vector data according to the data format comprises:

[0113] identifying data memory of the service vector data to obtain a data memory result;

[0114] when the data memory result is first memory data, encoding processing of the service vector data according to a preset type encoding algorithm and the data format to obtain a plurality of encoding data;

[0115] when the data memory result is second memory data, format conversion of the service vector data to obtain a plurality of encoding data.

[0116] In detail, the first memory data is large memory data, and the second memory data is small memory data.

[0117] Specifically, the encoding processing of the service vector data according to the preset type encoding algorithm and the data format to obtain a plurality of encoding data comprises:

[0118] when the data format of the service vector data is a list type, secondary format conversion of the service vector data and compression processing of the data after the secondary format conversion to obtain encoding data;

[0119] when the data format of the service vector data is a dictionary type, serialization processing of the service vector data and compression processing of the data after the serialization processing to obtain encoding data;

[0120] when the data format of the service vector data is a string type, conversion of the data format of the service vector data to a byte format and compression processing of the data in the byte format to obtain encoding data.

[0121] In detail, the secondary format conversion refers to conversion of the service vector data to a numpyarray format and then to a byte format, and then compression, and the serialization processing refers to pickle.

[0122] Preferably, when the data memory result is the second memory data, the format conversion of the service vector data is consistent with the large memory data, but the compression processing is excluded.

[0123] The encoding of the policy transaction data not only optimizes the space, but also optimizes the network transmission, because large memory reading and writing also cause network transmission blockage.

[0124] Specifically, a plurality of the encoding data is stored into a preset memory database, wherein the memory database is Redis.

[0125] Step two, receiving the data extraction request sent by the sender, analyzing the request type of the data extraction request to obtain a type analysis result.

[0126] In the embodiment of the application, the data extraction request sent by the sender refers to the request data requirement sent by the program.

[0127] Specifically, the request type analysis of the data extraction request is performed to obtain a type analysis result, comprising:

[0128] The request content is obtained by performing request analysis on the data extraction request;

[0129] If the request content contains a requirement for more than or equal to a preset number of data, the type analysis result is output as a first type;

[0130] If the request content contains a requirement for less than a preset number of data, the type analysis result is output as a second type.

[0131] In detail, the request content contains a requirement for more than or equal to a preset number of data when the program requests a large amount of insurance transaction data from Redis at a time, and the request content contains a requirement for less than a preset number of data when the program requests a small amount of insurance transaction data from Redis multiple times.

[0132] The request type analysis of the data extraction request can facilitate subsequent data extraction by different connectors, thereby improving the efficiency of insurance transaction data extraction.

[0133] Step three, when the type analysis result meets a first preset condition, a synchronous connector in a reference connector is called to connect the sender and the memory database, and when the type analysis result meets a second preset condition, an asynchronous connector in the reference connector is called to connect the sender and the memory database.

[0134] In the embodiment of the application, the first preset condition refers to the first type, and the second preset condition refers to the second type. The reference connector can be a synchronous connector or an asynchronous connector. The synchronous connector.

[0135] Specifically, when the type analysis result meets the first preset condition, i.e., the program needs to request a large amount of data from Redis at one time, a synchronous connector is used, because pipeline mode needs to be used at this time, pipeline mode is a mode for processing input insurance transaction data through a series of stages set in advance, and the characteristic of the mode is that Redis compresses multiple request commands into one command for execution, according to the characteristic, using the synchronous connector at this time can reduce the time consumption caused by frequent network transmission.

[0136] Further, when the type analysis result meets the second preset condition, i.e., when the program needs to request a small amount of insurance transaction data from redis multiple times, an asynchronous connector is used, because discrete zadd mode needs to be used multiple times at this time for sequentially adding elements to a set, hmset mode is used for setting field values to a hash table, sadd mode is used for adding one or more elements to a set, set mode is used for automatically excluding duplicate elements, and frequent network waiting will become a bottleneck, so the asynchronous connector can avoid a large amount of network waiting time.

[0137] Step four, when the reference connector is connected with the memory database, the extracted data is decoded according to the memory return value of the memory database, and standard extraction data is obtained.

[0138] In the embodiment of the application, the decoding of the extracted data according to the memory return value of the memory database to obtain standard extraction data comprises:

[0139] The memory return value is compared with a reference value, and the type corresponding to the reference value consistent with the comparison is taken as the return type of the memory return value;

[0140] The extracted data is decoded according to different return types to obtain standard extraction data.

[0141] In detail, according to which encoding type the Redis return value belongs to, an inverse operation is performed.

[0142] Specifically, the decoding of the extracted data according to different return types to obtain standard extraction data comprises:

[0143] When the return type is a list type, the memory return value is decompressed, and the decompressed result is format-converted to obtain standard extraction data;

[0144] When the return type is a dictionary type, the memory return value is deserialized, and the deserialized data is converted into a dictionary, and the dictionary is taken as standard extraction data;

[0145] When the return type is a string type, the memory return value is decompressed to obtain standard extraction data.

[0146] In detail, according to which encoding type the Redis return value belongs to, a reverse operation is performed, for example, when the return value is a large memory list data, first decompression is performed, and then byte is converted into a multi-dimensional array numpyarray; when the return value is a large memory dictionary, first pickle deserialization is performed, wherein pickle is a stack-based programming language, and then byte is converted into a dictionary.

[0147] Further, after the standard extraction data is obtained, the method further comprises:

[0148] The standard extraction data is transmitted to the sending end.

[0149] In the embodiment of the application, the data format of the service vector data is recognized, the service vector data is encoded according to the data format, a plurality of pieces of encoded data are obtained, and the plurality of pieces of encoded data are stored in the memory database, the data encoding processing can reduce the memory occupied by the data and improve environmental adaptability. The request type analysis of the data extraction request is performed to obtain a type analysis result, and a synchronous connector or an asynchronous connector is selected for connection according to the type analysis result, thereby improving the access efficiency and further improving the data extraction efficiency. Therefore, the data extraction device provided by the application can improve the low efficiency of the policy transaction data extraction.

[0150] As shown in Figure 4 Fig. 1 is a structural schematic diagram of an electronic device for implementing a data extraction method according to an embodiment of the application.

[0151] The electronic device 1 can include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and can further include a computer program, such as a data extraction program, stored in the memory 11 and executable on the processor 10.

[0152] The processor 10 may, in some embodiments, be composed of integrated circuits, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits of the same or different functions, including one or more combinations of central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connects various components of the electronic device through various interfaces and lines, executes programs or modules stored in the memory 11 (for example, executes a data extraction program, etc.), and calls data stored in the memory 11 to perform various functions and process data of the electronic device.

[0153] The memory 11 includes at least one type of readable storage medium, including a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. The memory 11 may, in some embodiments, be an internal storage unit of the electronic device, for example, a mobile hard disk of the electronic device. The memory 11 may, in other embodiments, also be an external storage device of the electronic device, for example, a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 11 may include both an internal storage unit and an external storage device of the electronic device. The memory 11 can be used not only to store application software and various data installed in the electronic device, for example, the code of the data extraction program, etc., but also to temporarily store data that has been output or will be output.

[0154] The communication bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable connection and communication between the memory 11 and at least one processor 10, etc.

[0155] The communication interface 13 is used for communication between the electronic device and other devices, including a network interface and a user interface. Optionally, the network interface can include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the electronic device and other electronic devices. The user interface can be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface can also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch screen, etc. Among them, the display can also be appropriately called a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0156] Figure 4 Only the electronic device with components is shown, and those skilled in the art can understand that, Figure 4 The structure shown does not constitute a limitation on the electronic device 1, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0157] For example, although not shown, the electronic device can also include a power source (such as a battery) to power each component. Preferably, the power source can be logically connected to the at least one processor 10 through a power management device, so that functions such as charge management, discharge management, and power consumption management can be realized through the power management device. The power source can also include one or more direct current or alternating current power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and any other components. The electronic device can also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which are not described here.

[0158] It should be understood that the embodiments are only for illustration and are not limited in the scope of the patent application by this structure.

[0159] The data extraction program stored in the memory 11 in the electronic device 1 is a combination of multiple instructions, which, when executed in the processor 10, can achieve:

[0160] Obtain a plurality of service vector data, respectively identify the data format of the plurality of service vector data, and perform data encoding on the service vector data according to the data format to obtain a plurality of encoded data, and store the plurality of encoded data in a preset memory database;

[0161] Receive a data extraction request sent by a sender, analyze the request type of the data extraction request to obtain a type analysis result;

[0162] When the type analysis result meets a first preset condition, a synchronous connector in the reference connector is called to connect the sender and the in-memory database, and when the type analysis result meets a second preset condition, an asynchronous connector in the reference connector is called to connect the sender and the in-memory database.

[0163] When the reference connector is connected with the in-memory database, the extracted data is decoded according to a memory return value of the in-memory database to obtain standard extracted data.

[0164] Specifically, the specific implementation method of the processor 10 to the above instructions can refer to the description of the related steps in the corresponding embodiment of the drawings, and will not be described here.

[0165] Further, the modules / units integrated in the electronic device 1 are implemented in the form of software function units and sold or used as independent products, which can be stored in a storage medium. The storage medium can be volatile or non-volatile. For example, the computer readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM).

[0166] The application also provides a storage medium, and the readable storage medium stores a computer program, and the computer program can realize the following when executed by a processor of an electronic device:

[0167] A plurality of service vector data are acquired, data formats of the plurality of service vector data are identified respectively, the service vector data are encoded according to the data formats to obtain a plurality of encoded data, and the plurality of encoded data are stored in a preset in-memory database.

[0168] A data extraction request sent by a sender is received, and a request type of the data extraction request is analyzed to obtain a type analysis result.

[0169] When the type analysis result meets a first preset condition, a synchronous connector in the reference connector is called to connect the sender and the in-memory database, and when the type analysis result meets a second preset condition, an asynchronous connector in the reference connector is called to connect the sender and the in-memory database.

[0170] When the reference connector is connected with the in-memory database, the extracted data is decoded according to a memory return value of the in-memory database to obtain standard extracted data.

[0171] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other manners. For example, the described apparatus embodiments are merely illustrative. For example, the division of the modules is merely logical function division. There can be another division manner in actual implementation.

[0172] The modules illustrated as separated components can or can not be physically separated, and the components illustrated as modules can or can not be physical units, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0173] In addition, each function module in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software function modules.

[0174] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

[0175] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any additional reference signs in the claims should not be considered as limiting the claims involved.

[0176] The blockchain referred to in the present application is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies. Blockchain, in essence, is a decentralized database, which is a series of data blocks associated using cryptography. Each data block contains information about a batch of network transactions, used to verify the validity (anti-fake) of the information and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0177] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Theory, method, technology and application system.

[0178] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. Terms such as "first" and "second" are used to indicate names and do not imply any particular order.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A data extraction method, characterized by, The method comprises: Obtaining a plurality of service vector data, respectively identifying the data format of the plurality of service vector data, and data encoding the service vector data according to the data format to obtain a plurality of encoded data, and storing the plurality of encoded data in a preset in-memory database; Receiving a data extraction request sent by a sender, analyzing the request type of the data extraction request to obtain a type analysis result; When the type analysis result meets a first preset condition, a synchronous connector in a reference connector is called to connect the sender and the in-memory database in a pipeline mode, and when the type analysis result meets a second preset condition, an asynchronous connector in the reference connector is called to connect the sender and the in-memory database multiple times using discretized zadd, hmset, sadd and set commands, the first preset condition being that the data extraction request indicates a one-time request for a large amount of data, and the second preset condition being that the data extraction request indicates a multiple-time request for a small amount of data; When the reference connector is connected to the in-memory database, the extracted data is decoded according to the memory return value of the in-memory database to obtain standard extraction data; The data encoding of the service vector data according to the data format comprises: identifying the data memory of the service vector data to obtain a data memory result; when the data memory result is large memory data, performing format conversion or serialization processing on the service vector data according to a preset type encoding algorithm and the data format, and compressing the processed data to obtain a plurality of encoded data; when the data memory result is small memory data, performing format conversion on the service vector data to obtain a plurality of encoded data.

2. The data extraction method of claim 1, wherein, The respective identification of the data format of the plurality of service vector data comprises: Obtaining a type judgment function and a plurality of reference types, randomly combining the service vector data with the plurality of reference types to obtain a plurality of input combinations; According to the type judgment function, the input combinations are respectively judged, and when the judgment is passed, the reference type in the input combination is taken as the data format of the service vector data.

3. The data extraction method of claim 1, wherein, The format conversion or serialization processing of the service vector data according to the preset type encoding algorithm and the data format, and the compression of the processed data to obtain a plurality of encoded data, comprises: When the data format of the service vector data is a list type, the service vector data is converted twice, and the data after the second format conversion is compressed to obtain encoded data; When the data format of the service vector data is a dictionary type, the service vector data is serialized, and the serialized data is compressed to obtain encoded data; When the data format of the service vector data is a string type, the data format of the service vector data is converted to a byte format, and the byte format data is compressed to obtain encoded data.

4. The data extraction method of claim 1, wherein, The data decoding of the extracted data according to the memory return value of the memory database obtains standard extracted data, and the data decoding of the extracted data according to different return types includes: The memory return value is compared with a reference value, and a type corresponding to a consistent reference value is taken as a return type of the memory return value; The extracted data is decoded according to different return types to obtain standard extracted data.

5. The data extraction method of claim 4, wherein, The data decoding of the extracted data according to the memory return value of the memory database obtains standard extracted data, and the data decoding of the extracted data according to different return types includes: When the return type is a list type, the memory return value is decompressed, and the decompressed result is format-converted to obtain standard extracted data; When the return type is a dictionary type, the memory return value is deserialized, and the deserialized data is converted into a dictionary, and the dictionary is taken as standard extracted data; When the return type is a string type, the memory return value is decompressed to obtain standard extracted data.

6. A data extraction apparatus for implementing the data extraction method according to any one of claims 1 to 5, characterized by, The device includes: A data encoding module is configured to obtain a plurality of service vector data, identify data formats of the plurality of service vector data respectively, encode the service vector data according to the data formats to obtain a plurality of encoded data, and store the plurality of encoded data in a preset memory database; A type analysis module is configured to receive a data extraction request sent by a sender, analyze a request type of the data extraction request to obtain a type analysis result; A database connection module is configured to connect the sender and the memory database by calling a synchronous connector in a reference connector when the type analysis result meets a first preset condition, and connect the sender and the memory database by calling an asynchronous connector in the reference connector when the type analysis result meets a second preset condition; A data decoding module is configured to decode extracted data according to a memory return value of the memory database when the reference connector is connected with the memory database to obtain standard extracted data.

7. An electronic device, comprising: The electronic device includes: at least one processor; and a memory connected with the at least one processor in communication; wherein The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the data extraction method of any one of claims 1 to 5.

8. A storage medium storing a computer program, characterized by The computer program is executed by the processor to implement the data extraction method of any one of claims 1 to 5.

Images