Unstructured data processing method, device, electronic device and storage medium

Abstract

An embodiment of the present invention provides an unstructured data processing method, device, electronic device and storage medium. The unstructured data processing method includes obtaining a plurality of original unstructured data and a plurality of original description data, and there is a corresponding relationship between the plurality of original unstructured data and the plurality of original description data. A relational data table is constructed based on the corresponding relationship between the plurality of original unstructured data and the plurality of original description data. The plurality of original unstructured data and the plurality of original description data are spliced ​​according to the relational data table to generate a target storage path for each of the plurality of original unstructured data. The target storage path is saved to a data warehouse. The solution of the embodiment of the present invention can quickly find the corresponding original unstructured data according to the target storage path when the original unstructured data is needed, thereby improving the efficiency of obtaining unstructured data.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a method, device, electronic device, and storage medium for processing unstructured data. Background Art

[0002] With the development of intelligent driving models and intelligent recognition models, model calculation and training require a large amount of uncompressed unstructured data such as images and videos.

[0003] Due to network bandwidth limitations, existing technologies typically store unstructured data collected by vehicles in the cloud as compressed data. Model calculation and training require a series of operations, including downloading, decompression, and matching, to retrieve the raw, uncompressed data.

[0004] This data processing method results in low efficiency in obtaining uncompressed raw unstructured data. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an unstructured data processing method, device, electronic device, and storage medium to at least solve the above-mentioned problems.

[0006] According to a first aspect of an embodiment of the present invention, a method for processing unstructured data is provided. The method includes obtaining a plurality of original unstructured data and a plurality of original description data, wherein the plurality of original unstructured data and the plurality of original description data have a corresponding relationship. A relational data table is constructed based on the corresponding relationship between the plurality of original unstructured data and the plurality of original description data. The plurality of original unstructured data and the plurality of original description data are concatenated according to the relational data table to generate a target storage path for each of the plurality of original unstructured data. The target storage paths are saved in a data warehouse.

[0007] In one implementation, obtaining the plurality of raw unstructured data and the plurality of raw description data includes obtaining a data compression package of an unstructured data file, the data compression package including the plurality of compressed unstructured data and a compressed description data table. The data compression package is decompressed to obtain the plurality of raw unstructured data and the raw description data table, the raw description data table being used to store a plurality of data information of the plurality of raw unstructured data, wherein each piece of data information includes a data type, a byte length, a data number, and an acquisition device number. The raw description data table is traversed to determine the plurality of data information for the plurality of raw unstructured data as the plurality of raw description data.

[0008] In another implementation, obtaining the data compression package of the unstructured data file includes reading an initial storage path of the unstructured data file from a data warehouse, and downloading the data compression package of the unstructured data file from the first cloud according to the initial storage path.

[0009] In another implementation, obtaining the data compression package of the unstructured data file includes uploading the unstructured data file to the first cloud and saving the initial storage path to the data warehouse.

[0010] In another implementation, the unstructured data processing method further includes uploading the plurality of original unstructured data to a second cloud.

[0011] In another implementation, uploading the plurality of original unstructured data to the second cloud includes converting the plurality of original unstructured data into a binary byte stream, and uploading the binary byte stream to the second cloud.

[0012] In another implementation, converting the plurality of raw unstructured data into a binary byte stream includes determining whether an input account password is consistent with a preset account password. If the input account password is consistent with the preset account password, executing a conversion instruction for the plurality of raw unstructured data to convert the plurality of raw unstructured data into a binary byte stream.

[0013] According to a second aspect of an embodiment of the present invention, an unstructured data processing device is provided, comprising an acquisition module for acquiring a plurality of original unstructured data and a plurality of original description data, wherein the plurality of original unstructured data and the plurality of original description data have a corresponding relationship; a construction module for constructing a relational data table based on the corresponding relationship between the plurality of original unstructured data and the plurality of original description data. A generation module for concatenating the plurality of original unstructured data and the plurality of original description data according to the relational data table to generate a target storage path for each of the plurality of original unstructured data; and a transmission module for saving the target storage path to a data warehouse.

[0014] According to a third aspect of an embodiment of the present invention, there is provided an electronic device comprising a processor and a memory storing a program, wherein the program comprises instructions, and when the instructions are executed by the processor, the processor executes the method according to the first aspect.

[0015] According to a fourth aspect of an embodiment of the present invention, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method of the first aspect is implemented.

[0016] In the solution of an embodiment of the present invention, a method for processing unstructured data is provided. This method obtains multiple uncompressed raw unstructured data and multiple raw description data and constructs a relational data table of the multiple raw unstructured data and multiple raw description information. The multiple raw unstructured data and multiple raw description information are spliced ​​in advance, so that the multiple raw unstructured data and their original description information correspond one-to-one, saving the time of downloading, decompressing, and matching information from the cloud when the raw unstructured data is needed. By generating target storage paths for each of the multiple raw unstructured data, the corresponding raw unstructured data can be quickly found according to the target storage paths when the raw unstructured data is needed, thereby improving the efficiency of obtaining the raw unstructured data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0018] Figure 1 This is a flowchart of the steps of the unstructured data processing method according to an embodiment of the present invention.

[0019] Figure 2 For Figure 1 A structural block diagram of an unstructured data processing device corresponding to an embodiment.

[0020] Figure 3 FIG. 4 is a schematic structural diagram of an electronic device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to have a clearer understanding of the technical features, purposes and effects of the embodiments of the present invention, specific implementation methods of the embodiments of the present invention are now described with reference to the accompanying drawings.

[0022] In this document, “illustrative” means “serving as an example, instance or illustration”, and any diagram or implementation described in this document as “illustrative” should not be interpreted as a more preferred or more advantageous technical solution.

[0023] To simplify the drawings, only the parts relevant to the present invention are schematically shown in each figure, and they do not represent the actual structure of the product. In addition, to simplify the drawings and facilitate understanding, in some figures, only one or more components with the same structure or function are schematically depicted or labeled.

[0024] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.

[0025] To facilitate understanding, before describing the specific embodiments of the present invention in detail, the prior art of the unstructured data processing method, device, electronic device, and storage medium of the present invention is first exemplified.

[0026] In existing technologies, model calculation and training require unstructured data. To ensure accurate results, large amounts of raw unstructured data are typically used for calculation and training. However, this large amount of raw unstructured data collected by the vehicle consumes a large amount of memory, and vehicle-side storage resources are limited. Therefore, this large amount of raw unstructured data is typically compressed into a compressed package and uploaded to the cloud for storage.

[0027] When model calculation and training require raw unstructured data, it can be obtained by downloading and decompressing it from the cloud. However, since a compressed package generally contains multiple unstructured data and a descriptive data table for these unstructured data, after decompression, the obtained raw unstructured data needs to be matched with the original descriptive data in the original descriptive data table. When processing a large number of compressed packages, traditional data processing methods take a long time to match information. In addition, the downloading, decompression, information matching, and uploading of the decompressed data are all performed serially, which takes a long time to process and results in low efficiency in obtaining raw unstructured data.

[0028] The specific implementation of the embodiment of the present invention is further described below with reference to the accompanying drawings of the embodiment of the present invention.

[0029] Figure 1 The following is a flowchart describing a method for processing unstructured data according to an embodiment of the present invention, which mainly includes the following steps:

[0030] Step S110 , obtaining a plurality of original unstructured data and a plurality of original description data, wherein the plurality of original unstructured data and the plurality of original description data have a corresponding relationship.

[0031] It should be understood that unstructured data is irregularly or incompletely structured and lacks a predefined data model. Examples include images, various reports, audio, video, and radar point clouds. Descriptive data is used to describe the characteristics of unstructured data. Raw unstructured data is uncompressed unstructured data, and raw descriptive data is uncompressed descriptive data.

[0032] It should also be understood that the obtained multiple original unstructured data and the multiple original description data are in a one-to-one correspondence.

[0033] Step S120: constructing a relational data table based on the correspondence between the plurality of original unstructured data and the plurality of original description data.

[0034] Exemplarily, two relational data tables are constructed for the plurality of raw unstructured data and the plurality of raw description data, respectively. One relational data table stores the plurality of raw unstructured data, and the other relational data table stores the plurality of raw description data. In this case, the plurality of raw unstructured data and the plurality of raw description data are both stored in the form of binary byte streams. The plurality of raw unstructured data and the plurality of raw description data in the two relational data tables are in a one-to-one correspondence.

[0035] By constructing a relational data table, the efficiency of subsequent data splicing is improved, thereby improving the efficiency of obtaining unstructured data.

[0036] Step S130 , concatenating the plurality of original unstructured data and the plurality of original description data according to the relational data table to generate target storage paths for the plurality of original unstructured data.

[0037] Exemplarily, a join operation of the Spark distributed computing engine is used to join a plurality of original unstructured data and a plurality of original description data in one-to-one correspondence in two relational data tables to generate a target storage path.

[0038] It should be understood that the Spark distributed computing engine is a big data analysis engine. The solutions of the embodiments of the present invention involve massive amounts of raw unstructured data and also involve splicing operations based on relational data tables. To improve data processing efficiency, the solutions of the present invention utilize the Spark distributed computing engine. When processing massive amounts of unstructured data, multiple computer nodes can be set up so that tasks such as downloading compressed data packages, decompressing them, matching information, and then uploading them after decompression are performed simultaneously on multiple computer nodes.

[0039] It should also be understood that a join operation is a connection operation used to connect specific parts of multiple tables to generate new data.

[0040] As another example, through a join operation, original unstructured data with a data number and original description data for the original unstructured data are extracted from two relational tables according to the data number of the original unstructured data.

[0041] It should also be understood that the target storage path indicates the route to find multiple raw unstructured data. There can be multiple target storage paths, and each raw unstructured data has its own storage path. When a specific raw unstructured data is needed, it can be quickly found through the storage path corresponding to the raw unstructured data.

[0042] By generating target storage paths for multiple original unstructured data, the corresponding original unstructured data can be quickly found according to the target storage paths, thereby improving the efficiency of obtaining the original unstructured data.

[0043] Step S140: Save the target storage path to the data warehouse.

[0044] In summary, in the solution of the embodiment of the present invention, a method for processing unstructured data is provided. The method obtains multiple uncompressed original unstructured data and multiple original description data and constructs a relational data table of multiple original unstructured data and multiple original description information, and splices multiple original unstructured data and multiple original description information in advance, so that multiple original unstructured data and their original description information correspond one to one, saving the time of downloading, decompressing, and matching information from the cloud when the original unstructured data is needed. By generating the target storage path of each of the multiple original unstructured data, the corresponding original unstructured data can be quickly found according to the target storage path when the original unstructured data is needed, thereby improving the efficiency of obtaining the original unstructured data.

[0045] In one implementation, obtaining the plurality of raw unstructured data and the plurality of raw description data includes obtaining a data compression package of an unstructured data file, the data compression package including the plurality of compressed unstructured data and a compressed description data table. The data compression package is decompressed to obtain the plurality of raw unstructured data and the raw description data table, the raw description data table being used to store a plurality of data information of the plurality of raw unstructured data, wherein each piece of data information includes a data type, a byte length, a data number, and an acquisition device number. The raw description data table is traversed to determine the plurality of data information for the plurality of raw unstructured data as the plurality of raw description data.

[0046] It should be understood that compressed unstructured data and original unstructured data have a one-to-one correspondence. Compressing one original unstructured data yields one compressed unstructured data, and decompressing one compressed unstructured data yields one original unstructured data. Compressing multiple original unstructured data yields multiple compressed unstructured data, and decompressing multiple compressed unstructured data yields multiple original unstructured data.

[0047] Furthermore, there is a one-to-one correspondence between compressed description data tables and original description data tables. Compressing an original description data table yields a corresponding compressed description data table, and decompressing a compressed description data table yields a corresponding original description data table. Compressing multiple original description data tables yields multiple compressed description data tables, and decompressing multiple compressed description data tables yields multiple original description data tables.

[0048] It should also be understood that obtaining multiple raw unstructured data and multiple raw description data first requires obtaining a data compression package of the unstructured data file. This data compression package can be a single data compression package or multiple data compression packages. A single data compression package can include multiple compressed unstructured data and a compressed description data table, or it can include multiple compressed unstructured data and multiple compressed description data tables. The specific number of data compression packages and compressed description data tables is not limited here and can be determined based on actual application conditions.

[0049] The following describes how to obtain a data compression package of an unstructured data file, where the data compression package includes multiple compressed unstructured data and a compression description data table:

[0050] A data compression package of an unstructured data file is obtained, wherein the data compression package includes a plurality of compressed unstructured data and a compression description data table. The data compression package is decompressed to obtain a plurality of original unstructured data and an original description data table. The original description data table is used to store a plurality of data information of the plurality of original unstructured data, wherein each piece of data information includes a data type, a byte length, a data number, and an acquisition device number. The original description data table is traversed to determine a plurality of data information of the plurality of original unstructured data as a plurality of original description data.

[0051] In this embodiment, since a data compression packet includes multiple compressed unstructured data and a compressed description data table, and there is a one-to-one correspondence between the compressed unstructured data and the original unstructured data, and there is also a one-to-one correspondence between the compressed description data table and the original description data table, decompressing the data compression packet can obtain multiple original unstructured data and an original description data table.

[0052] Taking a data compression package of an unstructured data file as another example, a data compression package includes multiple compressed unstructured data and multiple compression description data tables, as described below:

[0053] A data compression package of an unstructured data file is obtained, wherein the data compression package includes multiple compressed unstructured data and multiple compressed description data tables. The data compression package is decompressed to obtain multiple original unstructured data and multiple original description data tables. The multiple original description data tables are used to store multiple data information of the multiple original unstructured data, wherein each data information includes a data type, a byte length, a data number, and an acquisition device number. The multiple original description data tables are traversed to determine the multiple data information of the multiple original unstructured data as multiple original description data.

[0054] In another implementation, obtaining the data compression package of the unstructured data file includes uploading the unstructured data file to the first cloud and saving the initial storage path to the data warehouse.

[0055] It should be understood that the first cloud mentioned in the solution of the embodiment of the present invention can be the Internet of Vehicles cloud service end. In actual situations, the vehicle end will collect massive amounts of original unstructured data through sensors. In order to reduce the pressure on local memory, the vehicle end will usually package the massive amounts of original unstructured data into unstructured data files, and then compress the unstructured data files to obtain data compression packages, and then upload the data compression packages of the unstructured data files to the Internet of Vehicles cloud service end, and then the Internet of Vehicles cloud service end will store and manage the data compression packages of these unstructured data files.

[0056] It should also be understood that in the solution of the embodiment of the present invention, the data warehouse is used to store the initial storage path and the target storage path. The target storage path and the initial storage path have their own data tables, and the data table of the target storage path will not overwrite the data table of the initial storage path.

[0057] In another implementation, obtaining the data compression package of the unstructured data file includes reading an initial storage path of the unstructured data file from a data warehouse, and downloading the data compression package of the unstructured data file from the first cloud according to the initial storage path.

[0058] It should be understood that when reading the initial storage path of the unstructured data file from the data warehouse, the spark connector is used to read the data table of the initial storage path.

[0059] It should also be understood that saving the initial storage path to the data warehouse means writing the data table of the initial storage path to the data warehouse.

[0060] Furthermore, before downloading the data compression package, the data compression package of the unstructured data file is converted into a binary byte stream to increase the download speed. In addition, downloading the data compression package and decompressing the data compression package can be performed simultaneously.

[0061] The data compression package of the unstructured data file can be quickly found from the first cloud through the initial storage path stored in the data warehouse, thereby improving the search efficiency of the data compression package.

[0062] In another implementation, the unstructured data processing method further includes uploading the plurality of original unstructured data to a second cloud.

[0063] It should be understood that the second cloud is the cloud where the original unstructured data is stored when performing model calculation and training. When the original unstructured data is needed, the original unstructured data can be obtained from the second cloud according to the target storage path of the data warehouse.

[0064] By storing the original unstructured data and the target storage path separately, the security of data storage is improved.

[0065] In another implementation, uploading the plurality of original unstructured data to the second cloud includes converting the plurality of original unstructured data into a binary byte stream, and uploading the binary byte stream to the second cloud.

[0066] In another implementation, converting the plurality of raw unstructured data into a binary byte stream includes determining whether an input account password is consistent with a preset account password. If the input account password is consistent with the preset account password, executing a conversion instruction for the plurality of raw unstructured data to convert the plurality of raw unstructured data into a binary byte stream.

[0067] By determining whether to execute the conversion instruction based on the account and password, the security of data conversion is improved. By converting multiple raw unstructured data into binary byte streams and uploading the binary byte streams to the second cloud, the speed of data upload is improved.

[0068] According to another embodiment of the present invention, there is provided an unstructured data processing apparatus. Figure 2 , the unstructured data processing device comprises:

[0069] The acquisition module 210 is configured to acquire a plurality of original unstructured data and a plurality of original description data, where the plurality of original unstructured data and the plurality of original description data have a corresponding relationship.

[0070] The construction module 220 is used to construct a relational data table based on the corresponding relationship between the multiple original unstructured data and the multiple original description data.

[0071] The generating module 230 is used to splice the multiple original unstructured data and the multiple original description data according to the relational data table to generate target storage paths for the multiple original unstructured data.

[0072] The transmission module 240 is used to save the target storage path to the data warehouse.

[0073] The acquisition module 210 is specifically configured to obtain a data compression package of an unstructured data file, the data compression package including multiple compressed unstructured data and a compressed description data table. The data compression package is decompressed to obtain multiple original unstructured data and an original description data table. The original description data table is configured to store multiple data information items related to the multiple original unstructured data items, each of which includes a data type, byte length, data number, and acquisition device number. The original description data table is traversed to determine the multiple data information items related to the multiple original unstructured data items as multiple original description data items.

[0074] The acquisition module 210 is further configured to read an initial storage path of the unstructured data file from the data warehouse and download a data compression package of the unstructured data file from the first cloud according to the initial storage path.

[0075] The transmission module 240 is further configured to upload the unstructured data file to the first cloud and save the initial storage path to the data warehouse.

[0076] The transmission module 240 is further configured to upload the plurality of original unstructured data to the second cloud.

[0077] In another implementation, the unstructured data processing apparatus of this embodiment further includes:

[0078] The conversion module is used to convert multiple original unstructured data into binary byte streams.

[0079] The conversion module is specifically used to determine whether the input account password is consistent with the preset account password. If the input account password is consistent with the preset account password, the conversion instruction for the multiple original unstructured data is executed to convert the multiple original unstructured data into a binary byte stream.

[0080] In an embodiment of the present invention, an unstructured data processing device is provided. This device obtains multiple uncompressed raw unstructured data and multiple raw description data and constructs a relational data table of the multiple raw unstructured data and multiple raw description information. This device pre-joins the multiple raw unstructured data and multiple raw description information, ensuring a one-to-one correspondence between the multiple raw unstructured data and their original description information. This saves the time required to download, decompress, and match the raw unstructured data from the cloud when the raw unstructured data is needed. By generating target storage paths for each of the multiple raw unstructured data, the corresponding raw unstructured data can be quickly found based on the target storage paths when the raw unstructured data is needed, thereby improving the efficiency of obtaining unstructured data.

[0081] The device of this embodiment is used to implement the corresponding methods in the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments. In addition, the functional implementation of each module in the device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments.

[0082] According to another embodiment of the present invention, an electronic device 300 is provided. Figure 3 , a block diagram of an electronic device 300 that can serve as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, user digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as user digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0083] The electronic device 300 may include a processor 302 , a communications interface 304 , a memory 306 , and a communication bus 308 .

[0084] The processor 302, the communication interface 304, and the memory 306 communicate with each other via the communication bus 308. The communication interface 304 is used to communicate with other electronic devices or servers.

[0085] The processor 302 is configured to execute the program 310 , and specifically may execute the relevant steps in the above method embodiment.

[0086] Specifically, the program 310 may include program codes, which include computer operation instructions.

[0087] Processor 302 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0088] The memory 306 is used to store the program 310. The memory 306 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0089] The program 310 may specifically be configured to cause the processor 302 to perform the following operations: obtain a plurality of raw unstructured data and a plurality of raw description data, wherein the plurality of raw unstructured data and the plurality of raw description data have a corresponding relationship; construct a relational data table based on the corresponding relationship between the plurality of raw unstructured data and the plurality of raw description data; concatenate the plurality of raw unstructured data and the plurality of raw description data according to the relational data table, and generate a target storage path for each of the plurality of raw unstructured data; and save the target storage path to a data warehouse.

[0090] In addition, the specific implementation of each step in program 310 can refer to the corresponding description of the corresponding steps and units in the above-mentioned method embodiment, and will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working process of the above-mentioned devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, and will not be repeated here.

[0091] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present invention can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present invention.

[0092] Exemplary embodiments of the present invention further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the methods of various embodiments of the present application.

[0093] The method according to the embodiment of the present invention described above can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a processor or programmable or hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0094] Thus far, specific embodiments of the present invention have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

[0095] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0096] In the description of the present invention, the terms "first" and "second" are used solely to facilitate description of different components or names and should not be construed as indicating or implying a sequential relationship, relative importance, or implicitly specifying the quantity of the technical features being described. Therefore, features specified as "first" or "second" may explicitly or implicitly include at least one of such features.

[0097] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0098] It should be understood that although this specification is described according to various embodiments, not every embodiment contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0099] The examples of the embodiments of the present invention are intended to briefly illustrate the technical features of the embodiments of the present invention so that those skilled in the art can intuitively understand the technical features of the embodiments of the present invention, and are not intended to improperly limit the embodiments of the present invention.

[0100] Finally, it should be noted that the above implementation methods are only used to illustrate the embodiments of the present invention, and are not limitations on the embodiments of the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the scope of patent protection of the embodiments of the present invention should be defined by the claims.

Claims

1. A method for processing unstructured data, characterized in that: include: Acquire a plurality of original unstructured data and a plurality of original description data, wherein the plurality of original unstructured data and the plurality of original description data have a corresponding relationship; Constructing a relational data table based on the correspondence between the plurality of original unstructured data and the plurality of original description data; this step specifically comprises: constructing two relational data tables for the plurality of original unstructured data and the plurality of original description data, respectively, wherein one relational data table stores the plurality of original unstructured data, and the other relational data table stores the plurality of original description data; The plurality of original unstructured data and the plurality of original description data are spliced ​​together according to the relational data table to generate target storage paths for the plurality of original unstructured data. This step specifically comprises: splicing the plurality of original unstructured data and the plurality of original description data in one-to-one correspondence in the two relational data tables through a join operation of the Spark distributed computing engine to generate target storage paths. The target storage path is saved in the data warehouse.

2. The method according to claim 1, characterized in that The acquiring of the plurality of original unstructured data and the plurality of original description data includes: Acquire a data compression package of an unstructured data file, wherein the data compression package includes a plurality of compressed unstructured data and a compression description data table; Decompressing the data compression package to obtain the plurality of original unstructured data and an original description data table, wherein the original description data table is used to store a plurality of data information of the plurality of original unstructured data, wherein each of the plurality of data information includes a data type, a byte length, a data number, and an acquisition device number; The original description data table is traversed, and a plurality of data information for the plurality of original unstructured data are determined as the plurality of original description data.

3. The method according to claim 2, characterized in that The step of obtaining the data compression package of the unstructured data file includes: Reading an initial storage path of the unstructured data file from the data warehouse; The data compression package of the unstructured data file is downloaded from the first cloud according to the initial storage path.

4. The method according to claim 3, characterized in that Before obtaining the data compression package of the unstructured data file, the method includes: Uploading the unstructured data file to the first cloud; Upload the initial storage path to the data warehouse.

5. The method according to claim 1, wherein The method further includes uploading the plurality of original unstructured data to a second cloud.

6. The method according to claim 5, characterized in that The uploading of the plurality of original unstructured data to the second cloud comprises: Converting the plurality of original unstructured data into a binary byte stream; Upload the binary byte stream to the second cloud.

7. The method according to claim 6, characterized in that The converting the plurality of original unstructured data into a binary byte stream comprises: Determine whether the entered account and password are consistent with the preset account and password; If the input account password is consistent with the preset account password, the conversion instruction for the multiple original unstructured data is executed to convert the multiple original unstructured data into the binary byte stream.

8. An unstructured data processing device, characterized in that: include: An acquisition module, configured to acquire a plurality of original unstructured data and a plurality of original description data, wherein the plurality of original unstructured data and the plurality of original description data have a corresponding relationship; A construction module, configured to construct a relational data table based on the corresponding relationship between the plurality of original unstructured data and the plurality of original description data; Specifically used for: respectively constructing two relational data tables for the plurality of original unstructured data and the plurality of original description data, wherein one relational data table stores the plurality of original unstructured data, and the other relational data table stores the plurality of original description data; a generating module, configured to concatenate the plurality of original unstructured data and the plurality of original description data according to the relational data table, and generate a target storage path for each of the plurality of original unstructured data; Specifically used for: splicing the multiple original unstructured data and the multiple original description data corresponding to each other in the two relational data tables through the join operation of the Spark distributed computing engine to generate a target storage path; The transmission module is used to save the target storage path to the data warehouse.

9. An electronic device, characterized in that: include: processor; Memory for storing programs; The program includes instructions, which, when executed by the processor, cause the processor to perform the steps of the method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that A computer program is stored thereon, which implements the steps of the method according to any one of claims 1 to 7 when the program is executed by a processor.

Images