Heterogeneous data query optimization method and device, electronic equipment and storage medium
By constructing data category matrix and training 3D convolutional neural network, the problem of inefficient query efficiency in heterogeneous data is solved, and efficient classification and query of heterogeneous data is achieved.
Patent Information
- Application Number
- CN202410811467.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-21
- Publication Date
- 2025-02-18
AI Technical Summary
In the prior art, heterogeneous data are not distinguished in the same data pool, resulting in inefficient efficiency when querying a certain type of data.
By obtaining key identification information of heterogeneous data, determining its attribute characteristics, building multiple data category matrices, establishing activation functions based on these matrices, training a 3D convolutional neural network, and finally inputting heterogeneous data into the neural network, outputting classification result.
Accurate classification marking of a large number of heterogeneous data is realized, and users can accurately locate all data of a certain type, thereby improving the efficiency of data query.
Smart Images
Figure CN119474135A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of database management technology, and in particular to a heterogeneous data query optimization method, device, electronic device and storage medium. Background Art
[0002] With the rapid development of information technology, the amount of data has exploded, and the diversity of data has become increasingly significant. Data is no longer limited to structured data. Unstructured, semi-structured data, and data in various formats are emerging in an endless stream. These data of different types, sources, and formats constitute the so-called heterogeneous data. Heterogeneous data sets usually contain multiple data types, such as relational data, document data, images, videos, sensor data, etc. These data require different processing technologies and storage structures. Different data sources may use different data models and storage formats, making data integration a major challenge for heterogeneous data query optimization.
[0003] In the prior art, in order to improve the efficiency and accuracy of heterogeneous data queries, a common method is to use data integration and unified query interface technology to integrate heterogeneous data from different data sources into a unified query platform. This platform can provide a unified query language and interface, allowing users to query data of different types and sources in the same way. However, when faced with a large amount of data, users cannot accurately locate and query a specific data type, such as being unable to quickly filter out video data or image data from a large amount of data, and query the desired target data from the filtered video data or image data. Therefore, putting all different types of heterogeneous data into the same data pool without distinguishing them will seriously affect the efficiency of subsequent data queries.
[0004] Therefore, a new data query optimization method is needed to overcome the problem of inefficiency in querying a certain type of data from a large amount of heterogeneous data. Summary of the invention
[0005] In view of this, it is necessary to provide a heterogeneous data query optimization method, device, electronic device and storage medium to solve the technical problem in the prior art that different types of heterogeneous data are not distinguished in the same data pool, resulting in low efficiency when querying a certain type of data.
[0006] In order to solve the above problems, the present invention provides a heterogeneous data query optimization method, comprising: Acquire key identification information of the heterogeneous data, and determine attribute characteristics of the heterogeneous data according to the key identification information; According to the attribute characteristics, a plurality of data category matrices are constructed, wherein each element position in the data category matrix represents a key identification information, and each element value represents an attribute characteristic; Establishing an activation function according to the category matrix, and establishing and training a 3D convolutional neural network according to the activation function; The heterogeneous data is input into the 3D convolutional neural network, and the classification result of the heterogeneous data is output.
[0007] In one possible implementation, the key identification information includes data type, content description and keywords; the attribute features of relational data include numerical type, text type and date type; the attribute features of document data include word frequency, word vector and syntactic structure; the attribute features of image data include pixel value, color histogram and texture features; the attribute features of video data include inter-frame relationship, motion information and visual features; the attribute features of sensor data include sensor type, sampling frequency and sensor reading.
[0008] In a possible implementation, constructing multiple data category matrices according to the attribute characteristics includes: With key identification information as parent nodes and attribute features as child nodes, an information tree representing heterogeneous data is constructed; The number of rows of the vector is determined according to the number of key identification information, and the number of columns of the vector is determined according to the number of attribute features, and a plurality of data category matrices are determined based on the information tree.
[0009] In a possible implementation, the 3D convolutional neural network includes a preset input layer, an activation layer, a preset hidden layer and a preset fully connected layer connected in sequence, and the preset hidden layer includes multiple preset convolutional layers and multiple preset pooling layers; the input of the activation layer is the eigenvalue output by the preset input layer, and the output of the activation layer is the target extracted eigenvalue, and the target extracted eigenvalue is used to characterize the classification result of heterogeneous data, and each pixel value in the target extracted eigenvalue is obtained according to the activation function.
[0010] In a possible implementation, establishing an activation function according to the category matrix, and establishing and training a 3D convolutional neural network according to the activation function, includes: According to the similarity of multiple category matrices, the similarity between the category matrices is determined; According to the similarity between multiple category matrices, establishing the activation function; According to the activation function, establishing the activation layer; Obtain the preset input layer, the preset hidden layer, and the preset fully connected layer that have been trained; The 3D convolutional neural network is established according to the activation layer, the preset input layer, the preset hidden layer and the preset fully connected layer.
[0011] In a possible implementation, the activation function is:
[0012] Among them, x and c represent two different category matrices, Represents different, Ed(x,c) represents the similarity between two different category matrices, Md(x,c) represents the Manhattan distance between two different type vectors, Cd(x,c) represents the cosine distance between two different type vectors, ω1, ω2, and ω3 are different weight coefficients respectively.
[0013] In a possible implementation, the step of inputting heterogeneous data into a 3D convolutional neural network and outputting a classification result of the heterogeneous data includes: The heterogeneous data is input into the 3D convolutional neural network, and the classification results of the heterogeneous data are output based on the key recognition information.
[0014] In a second aspect, the present invention further provides a heterogeneous data query optimization device, comprising: An acquisition module, used to acquire key identification information of heterogeneous data and determine attribute characteristics of the heterogeneous data according to the key identification information; A vector construction module, used to construct a plurality of data category matrices according to the attribute characteristics, wherein each element position in the data category matrix represents a key identification information, and each element value represents an attribute characteristic; A network training module, used to establish an activation function according to the category matrix, and to establish and train a 3D convolutional neural network according to the activation function; The classification module is used to input heterogeneous data into the 3D convolutional neural network and output the classification results of the heterogeneous data.
[0015] In a third aspect, the present invention further provides an electronic device, comprising: a processor and a memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in the heterogeneous data query optimization method described above are implemented.
[0016] In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the heterogeneous data query optimization method as described above.
[0017] The beneficial effects of the present invention are as follows: firstly, the key identification information of heterogeneous data is obtained, and the attribute characteristics of the heterogeneous data are determined according to the key identification information; then, according to the attribute characteristics, multiple data category matrices are constructed, each element position in the data category matrix represents a key identification information, and each element value represents an attribute characteristic; then, an activation function is established according to the category matrix, and a 3D convolutional neural network is established and trained according to the activation function; the 3D convolutional neural network is trained by the attribute characteristics and key identification information so that the neural network model can achieve accurate classification, and finally, the heterogeneous data is input into the 3D convolutional neural network, and the classification result of the heterogeneous data is obtained by output. Thus, by classifying and marking a large amount of heterogeneous data, the present invention enables users to accurately locate all data of a certain type and query target data from the corresponding category data, thereby improving the efficiency of data query. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] FIG1 is a method flow chart of an embodiment of a heterogeneous data query optimization method provided by the present invention; Figure 2 A method flow chart of an embodiment of step S102 in the heterogeneous data query optimization method provided by the present invention; Figure 3 A method flow chart of an embodiment of step S103 in the heterogeneous data query optimization method provided by the present invention; Figure 4 It is a schematic diagram of an embodiment of a heterogeneous data query optimization device provided by the present invention; Figure 5 It is a schematic diagram of the operating environment of an electronic device provided by the present invention. DETAILED DESCRIPTION
[0019] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.
[0020] A specific embodiment of the present invention discloses a heterogeneous data query optimization method, please refer to Figure 1 ,include: S101, obtaining key identification information of heterogeneous data, and determining attribute characteristics of the heterogeneous data according to the key identification information; S102, constructing a plurality of data category matrices according to the attribute characteristics, wherein each element position in the data category matrix represents a key identification information, and each element value represents an attribute characteristic; S103, establishing an activation function according to the category matrix, and establishing and training a 3D convolutional neural network according to the activation function; S104: Input the heterogeneous data into the 3D convolutional neural network, and output the classification result of the heterogeneous data.
[0021] In this embodiment, the key identification information of the heterogeneous data is first obtained, and the attribute characteristics of the heterogeneous data are determined according to the key identification information; then, according to the attribute characteristics, multiple data category matrices are constructed, each element position in the data category matrix represents a key identification information, and each element value represents an attribute characteristic; then, an activation function is established according to the category matrix, and a 3D convolutional neural network is established and trained according to the activation function; the 3D convolutional neural network is trained by the attribute characteristics and key identification information so that the neural network model can achieve accurate classification, and finally, the heterogeneous data is input into the 3D convolutional neural network, and the classification result of the heterogeneous data is obtained by output. Therefore, by classifying and marking a large amount of heterogeneous data, the present invention allows users to accurately locate all data of a certain type and query target data from the corresponding category data, thereby improving the efficiency of data query.
[0022] It should be noted that the key identification information indicates the data type or requirement category. For example, the key identification information may be the data type, content description, and keyword. The key identification information may also be set according to user requirements, such as marking data with text as one category and marking data without text as another category. In this case, the key identification information is text and non-text.
[0023] Furthermore, attribute features refer to specific attributes or features that describe data or objects. Different key identification information has different attribute features. For example, the attribute features of relational data include numerical, textual, and date types; the attribute features of document data include word frequency, word vector, and syntactic structure; the attribute features of image data include pixel value, color histogram, and texture features; the attribute features of video data include inter-frame relationships, motion information, and visual features; the attribute features of sensor data include sensor type, sampling frequency, and sensor readings. By defining the attribute features of key identification information, data can be classified quickly and accurately.
[0024] In a preferred embodiment, the step S102 constructs multiple data category matrices according to the attribute characteristics, see Figure 2 ,include: S201, constructing an information tree representing heterogeneous data with key identification information as a parent node and attribute features as child nodes; S202: Determine the number of rows of the vector according to the number of key identification information, and determine the number of columns of the vector according to the number of attribute features, and determine multiple data category matrices based on the information tree.
[0025] In this embodiment, when classifying data, heterogeneous data has multiple key identification information, and each key identification information has multiple attribute features. In order to better train the 3D convolutional neural network to enable it to have more accurate classification capabilities, it is necessary to list as many training data sets and labels as possible to ensure the comprehensiveness of the training data. Therefore, by taking the key identification information as the parent node and the attribute features as the child nodes, an information tree is constructed to display all possible categories, and the data of the key identification information is the number of rows of the vector, and the data of the attribute features is the number of columns of the vector to determine the category matrix.
[0026] It should be noted that the 3D convolutional neural network is trained with the category matrix as input, so that the 3D convolutional neural network has better learning ability. Specifically, the system implementing this method can analyze the key identification information and attribute features with typical behavioral characteristics from big data, and establish a category matrix based on these key identification information and attribute feature data, store it in the server and continuously optimize it. The details of how to obtain data and establish the category matrix are prior art and will not be explained in detail in this article.
[0027] In some embodiments, the 3D convolutional neural network includes a preset input layer, an activation layer, a preset hidden layer and a preset fully connected layer connected in sequence, and the preset hidden layer includes multiple preset convolutional layers and multiple preset pooling layers; the input of the activation layer is the eigenvalue output by the preset input layer, and the output of the activation layer is the target extracted eigenvalue, and the target extracted eigenvalue is used to characterize the classification result of heterogeneous data, and each pixel value in the target extracted eigenvalue is obtained according to the activation function.
[0028] In this embodiment, a 3D convolutional neural network is a deep learning model, which is a neural network structure specially designed for processing three-dimensional data. Different from the traditional two-dimensional convolutional neural network, the 3D convolutional neural network can process three-dimensional data with time and space information such as video, time series or volume data by considering the additional time dimension. Therefore, the 3D convolutional neural network is more suitable for processing heterogeneous data.
[0029] The image data in this embodiment includes three dimensions: width, height, and time of the image, and is therefore particularly suitable for processing by a 3D convolutional neural network. It is understandable that the specific structures and specific principles of the above-mentioned preset input layer, preset hidden layer, and preset fully connected layer in this embodiment are prior arts that can be understood by those skilled in the art, and therefore will not be described in detail herein.
[0030] In some embodiments, the activation function is established according to the category matrix, and a 3D convolutional neural network is established and trained according to the activation function. Figure 3 ,include: S301, determining the similarity between the category matrices according to the similarity of multiple category matrices; S302, establishing the activation function according to the similarity between multiple category matrices; S303, establishing the activation layer according to the activation function; S304, obtaining the trained preset input layer, the preset hidden layer, and the preset fully connected layer; S305. Establish the 3D convolutional neural network according to the activation layer, the preset input layer, the preset hidden layer and the preset fully connected layer.
[0031] In this embodiment, by establishing the similarity between multiple category matrices, the multiple category matrices can be further combined to establish an activation layer, so that the activation layer can further identify the data referred to by different expressions of the same key identification information, further improving the subsequent processing accuracy and processing speed.
[0032] In addition, the preset input layer, preset hidden layer and preset fully connected layer in this embodiment are not trained in the system in this embodiment, but are pre-trained under other conditions and directly installed in this system. The main improvement of this embodiment over the existing 3D convolutional neural network is the addition of an activation layer, which is set locally in the system and then combined with the preset input layer, preset hidden layer and preset fully connected layer to form a complete 3D convolutional neural network, which minimizes the data analysis time to improve the real-time data processing capability, so that the help-seeking action can be recognized at the fastest speed.
[0033] Furthermore, to calculate the similarity between multiple category vectors, feature similarity calculation methods (such as cosine similarity, Euclidean distance, etc.) can be used to measure the similarity or correlation score between each multiple category vector. Since the same data can be represented by multiple category matrices based on different key identification information or different feature attributes, by establishing the similarity between multiple category matrices and establishing activation layers through the similarity, different representations of the same data can be better identified and learned to improve the accuracy of data classification.
[0034] Specifically, the activation function is:
[0035] Among them, x and c represent two different category matrices, Represents different, Ed(x,c) represents the similarity between two different category matrices, Md(x,c) represents the Manhattan distance between two different type vectors, Cd(x,c) represents the cosine distance between two different type vectors, ω1, ω2, and ω3 are different weight coefficients respectively.
[0036] It should be noted that different methods of calculating distance can bring different beneficial effects, as follows: Euclidean distance can effectively measure the absolute distance between vectors and is suitable for the analysis of intensive data. It can highlight the degree of difference between data points and cluster similar data points together, which is conducive to distinguishing different clusters. Obviously, the data represented by different attribute characteristics of the same key identification information in this embodiment belongs to intensive data. Manhattan distance is suitable for processing data containing discrete attributes. For discrete attributes, it is obvious that the data represented by different key identification information in this embodiment has discrete attributes. Cosine distance can ignore the absolute size of the vector and only focus on the directional differences between vectors. This makes cosine distance very useful when processing high-dimensional data sets and can avoid the problem of dimensionality disaster. In the field of text processing, cosine distance can effectively calculate the similarity between texts and help cluster similar texts together.
[0037] This embodiment combines multiple distance calculation methods to form a similar distance as a distance judgment benchmark in the similarity process of the present invention, so that the similarity has the advantages of the above three distances. In practice, the importance of different distance calculation methods can be weighed by three different weight coefficients ω1, ω2, and ω3.
[0038] Based on the above heterogeneous data query optimization method, the embodiment of the present invention also provides a heterogeneous data query optimization device, please refer to Figure 4 , including: an acquisition module 410, a vector construction module 420, a network training module 430 and a classification module 440.
[0039] An acquisition module 410 is used to acquire key identification information of the heterogeneous data and determine attribute characteristics of the heterogeneous data according to the key identification information; A vector construction module 420 is used to construct a plurality of data category matrices according to the attribute characteristics, wherein each element position in the data category matrix represents a key identification information, and each element value represents an attribute characteristic; A network training module 430, used to establish an activation function according to the category matrix, and to establish and train a 3D convolutional neural network according to the activation function; The classification module 440 is used to input the heterogeneous data into the 3D convolutional neural network and output the classification results of the heterogeneous data.
[0040] like Figure 5 As shown, based on the above heterogeneous data query optimization method, the present invention also provides an electronic device, which can be a computing electronic device such as a mobile terminal, a desktop computer, a notebook, a palm computer, and a server. The electronic device includes a processor 510, a memory 520, and a display 530. Figure 5 Only some components of the electronic device are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.
[0041] In some embodiments, the memory 520 may be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device. In other embodiments, the memory 520 may also be an external storage electronic device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 520 may also include both an internal storage unit of the electronic device and an external storage electronic device. The memory 520 is used to store application software and various types of data installed in the electronic device, such as program codes installed in the electronic device. The memory 520 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a heterogeneous data query optimization program 540 is stored on the memory 520, and the heterogeneous data query optimization program 540 can be executed by the processor 510, thereby realizing the heterogeneous data query optimization method of each embodiment of the present application.
[0042] In some embodiments, the processor 510 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes stored in the memory 520 or process data, such as executing heterogeneous data query optimization methods.
[0043] In some embodiments, the display 530 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 530 is used to display information on the heterogeneous data query optimization electronic device and to display a visual user interface. The components 510-530 of the electronic device communicate with each other via a system bus.
[0044] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.
[0045] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.
Claims
1. A heterogeneous data query optimization method, characterized in that: include: Acquire key identification information of the heterogeneous data, and determine attribute characteristics of the heterogeneous data according to the key identification information; According to the attribute characteristics, a plurality of data category matrices are constructed, wherein each element position in the data category matrix represents a key identification information, and each element value represents an attribute characteristic; Establishing an activation function according to the category matrix, and establishing and training a 3D convolutional neural network according to the activation function; The heterogeneous data is input into the 3D convolutional neural network, and the classification result of the heterogeneous data is output.
2. The heterogeneous data query optimization method according to claim 1, characterized in that: The key identification information includes data type, content description and keywords; the attribute features of relational data include numerical type, text type and date type; the attribute features of document data include word frequency, word vector and syntactic structure; the attribute features of image data include pixel value, color histogram and texture features; the attribute features of video data include inter-frame relationship, motion information and visual features; The attribute characteristics of sensor data include sensor type, sampling frequency, and sensor readings.
3. The heterogeneous data query optimization method according to claim 2, characterized in that: The step of constructing a plurality of data category matrices according to the attribute characteristics includes: With key identification information as parent nodes and attribute features as child nodes, an information tree representing heterogeneous data is constructed; The number of rows of the vector is determined according to the number of key identification information, and the number of columns of the vector is determined according to the number of attribute features, and a plurality of data category matrices are determined based on the information tree.
4. The heterogeneous data query optimization method according to claim 1, characterized in that: The 3D convolutional neural network includes a preset input layer, an activation layer, a preset hidden layer and a preset fully connected layer connected in sequence, and the preset hidden layer includes a plurality of preset convolutional layers and a plurality of preset pooling layers; The input of the activation layer is the feature value output by the preset input layer, and the output of the activation layer is the target extraction feature value, which is used to characterize the classification result of heterogeneous data, and each pixel value in the target extraction feature value is obtained according to the activation function.
5. The heterogeneous data query optimization method according to claim 4 is characterized in that: The step of establishing an activation function according to the category matrix, and establishing and training a 3D convolutional neural network according to the activation function comprises: According to the similarity of multiple category matrices, the similarity between the category matrices is determined; Establishing the activation function according to the similarity between multiple category matrices; According to the activation function, establishing the activation layer; Obtain the preset input layer, the preset hidden layer and the preset fully connected layer that have been trained; The 3D convolutional neural network is established according to the activation layer, the preset input layer, the preset hidden layer and the preset fully connected layer.
6. The heterogeneous data query optimization method according to claim 5, characterized in that: The activation function is: Among them, x and c represent two different category matrices, Represents different, Ed(x,c) represents the similarity between two different category matrices, Md(x,c) represents the Manhattan distance between two different type vectors, Cd(x,c) represents the cosine distance between two different type vectors, ω1, ω2, and ω3 are different weight coefficients respectively.
7. The heterogeneous data query optimization method according to claim 1, characterized in that: The step of inputting heterogeneous data into a 3D convolutional neural network and outputting classification results of the heterogeneous data includes: The heterogeneous data is input into the 3D convolutional neural network, and the classification results of the heterogeneous data are output based on the key recognition information.
8. A heterogeneous data query optimization device, characterized in that: include: An acquisition module, used to acquire key identification information of heterogeneous data and determine attribute characteristics of the heterogeneous data according to the key identification information; A vector construction module, used to construct a plurality of data category matrices according to the attribute characteristics, wherein each element position in the data category matrix represents a key identification information, and each element value represents an attribute characteristic; A network training module, used to establish an activation function according to the category matrix, and to establish and train a 3D convolutional neural network according to the activation function; The classification module is used to input heterogeneous data into the 3D convolutional neural network and output the classification results of the heterogeneous data.
9. An electronic device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in the heterogeneous data query optimization method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the heterogeneous data query optimization method as described in any one of claims 1-7.