Public data platform retrieval method and system based on multiple data sources

Abstract

The invention discloses a public data platform retrieval method and system based on multiple data sources, and relates to the technical field of data processing. Obtaining an original data set, and converting audio data in the original data set to obtain audio text data; performing feature detection on the image data to obtain a feature identification tag set, and fusing the images according to feature identification tags to obtain an initial data set; performing semantic recognition on the initial data to obtain semantic text data, combining the semantic text data, the audio text data and the text data to obtain target text data, and performing classification to obtain a classification data set; and uploading the classification data set to a public data platform, so that the public data platform performs updating. According to the method, feature extraction and semantic analysis are carried out on the data to generate text description and classify the text description, the retrieval efficiency of the data is improved, more accurate classification labels are provided for a retrieval system, the retrieval precision is improved, the classified data set is uploaded to a platform to be updated, and a data storage structure is optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a public data platform retrieval method and system based on multiple data sources. BACKGROUND

[0002] With the rapid development of information technology, multi-data source public data platforms have emerged and rapidly developed. It has gathered massive data from different fields and sources, providing rich resources for scientific research, data analysis and decision support. However, the heterogeneity and diversity of data have brought many challenges in storage, management and retrieval.

[0003] Patent No. CN118606334A discloses a data retrieval library updating method, a data retrieval method and related devices; new full data is obtained at the start time of each data update period, and when the new full data changes, the full data stored in the preset full data retrieval library is replaced with the new full data. Incremental data between the construction start time of the new full data and the end time of the data update period is obtained, and the preset incremental data retrieval library is updated according to the incremental data. The data sets provided by the full data retrieval library and the incremental data retrieval library are used for data retrieval. Since the update of the full data is performed in each data update period, the incremental data between the construction start time of the new full data and the end time of the data update period is obtained, and the update of the incremental data is performed, the real-time update of the data can be realized, the accuracy of the data in the data set is ensured, and the accuracy of the retrieval is improved.

[0004] In the prior art, since the public data platform contains data from multiple data sources, the sources are heterogeneous and the modalities are diverse, the platform is difficult to implement data storage classification, resulting in low retrieval efficiency and poor accuracy. SUMMARY

[0005] The present application aims to solve the problem of low retrieval efficiency and poor accuracy caused by the difficulty of transparent data storage in the public data platform containing data from multiple data sources, and proposes a public data platform retrieval method and system based on multiple data sources.

[0006] The first aspect of the present application provides a public data platform retrieval method based on multiple data sources, which comprises:

[0007] Obtaining an original data set, and converting audio data in the original data set to obtain audio text data; the original data set includes text data, audio data and image data;

[0008] Feature recognition labels are detected from image data in the original data set, and images in the image data are fused according to the feature recognition labels in the feature recognition label set to obtain an initial data set;

[0009] Semantic text data is recognized from each initial data in the initial data set, the semantic text data, audio text data and text data are combined to obtain target text data, and classification data sets are obtained by classifying the target text data;

[0010] The classification data sets are uploaded to a public data platform for updating.

[0011] Optionally, feature recognition labels are detected from each initial data in the initial data set, including:

[0012] Feature recognition labels are detected from the original data in the original data set by a target model, and the feature recognition labels are combined to obtain a feature recognition label set;

[0013] The target model is improved based on a YOLOv8 model, including:

[0014] The backbone network is replaced by a target backbone network, the C2f module in the neck network is replaced by a target improvement module, and the Upsample in the neck network is replaced by CARAFE to obtain a target model;

[0015] The working process of the target backbone network includes:

[0016] An original feature tensor is obtained, the original feature tensor is input into an embedding module group and three merging module groups in sequence to obtain an initial feature tensor, the initial feature tensor is input into an SPPF module to obtain a target feature tensor, and the target feature tensor is taken as the output of the target backbone network; the embedding module group is composed of an embedding layer and a FasterNet module; the merging module group is composed of a merging layer and a FasterNet module.

[0017] Optionally, the working principle of the target improvement module includes:

[0018] An original feature map is obtained, local average pooling is performed on the original feature map to obtain a first feature map, reshaping, convolution and reshaping are sequentially performed on the first feature map to obtain a second feature map;

[0019] Global average pooling, reshaping, convolution and uniform average pooling are sequentially performed on the first feature map to obtain a third feature map, the second feature map and the third feature map are fused to obtain a fourth feature map;

[0020] performing a uniform average pooling operation on the fourth feature map to obtain a target fourth feature map, performing element-by-element multiplication between the target fourth feature map and the original feature map to obtain a fifth feature map, and taking the fifth feature map as an output of the target improvement module.

[0021] Optionally, performing semantic recognition on each initial data in the initial data set to obtain semantic text data, including:

[0022] inputting target initial data in the initial data set into a first preset model to obtain a bounding box set and a relation tuple set; the target initial data is any one of the initial data set;

[0023] performing subject extraction on the relation tuple set to obtain a semantic node, performing predicate extraction on the relation tuple set to obtain a semantic edge, and constructing a semantic graph according to the semantic node and the semantic edge;

[0024] performing geometric feature extraction on a target bounding box in the bounding box set to obtain a spatial node, determining a spatial edge between node features according to the relation tuple set, and combining a spatial graph according to each spatial node and spatial edge;

[0025] performing feature extraction on the semantic graph to obtain a semantic high-level feature, performing feature extraction on the spatial graph to obtain a spatial high-level feature, and performing multi-layer convolution operation on the target initial data by a second preset model to obtain a visual feature;

[0026] fusing the semantic high-level feature, the spatial high-level feature and the visual feature to obtain a fusion feature vector, inputting the fusion feature vector into an LSTM to obtain a semantic feature, and combining each semantic feature to obtain semantic text data.

[0027] Optionally, classifying the target text data to obtain a classification data set, including:

[0028] performing word segmentation on the semantic feature in the target text data to obtain a word unit, performing a preset operation on each word unit to obtain a target word unit, and combining each target word unit to obtain a static word vector;

[0029] multiplying the static word vector and a preset weight matrix group to obtain a Q matrix, a K matrix and a V matrix, and calculating an attention weight matrix according to the Q matrix and the K matrix; the preset weight matrix group includes a Q weight matrix, a K weight matrix and a V weight matrix;

[0030] calculating the attention weight matrix and the V matrix to obtain a self-attention output, performing nonlinear transformation on the self-attention output to obtain a dynamic word vector, and combining each dynamic word vector to obtain a classification data set.

[0031] In a second aspect, the present application provides a multi-data-source-based public data platform retrieval system, which comprises:

[0032] An audio conversion module is configured to obtain an original data set, and convert audio data in the original data set to obtain audio text data; the original data set comprises text data, audio data and image data.

[0033] A label identification module is configured to perform feature detection on the image data in the original data set to obtain a feature identification label set, and fuse images in the image data according to a feature identification label in the feature identification label set to obtain an initial data set; the image data comprises a plurality of images.

[0034] A data classification module is configured to perform semantic recognition on each initial data in the initial data set to obtain semantic text data, combine the semantic text data, audio text data and text data to obtain target text data, and classify the target text data to obtain a classification data set.

[0035] A platform updating module is configured to upload the classification data set to a public data platform to update the public data platform.

[0036] Optionally, the label identification module is further configured to perform feature detection on the original data in the original data set by a target model to obtain a feature identification label, and combine each feature identification label to obtain a feature identification label set.

[0037] The target model based on the improved YOLOv8 model comprises:

[0038] The backbone network is replaced by a target backbone network, the C2f module in the neck network is replaced by a target improvement module, and the Upsample in the neck network is replaced by CARAFE to obtain the target model.

[0039] The working process of the target backbone network comprises:

[0040] An original feature tensor is obtained, the original feature tensor is sequentially input into an embedding module group and three merging module groups to obtain an initial feature tensor, the initial feature tensor is input into an SPPF module to obtain a target feature tensor, and the target feature tensor is taken as the output of the target backbone network; the embedding module group is composed of an embedding layer and a FasterNet module; and the merging module group is composed of a merging layer and a FasterNet module.

[0041] Optionally, the working principle of the target improvement module comprises:

[0042] obtain a first feature map by performing a local average pooling operation on the original feature map, and obtain a second feature map by sequentially performing a reshaping operation, a convolution operation and a reshaping operation on the first feature map;

[0043] obtain a third feature map by sequentially performing a global average pooling operation, a reshaping operation, a convolution operation and a uniform average pooling operation on the first feature map, and obtain a fourth feature map by fusing the second feature map and the third feature map;

[0044] obtain a target fourth feature map by performing a uniform average pooling operation on the fourth feature map, obtain a fifth feature map by performing element-by-element multiplication between the target fourth feature map and the original feature map, and take the fifth feature map as an output of the target improvement module.

[0045] Optionally, the data classification module comprises:

[0046] a preprocessing module configured to input target initial data in the initial data set into a first preset model to obtain a bounding box set and a relation tuple set; the target initial data is any one of the initial data set;

[0047] a semantic graph construction module configured to extract a subject from the relation tuple set to obtain a semantic node, extract a predicate from the relation tuple set to obtain a semantic edge, and construct a semantic graph according to the semantic node and the semantic edge;

[0048] a spatial graph construction module configured to extract a geometric feature from a target bounding box in the bounding box set to obtain a spatial node, determine a spatial edge between node features according to the relation tuple set, and combine each spatial node and spatial edge to obtain a spatial graph;

[0049] a visual feature extraction module configured to extract a semantic high-level feature from the semantic graph, extract a spatial high-level feature from the spatial graph, and perform a multi-layer convolution operation on the target initial data by using a second preset model to obtain a visual feature;

[0050] a feature fusion module configured to fuse the semantic high-level feature, the spatial high-level feature and the visual feature to obtain a fusion feature vector, input the fusion feature vector into an LSTM to obtain a semantic feature, and combine each semantic feature to obtain semantic text data.

[0051] Optionally, the data classification module further comprises:

[0052] a semantic word segmentation module configured to segment a semantic feature in the target text data to obtain a word unit, perform a preset operation on each word unit to obtain a target word unit, and combine each target word unit to obtain a static word vector;

[0053] The weight calculation module is configured to multiply the static word vector with a preset weight matrix group to obtain a Q matrix, a K matrix and a V matrix, and calculate an attention weight matrix according to the Q matrix and the K matrix; the preset weight matrix group includes a Q weight matrix, a K weight matrix and a V weight matrix;

[0054] The vector generation module is configured to calculate the attention weight matrix and the V matrix to obtain a self-attention output, perform nonlinear transformation on the self-attention output to obtain a dynamic word vector, and combine each dynamic word vector to obtain a classification data set.

[0055] The application has the following beneficial effects:

[0056] The application provides a public data platform retrieval method based on multiple data sources. The method includes the following steps: obtaining an original data set, converting audio data in the original data set to obtain audio text data, performing feature detection on image data to obtain a feature recognition label set, fusing images according to the feature recognition label to obtain an initial data set, performing semantic recognition on the initial data to obtain semantic text data, combining the semantic text data, the audio text data and text data to obtain target text data, classifying the target text data to obtain a classification data set, and uploading the classification data set to a public data platform to update the public data platform. The method extracts features from data and performs semantic analysis to generate a text description and classification, improves data retrieval efficiency, provides more accurate classification labels for a retrieval system, improves retrieval accuracy, uploads the classified data set to the platform for updating, and optimizes data storage structure. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 The application provides a flowchart of a public data platform retrieval method based on multiple data sources.

[0058] Figure 2 The application provides a framework diagram of a public data platform retrieval system based on multiple data sources. DETAILED DESCRIPTION

[0059] To further illustrate the technical means and effects adopted by the application to achieve the predetermined application purposes, the specific embodiments, structures, features and effects of the application are described in detail below with reference to the drawings and preferred embodiments.

[0060] The application provides a public data platform retrieval method based on multiple data sources. Referring to Figure 1 , Figure 1 The application provides a flowchart of a public data platform retrieval method based on multiple data sources. The method includes the following steps:

[0061] S101, obtain an original data set, and transform audio data in the original data set to obtain audio text data;

[0062] S102, perform feature detection on image data in the original data set to obtain a feature recognition label set, and fuse images in the image data according to feature recognition labels in the feature recognition label set to obtain an initial data set;

[0063] S103, perform semantic recognition on each initial data in the initial data set to obtain semantic text data, combine the semantic text data, the audio text data and the text data to obtain target text data, and perform classification on the target text data to obtain a classification data set;

[0064] S104, upload the classification data set to a public data platform to update the public data platform.

[0065] The original data set includes text data, audio data and image data, and the image data includes multiple images.

[0066] According to the public data platform retrieval method based on multiple data sources provided by the embodiment of the application, the data is subjected to feature extraction and semantic analysis to generate a text description and classification, the retrieval efficiency of the data is improved, more accurate classification labels are provided for the retrieval system, the retrieval precision is improved, the data set after classification is uploaded to the platform for updating, and the data storage structure is optimized.

[0067] In an implementation manner, the public data platform refers to a comprehensive platform built by a public institution or an authorized organization, which integrates, governs, opens and shares public domain data, for example, traffic data and environmental data; for example, the Shanghai public data opening platform.

[0068] In an implementation manner, the multiple data sources refer to data in diversified forms from different collection channels, including text (for example, document, social media content, log record and the like), audio (for example, voice call, environmental sound and the like), image (for example, photo, remote sensing image, medical slice and the like), video (for example, monitoring video, short video, film and television segment and the like) and the like, the public data platform performs text conversion on the data of itself, and performs data classification based on the text features, so as to improve the retrieval efficiency.

[0069] In one implementation, for audio in multiple data sources, it can be converted into text through voice-to-text conversion; through feature extraction and semantic analysis of picture (video) data, text description is generated and classified, which can provide more accurate index information for the public data platform; multimedia data (pictures or videos) are converted into structured text information, which can directly search data based on text information, avoid direct processing of original data, and improve search efficiency.

[0070] In one implementation, semantic recognition and classification are performed on pictures (videos) to generate semantic text data and perform classification, which can provide more accurate classification labels for data retrieval; the classification labels reflect the semantic content of the data, so that the retrieval system can accurately locate the relevant content and improve the accuracy of data retrieval.

[0071] In one implementation, the classified data set after feature extraction and semantic analysis is uploaded to the public data platform for updating, which can provide the platform with clearer and more consistent data storage structure. The data manageability is improved, and the data storage transparency is enhanced, and the platform can better support data retrieval and management.

[0072] In one implementation, the original data in the original data set is fused according to the feature recognition labels in the feature recognition label set to obtain an initial data set, that is, the feature recognition labels are added to the original data to obtain the initial data, and each initial data set is combined to obtain the initial data set.

[0073] In one embodiment, feature detection is performed on each original data in the original data set to obtain a feature recognition label set, including:

[0074] The feature recognition labels are obtained by performing feature detection on the original data in the original data set through the target model, and each feature recognition label is combined to obtain the feature recognition label set;

[0075] The improvement of the target model based on the YOLOv8 model includes:

[0076] The backbone network is replaced by the target backbone network, the C2f module in the neck network is replaced by the target improvement module, and the Upsample in the neck network is replaced by CARAFE to obtain the target model;

[0077] The workflow of the target backbone network includes:

[0078] The original feature tensor is obtained, the original feature tensor is sequentially input into an embedding module group and three merging module groups to obtain an initial feature tensor, the initial feature tensor is input into an SPPF module to obtain a target feature tensor, and the target feature tensor is taken as an output of a target backbone network; the embedding module group is composed of an embedding layer and a FasterNet module; and the merging module group is composed of a merging layer and a FasterNet module.

[0079] In an implementation manner, the embedding layer is an input processing module of the FasterNet, which converts the original feature tensor into a feature map that can be processed by the model and performs preliminary dimension adjustment; for example, the original feature tensor is usually three-dimensional (HxWx3, i.e., heightxwidthxRGB channel), and the embedding layer converts it into a high-dimensional feature map (H'xW'xC) through convolution operation, where C is the initial channel number of the model; for example, for an input image of 224x224, the embedding layer may use convolution with stride=4 to reduce the spatial size to 56x56, and expand the channel number from 3 to the initial dimension; reducing the spatial dimension of the original image while increasing the channel dimension provides a richer semantic basis for subsequent feature extraction, and reduces the computational amount through downsampling.

[0080] In an implementation manner, the FasterNet module (FasterNetBlock) comprises: a PWConv1 (point-wise convolution 1): using 1x1 convolution to expand the channel number of the input feature (for example, by 2 times), to provide richer feature dimension for subsequent calculation; a PConv (partial convolution): only 3x3 convolution is performed on part of the channels (for example, 1 / 4 channels) of the input feature, and the remaining channels are directly reserved, which reduces redundant calculation while maintaining feature expression ability by reserving most channel information; a PWConv2 (point-wise convolution 2): using 1x1 convolution to restore the channel number to the input dimension, while fusing information of all channels (including channels processed by the PConv and channels not processed); under the premise of ensuring feature extraction ability, the PConv reduces the computational amount and memory access cost of the convolution operation, thereby improving the inference speed of the model.

[0081] In an implementation manner, the merging layer (Merging) is a regular 2x2 convolution with a stride of 2; it can perform downsampling operation on the input feature map, reduce the spatial size of the feature map, and increase the channel number, to provide a suitable input format for the subsequent FasterNet block, which is helpful to extract higher-level features.

[0082] In an implementation, the target backbone network gradually extracts and fuses features through the cascade processing of the embedding module group and the merging module group; the embedding layer converts the original feature tensor into a high-dimensional feature map, reduces the spatial dimension while increasing the channel dimension, and provides a richer semantic basis for subsequent feature extraction; the merging layer further reduces the spatial size of the feature map and increases the number of channels through downsampling operation, and provides a suitable input format for extracting higher-level features; the efficiency of feature extraction is improved, and the computational amount is reduced.

[0083] In one embodiment, the working principle of the target improvement module includes:

[0084] Obtain an original feature map, perform a local average pooling operation on the original feature map to obtain a first feature map, and sequentially perform reshaping, convolution, and reshaping operations on the first feature map to obtain a second feature map;

[0085] Perform a global average pooling, reshaping, convolution, and uniform average pooling operation on the first feature map in sequence to obtain a third feature map, and fuse the second feature map and the third feature map to obtain a fourth feature map;

[0086] Perform a uniform average pooling operation on the fourth feature map to obtain a target fourth feature map, multiply the target fourth feature map with the original feature map element by element to obtain a fifth feature map, and take the fifth feature map as the output of the target improvement module.

[0087] In an implementation, local average pooling: calculate the average value in the input local neighborhood, and replace the neighborhood with the average value, reduce the spatial resolution of the feature map (reduce the size), provide summary information of local area, and introduce certain translation invariance. The spatial position information of the input data is retained, but more emphasis is placed on local characteristics.

[0088] In an implementation, global average pooling: calculate the average value of all elements in the entire spatial dimension (height x width) of the input, and generate a single value (or a low-dimensional vector, the number of channels remains unchanged); compress the spatial information to each channel to obtain a channel description vector.

[0089] In an implementation, reshaping: change the shape (dimension) of the data tensor without changing its data content and order; addition adds the elements at corresponding positions of two feature maps one by one, and combines the information from different paths (local average pooling and global average pooling); addition allows the model to utilize the complementary features extracted by the two paths at the same time.

[0090] In an implementation, local and global information of the feature map is extracted through local average pooling and global average pooling operations respectively. The local average pooling reduces the spatial resolution of the feature map, provides summary information of the local region and introduces translation invariance, while the global average pooling compresses the spatial information onto each channel to generate a channel description vector. The feature map is further processed through reshaping, convolution and other operations, and the local and global feature maps are fused to obtain a richer feature representation. The fused feature map is combined with the original feature map through element-wise multiplication to generate a feature map with stronger expression ability and discrimination, which is output as the target improvement module. The feature map can better capture the details and context information of the feature, and improve the understanding and processing ability of the model for the input data.

[0091] In one embodiment, semantic text data is obtained by performing semantic recognition on each initial data in the initial data set, including:

[0092] The target initial data in the initial data set is input into the first preset model to obtain a bounding box set and a relation tuple set; the target initial data is any one of the initial data set;

[0093] The subject of the relation tuple set is extracted to obtain a semantic node, and the predicate of the relation tuple set is extracted to obtain a semantic edge, and a semantic graph is constructed according to the semantic node and the semantic edge;

[0094] The geometric features of the target bounding box in the bounding box set are extracted to obtain a spatial node, the spatial edges between the node features are determined according to the relation tuple set, and a spatial graph is obtained by combining the spatial nodes and the spatial edges;

[0095] The semantic high-level features are obtained by feature extraction on the semantic graph, the spatial high-level features are obtained by feature extraction on the spatial graph, and the visual features are obtained by performing multi-layer convolution operation on the target initial data through the second preset model;

[0096] The semantic high-level features, the spatial high-level features and the visual features are fused to obtain a fusion feature vector, the fusion feature vector is input into the LSTM to obtain semantic features, and the semantic text data is obtained by combining the semantic features.

[0097] In an implementation, the bounding box coordinates (including geometric information such as target center and size) and 〈subject-predicate-object〉 relation pairs of the target in the image are extracted through the first preset model RelTR model (based on the Transformer architecture, including feature encoder, entity decoder and triple decoder), for example: car-on-street. The bounding box is the core data for constructing the spatial graph, and the relation tuple is the core data for constructing the semantic graph.

[0098] In an implementation, unique entities (i.e. subjects, i.e. subjects and objects) in the relational tuples are extracted as nodes of the semantic graph, represented by node labels (nodes); predicates in the relational tuples are extracted as labels of edges between nodes (edges); semantic nodes and semantic edges are converted into numerical feature vectors through word embedding, and a semantic graph is constructed based on the numerical feature vectors.

[0099] In an implementation, the bounding boxes of each target are taken as nodes, geometric features are extracted: center coordinates, size as spatial nodes, and spatial edges between nodes are determined based on the relational tuples; spatial features of the edges are calculated: angles, distances, and overlap degrees between nodes, which are classified into semantic features such as "up, down, left, right", "near, middle, far", and a spatial graph is constructed.

[0100] In an implementation, the same architecture of GCN (containing 2 EdgeConv layers and 1 GCNConv layer) is used for the semantic graph and the spatial graph; the EdgeConv layer captures local relationships between nodes, and the GCNConv layer propagates global features, finally outputting context-aware semantic high-level features and spatial high-level features; a second preset model (ResNet-152 model) is used to perform multi-layer convolution operations on the target initial data (image), generating visual features encoding high-level visual information.

[0101] In an implementation, Bahdanau attention mechanism is applied to the semantic high-level features, the spatial high-level features, and the visual features respectively, and attention weights are calculated in combination with the current hidden state; the three kinds of features are weighted and summed according to the weights, obtaining the semantic features, spatial features, and visual features after attention; a gating scalar (sigmoid transformation based on the hidden state) is introduced to dynamically adjust the contribution degrees of the three kinds of features to obtain a fusion feature vector; the fusion feature vector is spliced with the word embedding and input into an LSTM model; the LSTM updates the hidden state and outputs the prediction result of the next word through a softmax function; the above process is repeated until a complete image description text is generated; the accuracy and richness of the image description are improved, the understanding ability of the model for complex image scenes is enhanced, and the effect and efficiency of image description generation are improved.

[0102] In an embodiment, the target text data is classified to obtain a classification data set, including:

[0103] The semantic features in the target text data are segmented to obtain word units, and a preset operation is performed on each word unit to obtain a target word unit, and each target word unit is combined to obtain a static word vector;

[0104] The static word vector is multiplied by a preset weight matrix set to obtain a Q matrix, a K matrix and a V matrix, and an attention weight matrix is calculated according to the Q matrix and the K matrix; the preset weight matrix set includes a Q weight matrix, a K weight matrix and a V weight matrix;

[0105] The self-attention output is obtained by calculating the attention weight matrix and the V matrix, and the dynamic word vector is obtained by performing nonlinear transformation on the self-attention output, and the classification data set is obtained by combining the dynamic word vectors.

[0106] In an implementation, a preset operation is performed on each word unit to obtain a target word unit, for example: truncating or padding the text sequence to a fixed length; after obtaining the word units by word segmentation and generating the static word vectors, the static word vectors are multiplied by a preset weight matrix set (Q weight matrix, K weight matrix and V weight matrix) to generate a Q matrix, a K matrix and a V matrix, and an attention weight matrix is calculated, and finally a dynamic word vector is obtained through self-attention output and nonlinear transformation; the above process enables the word vector to be dynamically adjusted according to the context, thereby better capturing the semantic changes of words in different contexts and improving the understanding and expression ability of the model for semantics.

[0107] In an implementation, the multi-head self-attention mechanism enhances the ability of the model to capture context information. After generating the Q matrix, the K matrix and the V matrix, the attention weight matrix calculated from the Q matrix and the K matrix can measure the correlation between words, thereby realizing the weighted fusion of context information; when processing text, not only the current word itself is focused on, but also the context information is dynamically combined to adjust the semantic representation of the word, thereby improving the adaptability and accuracy of the dynamic word vector to complex semantic scenarios.

[0108] In an implementation, the introduction of nonlinear transformation further improves the expression ability of the dynamic word vector. The nonlinear transformation is performed by a feedforward neural network, so that the word vector can more flexibly represent complex semantic relationships, and the dynamic word vector can better capture subtle differences in semantics.

[0109] Based on the same inventive concept, the embodiments of the present application also provide a public data platform retrieval system based on multiple data sources. Referring to Figure 2 , Figure 2 A framework diagram of a public data platform retrieval system based on multiple data sources is provided for the embodiments of the present application, which includes:

[0110] The audio conversion module is configured to obtain an original data set and convert audio data in the original data set to obtain audio text data; the original data set includes text data, audio data and image data.

[0111] The label recognition module is configured to perform feature detection on image data in the original data set to obtain a feature recognition label set, and perform fusion on images in the image data according to a feature recognition label in the feature recognition label set to obtain an initial data set; the image data includes a plurality of images.

[0112] The data classification module is configured to perform semantic recognition on each initial data in the initial data set to obtain semantic text data, combine the semantic text data, audio text data and text data to obtain target text data, and perform classification on the target text data to obtain a classification data set.

[0113] The platform updating module is configured to upload the classification data set to a public data platform to update the public data platform.

[0114] In an embodiment, the label recognition module is further configured to perform feature detection on original data in the original data set by a target model to obtain a feature recognition label, and combine each feature recognition label to obtain the feature recognition label set.

[0115] The improvement of the target model based on the YOLOv8 model includes:

[0116] The backbone network is replaced by a target backbone network, the C2f module in the neck network is replaced by a target improvement module, and the Upsample in the neck network is replaced by CARAFE to obtain the target model.

[0117] The working process of the target backbone network includes:

[0118] The original feature tensor is obtained, the original feature tensor is input into an embedding module group and three merging module groups in sequence to obtain an initial feature tensor, the initial feature tensor is input into an SPPF module to obtain a target feature tensor, and the target feature tensor is taken as an output of the target backbone network; the embedding module group is composed of an embedding layer and a FasterNet module; and the merging module group is composed of a merging layer and a FasterNet module.

[0119] In an embodiment, the working principle of the target improvement module includes:

[0120] The original feature map is obtained, local average pooling is performed on the original feature map to obtain a first feature map, reshaping, convolution and reshaping are performed on the first feature map in sequence to obtain a second feature map;

[0121] Global average pooling, reshaping, convolution and uniform average pooling are performed on the first feature map in sequence to obtain a third feature map, the second feature map and the third feature map are fused to obtain a fourth feature map.

[0122] Performing a uniform average pooling operation on the fourth feature map obtains a target fourth feature map, performing element-wise multiplication between the target fourth feature map and the original feature map obtains a fifth feature map, and taking the fifth feature map as the output of the target improvement module.

[0123] In one embodiment, the data classification module comprises:

[0124] The preprocessing module is configured to input target initial data in the initial data set into the first preset model to obtain a bounding box set and a relation tuple set; the target initial data is any one of the initial data set;

[0125] The semantic graph construction module is configured to extract a subject from the relation tuple set to obtain a semantic node, extract a predicate from the relation tuple set to obtain a semantic edge, and construct a semantic graph according to the semantic node and the semantic edge;

[0126] The spatial graph construction module is configured to extract a geometric feature from a target bounding box in the bounding box set to obtain a spatial node, determine a spatial edge between node features according to the relation tuple set, and combine the spatial nodes and the spatial edge to obtain a spatial graph;

[0127] The visual feature extraction module is configured to extract a semantic high-level feature from the semantic graph, extract a spatial high-level feature from the spatial graph, and perform a multi-layer convolution operation on the target initial data by using the second preset model to obtain a visual feature;

[0128] The feature fusion module is configured to fuse the semantic high-level feature, the spatial high-level feature, and the visual feature to obtain a fusion feature vector, input the fusion feature vector into the LSTM to obtain a semantic feature, and combine the semantic features to obtain semantic text data.

[0129] In one embodiment, the data classification module further comprises:

[0130] The semantic word segmentation module is configured to segment the semantic features in the target text data to obtain word units, perform a preset operation on each word unit to obtain a target word unit, and combine the target word units to obtain a static word vector;

[0131] The weight calculation module is configured to multiply the static word vector and a preset weight matrix group to obtain a Q matrix, a K matrix, and a V matrix, and calculate an attention weight matrix according to the Q matrix and the K matrix; the preset weight matrix group comprises a Q weight matrix, a K weight matrix, and a V weight matrix;

[0132] The vector generation module is configured to calculate the attention weight matrix and the V matrix to obtain a self-attention output, perform a nonlinear transformation on the self-attention output to obtain a dynamic word vector, and combine the dynamic word vectors to obtain a classification data set.

[0133] The above merely describes the preferred embodiments of the present application, and is not intended to limit the present application in any form. Although the present application has been described above with the preferred embodiments, it is not intended to limit the present application, and any person skilled in the art can make some changes or modifications to the above disclosed technical contents to obtain equivalent embodiments with equivalent changes, as long as the changes or modifications do not deviate from the technical solutions of the present application. Any brief modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application still belongs to the scope of the technical solutions of the present application.

Claims

1. A method for retrieving public data based on a multi-data-source public data platform, characterized in that, The method comprises: obtaining an original data set, and converting audio data in the original data set to obtain audio text data; the original data set comprises text data, audio data and image data; performing feature detection on image data in the original data set to obtain a feature recognition label set, and fusing images in the image data according to feature recognition labels in the feature recognition label set to obtain an initial data set; the image data comprises a plurality of images; performing semantic recognition on each initial data in the initial data set to obtain semantic text data, combining the semantic text data, audio text data and text data to obtain target text data, and classifying the target text data to obtain a classification data set; uploading the classification data set to a public data platform to update the public data platform. 2.The method of claim 1, wherein, performing feature detection on each original data in the original data set to obtain a feature recognition label set, comprising: performing feature detection on the original data in the original data set by a target model to obtain a feature recognition label, and combining each feature recognition label to obtain a feature recognition label set; the target model is improved based on a YOLOv8 model, comprising: replacing a backbone network with a target backbone network, replacing a C2f module in a neck network with a target improvement module, and replacing Upsample in the neck network with CARAFE to obtain a target model; a working process of the target backbone network comprises: obtaining an original feature tensor, inputting the original feature tensor into an embedding module group and three merging module groups in sequence to obtain an initial feature tensor, inputting the initial feature tensor into an SPPF module to obtain a target feature tensor, and taking the target feature tensor as an output of the target backbone network; the embedding module group is composed of an embedding layer and a FasterNet module; the merging module group is composed of a merging layer and a FasterNet module. 3.The method of claim 2, wherein, a working principle of the target improvement module comprises: obtaining an original feature map, performing local average pooling on the original feature map to obtain a first feature map, and performing reshaping, convolution and reshaping on the first feature map in sequence to obtain a second feature map; performing global average pooling, reshaping, convolution and uniform average pooling on the first feature map in sequence to obtain a third feature map, fusing the second feature map and the third feature map to obtain a fourth feature map; performing uniform average pooling on the fourth feature map to obtain a target fourth feature map, element-wise multiplying the target fourth feature map and the original feature map to obtain a fifth feature map, and taking the fifth feature map as an output of the target improvement module.

4. The method of claim 1, wherein, performing semantic recognition on each initial data in the initial data set to obtain semantic text data, comprising: inputting target initial data in the initial data set into a first preset model to obtain a bounding box set and a relation tuple set; the target initial data is any one of the initial data set; extracting a subject from the relation tuple set to obtain a semantic node, extracting a predicate from the relation tuple set to obtain a semantic edge, and constructing a semantic graph according to the semantic node and the semantic edge; Geometric feature extraction is performed on the target bounding box in the bounding box set to obtain a spatial node, a spatial edge between node features is determined according to the relation tuple set, and a spatial graph is obtained by combining each spatial node and spatial edge; Feature extraction is performed on the semantic graph to obtain semantic high-level features, feature extraction is performed on the spatial graph to obtain spatial high-level features, and a visual feature is obtained by performing multi-layer convolution on the target initial data through a second preset model; The semantic high-level features, the spatial high-level features and the visual feature are fused to obtain a fusion feature vector, the fusion feature vector is input into an LSTM to obtain semantic features, and the semantic features are combined to obtain semantic text data.

5. The method of claim 1, wherein, Classification is performed on the target text data to obtain a classification data set, including: Word units are obtained by performing word segmentation on the semantic features in the target text data, a target word unit is obtained by performing a preset operation on each word unit, and a static word vector is obtained by combining each target word unit; A Q matrix, a K matrix and a V matrix are obtained by multiplying the static word vector and a preset weight matrix group, and an attention weight matrix is calculated according to the Q matrix and the K matrix; the preset weight matrix group includes a Q weight matrix, a K weight matrix and a V weight matrix; The attention weight matrix and the V matrix are calculated to obtain a self-attention output, a dynamic word vector is obtained by performing a nonlinear transformation on the self-attention output, and the classification data set is obtained by combining each dynamic word vector.

6. A multi-data source based public data platform retrieval system, characterized in that, The system includes: An audio conversion module is configured to obtain an original data set, and convert audio data in the original data set to obtain audio text data; the original data set includes text data, audio data and image data; A label recognition module is configured to detect features in the image data in the original data set to obtain a feature recognition label set, and fuse images in the image data according to the feature recognition label in the feature recognition label set to obtain an initial data set; the image data includes multiple images; A data classification module is configured to perform semantic recognition on each initial data in the initial data set to obtain semantic text data, combine the semantic text data, the audio text data and the text data to obtain target text data, and classify the target text data to obtain a classification data set; A platform update module is configured to upload the classification data set to a public data platform to update the public data platform.

7. The multi-data source based public data platform retrieval system according to claim 6, wherein, The label recognition module is further configured to detect features in the original data in the original data set through a target model to obtain a feature recognition label, and combine each feature recognition label to obtain a feature recognition label set; The target model is improved based on a YOLOv8 model, including: Replacing the backbone network with a target backbone network, replacing the C2f module in the neck network with a target improvement module, and replacing the Upsample in the neck network with CARAFE to obtain a target model; The working process of the target backbone network includes: The original feature tensor is obtained, the original feature tensor is sequentially input into an embedding module group and three merging module groups to obtain an initial feature tensor, the initial feature tensor is input into an SPPF module to obtain a target feature tensor, and the target feature tensor is taken as an output of the target backbone network; the embedding module group is composed of an embedding layer and a FasterNet module; and the merging module group is composed of a merging layer and a FasterNet module.

8. The multi-data source based public data platform retrieval system of claim 7, wherein, The working principle of the target improvement module includes: The original feature map is obtained, local average pooling operation is performed on the original feature map to obtain a first feature map, reshaping, convolution and reshaping operations are sequentially performed on the first feature map to obtain a second feature map; Global average pooling, reshaping, convolution and uniform average pooling operations are sequentially performed on the first feature map to obtain a third feature map, the second feature map and the third feature map are fused to obtain a fourth feature map; Uniform average pooling operation is performed on the fourth feature map to obtain a target fourth feature map, the target fourth feature map is multiplied with the original feature map element by element to obtain a fifth feature map, and the fifth feature map is taken as an output of the target improvement module.

9. The multi-data source based public data platform retrieval system of claim 6, wherein, The data classification module includes: A preprocessing module is configured to input target initial data in the initial data set into a first preset model to obtain a bounding box set and a relation tuple set; the target initial data is any one in the initial data set; A semantic graph construction module is configured to extract a subject from the relation tuple set to obtain a semantic node, extract a predicate from the relation tuple set to obtain a semantic edge, and construct a semantic graph according to the semantic node and the semantic edge; A spatial graph construction module is configured to extract a geometric feature from a target bounding box in the bounding box set to obtain a spatial node, determine a spatial edge between node features according to the relation tuple set, and combine each spatial node and spatial edge to obtain a spatial graph; A visual feature extraction module is configured to extract a semantic high-level feature from the semantic graph, extract a spatial high-level feature from the spatial graph, and perform multi-layer convolution on the target initial data through a second preset model to obtain a visual feature; A feature fusion module is configured to fuse the semantic high-level feature, the spatial high-level feature and the visual feature to obtain a fusion feature vector, input the fusion feature vector into an LSTM to obtain a semantic feature, and combine each semantic feature to obtain semantic text data.

10. The multi-data source based public data platform retrieval system of claim 6, wherein, The data classification module further includes: A semantic word segmentation module is configured to segment a semantic feature in the target text data to obtain a word unit, perform a preset operation on each word unit to obtain a target word unit, and combine each target word unit to obtain a static word vector; A weight calculation module is configured to multiply the static word vector with a preset weight matrix group to obtain a Q matrix, a K matrix and a V matrix, and calculate an attention weight matrix according to the Q matrix and the K matrix; the preset weight matrix group includes a Q weight matrix, a K weight matrix and a V weight matrix. The vector generation module is configured to calculate the attention weight matrix and the V matrix to obtain a self-attention output, perform a nonlinear transformation on the self-attention output to obtain a dynamic word vector, and combine each dynamic word vector to obtain a classification data set.

Images