Data extraction method and device, equipment and medium

Abstract

The invention provides a data extraction method and device, equipment and a medium. The method comprises the steps that data to be extracted are obtained from a change database, change data obtained through change detection are stored in the change database, and the change data comprise at least two of text data, image data and video data; encoding the to-be-extracted data to obtain feature data of the to-be-extracted data; and performing data extraction on the feature data of the to-be-extracted data according to a preset requirement to obtain extracted data. According to the technical scheme, the data information extraction efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data extraction method, device, equipment and medium. Background Art

[0002] With the rapid development of Internet technology and the advent of the big data era, the information systems of large multinational companies are becoming increasingly complex, the amount of data involved is increasing dramatically, and the data types are becoming more and more diverse. The lightweight data exchange format (English: JavaScript Object Notation, referred to as JSON) has become an indispensable part of modern information systems because it is easy for people to read and write, and easy for machines to parse and generate. However, with the expansion of business scale and the increase in complexity, change detection and information extraction of JSON data has become a major challenge facing enterprises.

[0003] In the prior art, methods such as hash table-based change detection methods, field-by-field comparison methods, regular expression and template matching methods, and application programming interface (API) management and monitoring tools are mainly used to implement change detection and information extraction of JSON data.

[0004] However, when the data is highly complex and the data volume is large, the above method has low accuracy, resulting in low efficiency of the data extraction method. Summary of the invention

[0005] The present application provides a data extraction method, device, equipment and medium to solve the technical problem that the data extraction method has low accuracy and low efficiency when the data is highly complex and the data volume is large.

[0006] In a first aspect, the present application provides a data extraction method, comprising:

[0007] Acquire data to be extracted from a change database, wherein the change database stores change data obtained through change detection, wherein the change data includes at least two of text data, image data, and video data;

[0008] Performing encoding processing on the data to be extracted to obtain feature data of the data to be extracted;

[0009] Data extraction is performed on the characteristic data of the data to be extracted according to preset requirements to obtain extracted data.

[0010] Optionally, in the method as described above, encoding the data to be extracted to obtain feature data of the data to be extracted includes:

[0011] Determine text data to be extracted, image data to be extracted, and video data to be extracted according to the data to be extracted;

[0012] Performing data processing on the text data to be extracted to obtain text data to be encoded, wherein the data processing includes data cleaning, data segmentation and word embedding processing;

[0013] Performing feature processing on the image data to be extracted and the video data to be extracted to obtain the image data to be encoded and the video data to be encoded;

[0014] The text data to be encoded, the image data to be encoded and the video data to be encoded are encoded to obtain feature data of the data to be extracted.

[0015] Optionally, in the method as described above, encoding the text data to be encoded, the image data to be encoded, and the video data to be encoded to obtain feature data of the data to be extracted includes:

[0016] Inputting the text data to be encoded into a text encoder in a multimodal encoder to obtain text encoded data;

[0017] Inputting the image data to be encoded and the video data to be encoded into a visual encoder in the multimodal encoder to obtain visual encoding data;

[0018] The text encoding data and the visual encoding data are fused through a fusion layer in the multimodal encoder to obtain feature data of the data to be extracted;

[0019] Among them, the fusion layer in the multimodal encoder is a feature fusion layer obtained by training the neural network algorithm based on the attention mechanism.

[0020] Optionally, in the above method, before obtaining the data to be extracted from the change database, the method further includes:

[0021] Acquire JSON data, storage path and data type information of the JSON data from a source database;

[0022] The JSON data is indexed, and according to the storage path and data type information of the JSON data and the preset standard data information, a pre-acquired comparison model is used to perform change detection on the path and value in the JSON data to determine the changed database;

[0023] The comparison model is a model pre-trained based on a deep learning algorithm and used to detect changes in the storage path and value of data.

[0024] Optionally, in the method described above, before acquiring the JSON data, the storage path and data type information of the JSON data from the source database, the method further includes:

[0025] Collect JSON data from data sources, including API interfaces, databases, and file systems;

[0026] The JSON data is stored in the source database in the form of key-value pairs.

[0027] Optionally, in the method as described above, extracting the characteristic data of the data to be extracted according to preset requirements to obtain the extracted data includes:

[0028] According to the inherent information in the features of the decoder obtained in advance and the preset data extraction requirements, data extraction is performed on the feature data of the data to be extracted to obtain the extracted data.

[0029] Optionally, the method as described above further comprises:

[0030] The extracted data is structured, and the structured extracted data is stored in an extraction database using a database management tool or a programming language.

[0031] In a second aspect, the present application provides a data extraction device, comprising:

[0032] A to-be-extracted data acquisition module is used to acquire the to-be-extracted data from a change database, wherein the change database stores change data obtained through change detection, wherein the change data includes at least two of text data, image data, and video data;

[0033] A data encoding processing module, used for encoding the data to be extracted to obtain feature data of the data to be extracted;

[0034] The data extraction module is used to extract the characteristic data of the data to be extracted according to preset requirements to obtain the extracted data.

[0035] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0036] The memory stores computer-executable instructions;

[0037] The processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0039] The data extraction method, device, equipment and medium provided by the present application obtain the data to be extracted from a change database, wherein the change database stores the change data obtained through change detection, and the change data includes at least two of text data, image data and video data; encodes the data to be extracted to obtain feature data of the data to be extracted; extracts the feature data of the data to be extracted according to preset requirements to obtain means for extracting data, thereby achieving the effect of extracting data efficiently and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] Figure 1 A schematic diagram of a scenario for the data extraction method proposed in this application;

[0042] Figure 2 This is a flow chart of the first embodiment of the data extraction method proposed in this application;

[0043] Figure 3 This is a flow chart of the second embodiment of the data extraction method proposed in this application;

[0044] Figure 4 This is a flow chart of the third embodiment of the data extraction method proposed in this application;

[0045] Figure 5 This is a flow chart of the fourth embodiment of the data extraction method proposed in this application;

[0046] Figure 6 This is a flow chart of Embodiment 5 of the data extraction method proposed in this application;

[0047] Figure 7 This is a flow chart of Embodiment 6 of the data extraction method proposed in this application;

[0048] Figure 8 The overall architecture diagram of the data extraction method proposed in this application;

[0049] Fig. 9 A schematic diagram of the structure of the data extraction device proposed in this application;

[0050] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0051] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0052] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0053] With the rapid development of Internet technology and the advent of the big data era, the information systems of large multinational companies are becoming increasingly complex, the amount of data involved is increasing dramatically, and the data types are becoming more and more diverse. As the scale of business expands and the complexity increases, change detection and information extraction of JSON data have become a major challenge for enterprises. Traditional JSON change detection methods often rely on simple field-by-field comparisons. These methods can only extract information under a single data type, but are inefficient when processing large-scale data or complex data structures, and have difficulty capturing subtle changes.

[0054] In order to solve the above problems, the inventors have come up with a data extraction method that uses the powerful learning ability of deep learning large models to process multimodal data such as text and images / videos in JSON data through encoding and decoding, and accurately identifies key information from JSON data, significantly improving the efficiency, accuracy and comprehensiveness of JSON data processing tasks, achieving cross-modal information extraction and fusion, and bringing new opportunities and challenges to the development of related fields.

[0055] Figure 1 Schematic diagram of the scenario of the data extraction method provided by the present invention. Figure 1 As shown, it includes a data source 101, a change database 102 and an extraction database 103, wherein the data source 101 provides JSON data from different sources, and after change detection, the detected change data is stored in the change database 102, and the data in the change database 102 will enter the data extraction stage. After the extraction is completed, the extracted data can be stored in the extraction database 103 to support subsequent data analysis, report generation and application in various business scenarios, such as data analysis, decision support, etc. These applications can help enterprises better understand their business data, discover potential data problems, and thus formulate more effective management strategies.

[0056] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0057] Figure 2 This is a flow chart of the first embodiment of the data extraction method proposed in this application. Figure 1 As shown, the method may include:

[0058] S201. Obtain data to be extracted from a change database.

[0059] The change database stores the change data obtained through change detection, and the change data includes at least two of multimodal data such as text data, image data and video data. The data to be extracted refers to the data that needs to be extracted, and these data have been detected as change data.

[0060] In this step, the data in the change database needs to be preprocessed to meet the input requirements of the subsequent model. For example, for the text data in the change database, first clean and denoise it to remove possible noise and irrelevant characters, and then perform word segmentation. After word segmentation, the text data can be converted into a sequence of words or subwords. For image and video data, first extract key frames (for video data), and then apply computer vision technology to extract features. Convolutional neural networks are usually used to extract feature maps or object detection results of images or videos. These feature maps will also be used as input to the model to provide visual feature data.

[0061] S202: Encode the data to be extracted to obtain feature data of the data to be extracted.

[0062] Among them, encoding processing is to extract features from the data to be extracted through a specific model or algorithm, so that the data is converted into feature vectors or feature maps that can be processed by the machine learning model. Feature data refers to the numerical representation of the data to be extracted after encoding processing, which can reflect the feature information of different modalities such as text, images and videos.

[0063] In this step, a multimodal encoder is designed to process data in different modalities such as text, image, and video. The multimodal encoder can be composed of two parallel encoding modules: one dedicated to processing text features and the other to processing image / video features. Each encoder module encodes the input features separately and converts them into high-dimensional feature representations. To further integrate features from different modalities, the encoder output can be combined using an attention mechanism or a feature fusion layer to correlate text and image / video features.

[0064] After the encoder, a cross-modal interaction layer is added to promote the interaction between text and image / video features. This interaction layer can be implemented through technologies such as attention mechanism, graph neural network or transformer. The design of the cross-modal interaction layer can enhance the mutual understanding of different modal features and enable the model to better capture the implicit information in multimodal data.

[0065] S203, extracting feature data of the data to be extracted according to preset requirements to obtain extracted data.

[0066] In this step, a decoder can be used to process the encoded multimodal feature data to generate the required data extraction results. For example, depending on the requirements of the extraction task, the decoder can be a classifier or a sequence generator, etc. For example, the decoder can output labels or generate structured information based on the multimodal features to complete the data extraction task.

[0067] Specifically, the decoder extracts target information from the encoded multimodal features and generates extracted data, which can be used as input for further data analysis, storage, or applications.

[0068] The data extraction method proposed in the embodiment of the present application can efficiently process multiple data types such as text, images, and videos through this multimodal intelligence architecture, and realize the fusion and extraction of multimodal information through the cross-modal interaction layer. This architecture provides a more accurate and comprehensive solution in complex data processing tasks, which helps to extract valuable information from the change database.

[0069] Figure 3 This is a flow chart of the second embodiment of the data extraction method proposed in this application. Figure 3 As shown, in Figure 2 On the basis of the embodiment shown, the data to be extracted is encoded to obtain feature data of the data to be extracted. The specific process includes:

[0070] S301. Determine text data to be extracted, image data to be extracted, and video data to be extracted according to the data to be extracted.

[0071] In this step, it is necessary to identify the multimodal data content in the data based on the data to be extracted from the change database, including text data to be extracted, image data to be extracted, and video data to be extracted. For example, if the change database contains multiple types of data information, each data record needs to be scanned and parsed. The file extension, metadata tag, or data type tag can be used to identify which data belongs to text data, which belongs to image data, and which belongs to video data, so as to determine the different data types that need to be processed separately.

[0072] S302: Process the text data to be extracted to obtain text data to be encoded.

[0073] After obtaining the text data to be extracted, it is necessary to perform preliminary cleaning operations on the text data to be extracted, that is, data cleaning, to reduce the impact of noise on the model. For example, if the text data to be extracted contains web page content or user input content, the cleaning process will remove Hypertext Markup Language (HTML) tags or non-essential characters and retain the plain text information.

[0074] After the text data is cleaned, the text data will continue to be segmented. Rule-based methods or natural language processing tools can be used to segment continuous text content into independent words or sub-word sequences. For example, for Chinese text data, a rule-based method can be used to decompose the entire text into independent words or phrases to facilitate subsequent vector representation.

[0075] Finally, the pre-trained word embedding model is used to convert the segmented text data into a high-dimensional vector. These word embedding models can capture the semantic relationship of words and represent the text content in the form of vectors, which is convenient for the model to understand. For example, the pre-trained word embedding model is loaded, the segmentation results of the text data are input, and the corresponding word vectors are generated. These word vectors will be used as the input of the encoder to construct the feature representation of the text data to be encoded.

[0076] S303 , performing feature processing on the image data to be extracted and the video data to be extracted to obtain the image data to be encoded and the video data to be encoded.

[0077] For image data, it is first necessary to standardize the image data, including resizing, cropping, grayscale adjustment, etc., to ensure that the input image size is consistent, which is convenient for subsequent feature extraction. After the image data is standardized, a convolutional neural network model can be used to extract the high-dimensional feature vector of the image. The convolutional neural network model automatically extracts feature data such as edges, shapes, textures, etc. in the image data and generates a feature vector representation. Exemplarily, a pre-trained convolutional neural network model is loaded, and the image data is input into the model to extract the feature map of the image data and obtain a fixed-length vector representation for further processing.

[0078] For video data, key frame extraction technology can be used to reduce computational complexity. The extraction of key frames can be achieved by frame difference method, histogram similarity and other methods. After the key frames are extracted, a convolutional neural network model or a visual transformer can be applied to each frame to generate a feature vector for each frame. Exemplarily, a key frame extraction algorithm is executed on the video data, representative frames are selected, and features are extracted from each key frame using a pre-trained convolutional neural network model. The feature representation of the key frames is summarized to form a feature vector of the video for subsequent use by the model.

[0079] S304 , encoding the text data to be encoded, the image data to be encoded, and the video data to be encoded to obtain feature data of the data to be extracted.

[0080] In this step, a dedicated multimodal encoder can be used to jointly encode the data to be encoded in three different modes: text data, image data, and video data. The multimodal encoder includes a text encoding module and an image / video encoding module, and fuses the feature data after encoding. The text encoding module accepts the preprocessed text vector and generates a feature representation of the text; the image / video encoding module processes the image and video features and generates a visual feature representation. Through the feature fusion layer (such as an attention mechanism or feature splicing), the text and image / video features are cross-modally fused to form a comprehensive multimodal feature. The multimodal feature data after encoding is the feature data of the data to be extracted, and is ready to enter the extraction process to extract the required information.

[0081] The data extraction method proposed in the embodiment of the present application realizes effective feature extraction and cross-modal encoding processing of text data, image data and video data, provides a solid technical foundation for complex multimodal data processing tasks, and also provides more efficient and accurate support for subsequent decoding and data extraction.

[0082] Figure 4 This is a flow chart of the third embodiment of the data extraction method proposed in this application. Figure 4 As shown, in Figure 3 On the basis of the illustrated embodiment, the text data to be encoded, the image data to be encoded, and the video data to be encoded are encoded to obtain feature data of the data to be extracted, including:

[0083] S401. Input text data to be encoded into a text encoder in a multimodal encoder to obtain text encoded data.

[0084] In this step, the text data to be encoded after cleaning, word segmentation and word embedding can be input into the text encoder module in the multimodal encoder. For the text encoder module, a pre-trained natural language processing model can be used to generate a high-dimensional vector representation of the text data (i.e., text encoding data). Exemplarily, the text encoder encodes the input text data to be encoded to generate high-dimensional vectors with semantic representations. These high-dimensional vectors retain the semantic information of the text data, which is convenient for subsequent fusion with visual information.

[0085] In addition, the text encoder can also extract key features of text data through a multi-layer self-attention-based deep neural network model (Transformer) structure or other deep learning layers, and generate encoded data suitable for fusion. The output of each layer of the model provides different levels of semantic information in turn, which helps to capture the contextual relationship of the text.

[0086] S402: Input the image data to be encoded and the video data to be encoded into a visual encoder in a multimodal encoder to obtain visual encoding data.

[0087] In this step, the image data and video data to be encoded need to be input into the visual encoder module in the multimodal encoder for encoding processing. Among them, the visual encoder module is usually processed by convolutional neural network or visual transformer. Specifically, convolutional neural network is suitable for extracting low-level visual features such as edges, shapes, colors, etc. in image or video frames, while visual transformer is more suitable for understanding the image content as a whole. For example, for image data, the visual encoder will generate a high-dimensional vector (i.e., visual encoding data) representing objects, scenes or other visual features; for video data, the visual encoder can extract the feature vector of each key frame, and summarize or average the features of these frames to form an overall feature representation of the video. It should be noted that, similar to the text encoder, the visual encoder can include multiple layers of convolutional layers or self-attention mechanisms to gradually extract visual information at different levels, so that the feature data is gradually extracted from low-level pixel information to high-level semantic information, making it more expressive in cross-modal fusion.

[0088] S403, fusing the text encoding data and the visual encoding data through a fusion layer in the multimodal encoder to obtain feature data of the data to be extracted;

[0089] Among them, the fusion layer in the multimodal encoder is a feature fusion layer obtained by training the neural network algorithm based on the attention mechanism. In this step, the text encoding data and the visual encoding data are mainly input into the fusion layer in the multimodal encoder for feature fusion processing to obtain the final feature representation of the data to be extracted. Specifically, the fusion layer adopts a neural network algorithm based on the attention mechanism to achieve cross-modal fusion, wherein the attention mechanism can adaptively assign weights according to the importance or relevance of different modal data, so that the text and visual data can complement each other and strengthen the information when fused. Exemplarily, in the fusion process, the system aligns the semantic features in the text encoding data with the visual features in the visual encoding data through the attention mechanism. For example, the objects or scenes described in the text can correspond to the visual elements in the image / video, and the fusion layer combines the relevant features together to form a unified feature representation (i.e., the feature data of the data to be extracted).

[0090] It should be noted that the fusion layer uses an attention-based algorithm to establish associations by calculating the similarity between text and visual features. During the model training process, the supervision signal of the labeled data can be used to optimize the attention weights so that the model can more accurately identify and integrate multimodal information. The output after feature fusion contains a comprehensive expression of multimodal information, which can more comprehensively reflect the overall characteristics of the data to be extracted and provide a basis for the subsequent information extraction process.

[0091] The data extraction method provided in the embodiment of the present application can fuse the feature data extracted by the text encoder and the visual encoder in the multimodal encoder to generate a unified feature representation of the data to be extracted. This fusion method uses the attention mechanism to adaptively weight the association between text and visual features, achieving deep integration of multimodal data, thereby improving the accuracy and comprehensiveness of data extraction.

[0092] Figure 5 This is a flow chart of the fourth embodiment of the data extraction method proposed in this application. Figure 5 As shown, in Figures 2 to 4 Based on the embodiment, it is explained that before obtaining the data to be extracted from the change database, the method also includes:

[0093] S501. Acquire JSON data, storage path and data type information of the JSON data from a source database.

[0094] Before entering the acquisition and encoding process of the data to be extracted, you first need to obtain JSON data from the source database and perform change detection on it. Specifically, extract the data file in JSON format from the source database, and record the storage path and data type information of the data. Among them, the storage path is used to identify the physical location or logical location of the data in the database, and the data type information identifies the specific format and content structure of the data (such as text, image link or video link). For example, each record in the source database may contain different data types, such as text fields, image links, video links, etc., then this information will be extracted when obtaining data from the source database, laying the foundation for subsequent change detection and multimodal processing.

[0095] Furthermore, before obtaining JSON data, storage path and data type information of JSON data from the source database, it is necessary to collect JSON data from various data sources and store it in the source database in a standardized manner. This process includes the specific operations of collecting JSON data from the data source and storing it in the database in the form of key-value pairs. By collecting JSON data from different sources, data from various information sources can be aggregated and integrated, providing a diverse data foundation for subsequent change detection and multimodal processing. The specific process includes:

[0096] To collect JSON data from data sources, different data collection methods can be used according to different data source types, including API interfaces, databases, and file systems. For example, if the data source is an API interface, call the API interface through HTTP requests to obtain the returned JSON format data; for database-type data sources, access the database through database query languages ​​(such as SQL) or connection tools (such as JDBC, ODBC) to extract JSON data stored in the database table; for file system-type data sources, the system scans the JSON files in the specified folder, reads the file content and parses it into a JSON data structure.

[0097] The collected JSON data may have a nested hierarchical structure or an irregular data format. To facilitate subsequent indexing and quick retrieval, the JSON data can be standardized into key-value pairs and stored in the source database. Specifically, before the JSON data enters the source database, it is "flattened" to expand the nested JSON hierarchical structure and convert it into a key-value pair storage format. For example, each key corresponds to a path in the JSON structure, using the "path-field" format; each value corresponds to the data value on the path. In actual storage, each JSON data can be converted into a set of key-value pairs and stored in the source database in the form of key-value pairs.

[0098] S502, indexing the JSON data, and using a pre-acquired comparison model to perform change detection on the path and value in the JSON data according to the storage path and data type information of the JSON data and the preset standard data information, to determine the changed database;

[0099] Among them, the comparison model is a model pre-trained based on a deep learning algorithm for detecting the storage path of data and changes in values.

[0100] In this step, the obtained JSON data needs to be indexed, that is, the hierarchical structure of the JSON data needs to be parsed and an index of the data needs to be generated to associate the hierarchical path of the key-value pairs in the JSON data with the corresponding values, so that the data content of each path can be quickly accessed and compared during change detection, thereby building an index structure that can quickly retrieve data.

[0101] In addition, before extracting the changed data, a comparison model based on a deep learning algorithm is pre-trained. The model is trained with a large amount of labeled data and can identify changes in JSON data storage paths and values. During the model training process, the model needs to learn how to detect changes in structured data, especially differences in paths and content values, so as to have the ability to automatically detect changes in each path in the JSON structure (such as adding, deleting or modifying fields) so as to identify the differences when the data is updated or changed. In an achievable imitation stone brick, the model is trained through deep learning algorithms such as convolutional neural networks or recurrent neural networks to capture the change patterns of paths and data values ​​to achieve efficient and accurate change detection.

[0102] Exemplarily, the data change detection process may include: quickly locating each path and corresponding value of the JSON data through an index structure; using a pre-trained comparison model to compare the differences between the JSON data in the source database and the standard data path by path, checking whether the path structure has changed and whether the corresponding value has changed; for the paths and values ​​where changes are detected, marking them as changed data and outputting them to the change database.

[0103] Finally, the changed data after the comparison model detection will be summarized and stored in the change database to form a collection of data to be extracted, where the change data contained in the change database includes its path and the corresponding latest value, so that the subsequent processing steps can accurately extract the latest data changes. For example, the content stored in the change database may include the path of multimodal data such as text, images, and videos and the latest data value after the change, providing the latest data information for the subsequent encoding and information extraction process.

[0104] The data extraction method provided in the embodiment of the present application realizes the extraction, indexing and change detection of JSON data in the source database, and identifies and stores data with changes in structure and value in the change database. Using a pre-trained deep learning comparison model, changes in JSON data structure and content can be automatically detected to ensure that the latest change data can be obtained and updated in time before the multimodal data processing process to support subsequent information extraction tasks.

[0105] Figure 6 This is a flow chart of the fifth embodiment of the data extraction method proposed in this application. Figure 6 As shown, in Figures 2 to 4 Based on the embodiment, data extraction is performed on the feature data of the data to be extracted according to the preset requirements to obtain the extracted data, including:

[0106] S601 , extracting feature data of data to be extracted based on intrinsic information in features obtained in advance by a decoder and preset data extraction requirements to obtain extracted data.

[0107] This step describes the process of using a decoder to extract feature data from multimodal data. The purpose of data extraction is to extract information that meets preset requirements from multimodal features and convert it into structured, usable extracted data. The design of the decoder is usually based on a deep learning architecture, i.e., a recurrent neural network, a Transformer decoder, or a fully connected layer classifier. For example, if the data features to be extracted include text semantics and image features, the decoder can generate a corresponding text summary through a Transformer decoder, or perform object recognition on an image through a classifier, and output the category and location of the object recognized in the image.

[0108] Specifically, during the data extraction process, the decoder analyzes the input multimodal feature data and identifies the intrinsic information in the feature data. Intrinsic information refers to the semantic relationships, visual features, or structured information implicit in the multimodal feature data, which are retained by the preceding multimodal encoding and fusion layers. For example, if the input feature data contains semantic features of the text, the decoder can perform entity recognition based on word embedding information and extract specific names, locations, or time information; if the feature data contains visual features of video keyframes, the decoder can identify objects or activities in the video scene.

[0109] In addition, specific extraction requirements are set before data extraction, which stipulates what types of information need to be extracted from the feature data and in what format it should be presented. These extraction requirements can include target information such as data labels, text summaries, object classification, and sentiment analysis.

[0110] Therefore, after identifying the intrinsic information in the feature data, the decoder will output structured extracted data according to the extraction requirements, which contains the final refined results of the features of the extracted data. The output data can be labels, categories, summary text, numerical information, etc., depending on the output form of the decoder. For example, for a set of multimodal data containing text, images, and videos, if the extraction requirement is to identify object categories and generate image labels, the decoder will output the identified object categories and their position coordinates.

[0111] The data extraction method provided in the embodiment of the present application is based on the multimodal data extraction of the decoder, and can extract the extracted data that meets the task requirements from the fused multimodal feature data. The decoder uses the intrinsic information in the feature data and generates a structured output according to the preset extraction requirements, ensuring that the required information can be accurately extracted and the corresponding extracted data can be generated in the multimodal data processing task.

[0112] Figure 7 This is a flow chart of the sixth embodiment of the data extraction method proposed in this application. Figure 7 As shown, in Figures 2 to 4 Based on the embodiment, the method further includes:

[0113] S701. Structural processing is performed on the extracted data, and the extracted data after structural processing is stored in the extraction database using a database management tool or a programming language.

[0114] This step describes the process of structuring the extracted data generated by the decoder and storing the structured data in the extraction database.

[0115] First, the extracted data is structured and converted into a standardized table or hierarchical structure (such as table rows and columns or JSON format) for subsequent storage in the database. For example, for a set of multimodal data, if the information extracted by the decoder includes "text summary", "object category label" and "video frame time point", the structured processing will store this information in the "summary text", "label" and "time point" fields respectively, so that the data has a clear row and column structure.

[0116] For example, according to the content of the extracted data, the data is mapped to the preset fields. For example, the text content is stored in the "summary" field, the classification label is stored in the "category" field, and the video time point is stored in the "time point" field. The extracted data is then unified into table rows and columns or JSON hierarchical format for subsequent storage. For table data, each extracted data is stored as a row; for JSON data, key-value pairs are used to represent the content of each field.

[0117] After structuring, the data can be written into the extraction database using database management tools or programming languages. The extraction database is used to centrally store the processed multimodal data extraction results. Relational databases are usually selected as storage solutions, and the specific choice depends on the data structure and access requirements.

[0118] The data extraction method provided in the embodiment of the present application stores the multimodal extracted data output by the decoder in a standardized manner in an extraction database through structured processing, which is convenient for subsequent retrieval and analysis. Use database management tools or programming languages ​​to insert data to ensure efficient data storage and support structured queries. Structured storage provides reliable storage support for the use of multimodal data processing results, making it more convenient and efficient for enterprises or users to process and analyze extracted data.

[0119] Figure 8 This is the overall architecture diagram of the data extraction method proposed in this application. Figure 8 As shown, it includes data source layer, deep learning large model selection layer, data access and verification layer, data compliance check layer, data preprocessing layer, data storage layer, intelligent processing layer, change detection layer, data extraction layer, result processing and storage layer, log system layer, monitoring system layer, data compliance audit and reporting mechanism layer, exception handling and recovery mechanism layer, and user interface layer.

[0120] Among them, the data source layer is used to provide raw JSON data. The deep learning model selection layer is used to select the appropriate pre-trained deep learning model according to the specific task of data extraction. The data compliance check layer is to ensure that the data meets the requirements of relevant laws and regulations, industry standards and internal company policies when performing data extraction tasks. This includes checking the sensitivity, legality, accuracy, etc. of the data. In addition, for sensitive data, such as personal information, trade secrets, etc., desensitization processing is performed to reduce the risk of data leakage. Among them, desensitization methods include replacement, encryption, truncation, etc., and the specific method is determined according to the sensitivity and purpose of the data.

[0121] The log system layer records in detail various events and errors in the data extraction process, providing a basis for troubleshooting and performance optimization. The monitoring system layer can comprehensively monitor various activities of the extraction task, including user operations, system service status, resource usage, etc. This information provides a rich data source for the deep learning large model, enabling it to understand the system behavior more accurately, thereby optimizing the performance of change detection and intelligent information extraction. The data compliance audit and reporting mechanism layer provides management with comprehensive and objective information about the compliance status, which helps them make more informed decisions. Through regular reviews, potential data compliance risks can be identified and corresponding risk management measures can be formulated. This helps reduce the risks of legal disputes, fines, etc. caused by data compliance issues. The exception handling and recovery mechanism layer can monitor the change detection and intelligent information extraction process of JSON data in real time, and promptly discover and identify errors or anomalies in the data, such as format errors, type mismatches, missing data, etc. Once an error is detected, the exception handling mechanism will trigger corresponding corrective measures, such as automatically repairing erroneous data, replacing missing data with default values, etc., to ensure that the data processed subsequently is accurate. In addition, the exception handling and recovery mechanism layer also supports automatic recovery, that is, after detecting a specific type of exception, it can automatically perform a series of recovery operations to quickly restore the system to a normal state. This greatly reduces the need for manual intervention and improves the self-healing ability of the system; backup recovery, for abnormal situations that cannot be automatically recovered, the system usually relies on backup data for recovery; in short, the exception handling and recovery mechanism needs to ensure the integrity and availability of the backup data so that the system can be quickly restored to the most recent stable state when needed.

[0122] Fig. 9 This is a schematic diagram of the structure of the data extraction device proposed in this application. Fig. 9 As shown, the data extraction device 90 includes a to-be-extracted data acquisition module 901, a data encoding processing module 902, and an extracted data obtaining module 903, wherein:

[0123] The to-be-extracted data acquisition module 901 is used to acquire the to-be-extracted data from a change database, where the change database stores change data obtained through change detection, where the change data includes at least two of text data, image data, and video data;

[0124] A data encoding processing module 902 is used to encode the data to be extracted to obtain feature data of the data to be extracted;

[0125] The extracted data obtaining module 903 is used to extract the characteristic data of the data to be extracted according to preset requirements to obtain the extracted data.

[0126] The data encoding processing module 902 is further specifically used for:

[0127] According to the data to be extracted, determining the text data to be extracted, the image data to be extracted, and the video data to be extracted;

[0128] Perform data processing on the extracted text data to obtain the text data to be encoded. The data processing includes data cleaning, data segmentation and word embedding processing;

[0129] Perform feature processing on the image data to be extracted and the video data to be extracted to obtain the image data to be encoded and the video data to be encoded;

[0130] The text data to be encoded, the image data to be encoded and the video data to be encoded are encoded to obtain feature data of the data to be extracted.

[0131] Optionally, the data encoding processing module 902 is further specifically configured to:

[0132] Inputting the text data to be encoded into the text encoder in the multimodal encoder to obtain text encoded data;

[0133] Inputting the image data to be encoded and the video data to be encoded into a visual encoder in the multimodal encoder to obtain visual encoding data;

[0134] Through the fusion layer in the multimodal encoder, the text encoding data and the visual encoding data are fused to obtain the feature data of the data to be extracted;

[0135] Among them, the fusion layer in the multimodal encoder is a feature fusion layer obtained by training the neural network algorithm based on the attention mechanism.

[0136] Optionally, the to-be-extracted data acquisition module 901 is further specifically used for:

[0137] Obtain JSON data, storage path and data type information of JSON data from the source database;

[0138] The JSON data is indexed, and based on the storage path and data type information of the JSON data and the preset standard data information, a pre-acquired comparison model is used to detect changes in the path and value in the JSON data to determine the changed database;

[0139] Among them, the comparison model is a model pre-trained based on a deep learning algorithm for detecting the storage path of data and changes in values.

[0140] Optionally, the to-be-extracted data acquisition module 901 is further specifically used for:

[0141] Collect JSON data from data sources, including API interfaces, databases, and file systems;

[0142] Store JSON data in the source database as key-value pairs.

[0143] Optionally, the data extraction module 903 is further specifically used for:

[0144] According to the inherent information in the features obtained in advance by the decoder and the preset data extraction requirements, data extraction is performed on the feature data of the data to be extracted to obtain the extracted data.

[0145] Optionally, the data extraction device 90 is further specifically used for:

[0146] The extracted data is structured and stored in the extraction database using database management tools or programming languages.

[0147] Fig.10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.10 As shown, the electronic device 100 includes:

[0148] The electronic device 100 may include a processor 1001 with one or more processing cores, a memory 1002 with one or more computer-readable storage media, a communication component 1003 and other components. The processor 1001, the memory 1002 and the communication component 1003 are connected via a bus 1004.

[0149] In a specific implementation process, at least one processor 1001 executes computer-executable instructions stored in the memory 1002, so that at least one processor 1001 executes the above data extraction method.

[0150] The specific implementation process of the processor 1001 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0151] In the above Fig.10 In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0152] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0153] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0154] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple instructions are stored. The instructions can be loaded by a processor to execute the steps in any data extraction method provided in the embodiment of the present application.

[0155] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0156] Since the instructions stored in the storage medium can execute the steps in any data extraction method provided in the embodiments of the present application, the beneficial effects that can be achieved by any data extraction method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0157] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0158] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data extraction method, characterized in that: include: Acquire data to be extracted from a change database, wherein the change database stores change data obtained through change detection, wherein the change data includes at least two of text data, image data, and video data; Performing encoding processing on the data to be extracted to obtain feature data of the data to be extracted; Data extraction is performed on the characteristic data of the data to be extracted according to preset requirements to obtain extracted data.

2. The method according to claim 1, characterized in that The encoding process is performed on the data to be extracted to obtain feature data of the data to be extracted, including: Determine text data to be extracted, image data to be extracted, and video data to be extracted according to the data to be extracted; Performing data processing on the text data to be extracted to obtain text data to be encoded, wherein the data processing includes data cleaning, data segmentation and word embedding processing; Performing feature processing on the image data to be extracted and the video data to be extracted to obtain the image data to be encoded and the video data to be encoded; The text data to be encoded, the image data to be encoded and the video data to be encoded are encoded to obtain feature data of the data to be extracted.

3. The method according to claim 2, characterized in that The step of encoding the text data to be encoded, the image data to be encoded, and the video data to be encoded to obtain feature data of the data to be extracted includes: Inputting the text data to be encoded into a text encoder in a multimodal encoder to obtain text encoded data; Inputting the image data to be encoded and the video data to be encoded into a visual encoder in the multimodal encoder to obtain visual encoding data; The text encoding data and the visual encoding data are fused through a fusion layer in the multimodal encoder to obtain feature data of the data to be extracted; Among them, the fusion layer in the multimodal encoder is a feature fusion layer obtained by training the neural network algorithm based on the attention mechanism.

4. The method according to any one of claims 1 to 3, characterized in that: Before acquiring the data to be extracted from the change database, the method further includes: Acquire JSON data, storage path and data type information of the JSON data from a source database; The JSON data is indexed, and according to the storage path and data type information of the JSON data and the preset standard data information, a pre-acquired comparison model is used to perform change detection on the path and value in the JSON data to determine the changed database; The comparison model is a model pre-trained based on a deep learning algorithm and used to detect changes in the storage path and value of data.

5. The method according to claim 4, characterized in that Before acquiring the JSON data, the storage path and data type information of the JSON data from the source database, the method further includes: Collect JSON data from data sources, including API interfaces, databases, and file systems; The JSON data is stored in the source database in the form of key-value pairs.

6. The method according to any one of claims 1 to 3, characterized in that: The step of extracting the characteristic data of the data to be extracted according to preset requirements to obtain the extracted data includes: According to the inherent information in the features of the decoder obtained in advance and the preset data extraction requirements, data extraction is performed on the feature data of the data to be extracted to obtain the extracted data.

7. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: The extracted data is structured, and the structured extracted data is stored in an extraction database using a database management tool or a programming language.

8. A data extraction device, characterized in that: include: A to-be-extracted data acquisition module is used to acquire the to-be-extracted data from a change database, wherein the change database stores change data obtained through change detection, wherein the change data includes at least two of text data, image data, and video data; A data encoding processing module, used for encoding the data to be extracted to obtain feature data of the data to be extracted; The data extraction module is used to extract the characteristic data of the data to be extracted according to preset requirements to obtain the extracted data.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images