Multi-modal fusion management and control method and system applied to multi-source big data

Abstract

This application provides a multimodal fusion management and control method and system for multi-source big data, relating to the field of data processing and management technology. It acquires a multi-source big data set containing text, images, audio, and video types, each carrying a unique data source identifier and inherent feature information. Then, based on the data source identifier and inherent feature information, a cross-modal feature mapping table is constructed, and modal association chains are established to obtain a multimodal association chain set. Next, the association chain set is subjected to strength value calculation and sorting to mine management association points. Then, a management scenario feature library is constructed, and the management association points are matched with preset management scenarios to generate a multimodal management rule set. Finally, a management rule adaptation engine is built to adapt rules to different modal big data units in the multi-source big data set, generating multi-source big data fusion management and control instructions, thus achieving efficient fusion management and control of multi-source big data.

Description

Technical Field

[0001] This application relates to the field of data processing and management technology, and more specifically, to a multimodal fusion management method and system applied to multi-source big data. Background Technology

[0002] In today's digital age, data is experiencing explosive growth and comes from a wide range of sources, forming a multi-source big data landscape. This multi-source big data encompasses various data units such as text, images, audio, and video.

[0003] Currently, the processing of multi-source big data mostly involves independent analysis and management of single-modal data. For example, text data processing mainly focuses on keyword extraction and semantic analysis; image data processing focuses on pixel processing and contour recognition; audio data processing focuses on frequency analysis and tone recognition; and video data processing focuses on frame sequence analysis and dynamic trajectory tracking. However, these single-modal processing methods have obvious limitations and cannot fully utilize the inherent relationships between different modalities of data.

[0004] In real-world applications, data from different modalities are often interconnected and complementary, collectively reflecting a complete scenario or event. However, current technologies lack an effective method to integrate data from different modalities and conduct comprehensive management based on this integration. This makes it difficult to obtain comprehensive and accurate information when facing complex multi-source big data scenarios, hindering efficient and intelligent management and thus limiting the application value of multi-source big data. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a multimodal fusion management and control method and system applicable to multi-source big data.

[0006] According to a first aspect of this application, a multimodal fusion management and control method for multi-source big data is provided, the method comprising:

[0007] A multi-source big data set is obtained, which includes big data units of text type, image type, audio type, and video type. Each type of big data unit carries a unique data source identifier and inherent data feature information. The inherent feature information of text type big data units is keyword groups and semantic tendency information. The inherent feature information of image type big data units is pixel distribution information and contour feature information. The inherent feature information of audio type big data units is frequency distribution information and tone feature information. The inherent feature information of video type big data units is frame sequence feature information and dynamic trajectory feature information.

[0008] Based on the data source identifiers and inherent characteristics of big data units of different modal types, a cross-modal feature mapping table is constructed. Modal association chains between big data units of different modal types are established through the cross-modal feature mapping table to obtain a set of multimodal association chains.

[0009] The strength value of each association chain in the multimodal association chain set is calculated. The multimodal association chain set is sorted in descending order according to the strength value. The sliding window mining algorithm is used to mine the control association points in the sorted association chains to obtain the control association point set.

[0010] Construct a control scenario feature library, compare and match each control association point in the control association point set with the preset control scenarios in the control scenario feature library, and generate a multimodal control rule set based on the matching results;

[0011] A management rule adaptation engine is built. The multi-modal management rule set is input into the management rule adaptation engine. The management rule adaptation engine adapts the rules to different modal big data units in the multi-source big data set and generates multi-source big data fusion management instructions.

[0012] According to a second aspect of this application, a multimodal fusion management and control system for multi-source big data is provided. The multimodal fusion management and control system for multi-source big data includes a processor and a readable storage medium. The readable storage medium stores a program that, when executed by the processor, implements the aforementioned multimodal fusion management and control method for multi-source big data.

[0013] Based on any of the above aspects, by acquiring a multi-source big data set containing various types such as text, images, audio, and video, each carrying a unique data source identifier and inherent data characteristic information, a cross-modal feature mapping table is constructed based on the data source identifiers and inherent data characteristic information of big data units of different modal types. A multi-modal association chain set is then established, effectively uncovering the intrinsic connections between different modal data and achieving deep fusion of multi-modal data. The multi-modal association chain set is then subjected to strength value calculation and sorting, and a sliding window mining algorithm is used to mine control-related points, enabling precise location of key points with control value, improving the targeting and effectiveness of control. A control scenario feature library is constructed and its features are compared and matched with control-related points to generate a multi-modal control rule set, making the control rules more aligned with actual scenario needs. Finally, by building a control rule adaptation engine to adapt rules to different modal big data units in the multi-source big data set, multi-source big data fusion control instructions are generated, achieving efficient, intelligent, and comprehensive control of multi-source big data, greatly improving the control efficiency of multi-source big data. Attached Figure Description

[0014] Figure 1A flowchart illustrating the multimodal fusion management method for multi-source big data provided in this application embodiment is shown.

[0015] Figure 2 This paper illustrates a schematic diagram of the component structure of a multimodal fusion management and control system for multi-source big data provided in an embodiment of this application. Detailed Implementation

[0016] Figure 1 The diagram illustrates a flowchart of a multimodal fusion management and control method for multi-source big data provided in this application. It should be understood that in other embodiments, the order of some steps in the multimodal fusion management and control method for multi-source big data can be interchanged according to actual needs, or some steps can be omitted or deleted. The detailed steps of this multimodal fusion management and control method for multi-source big data are described below.

[0017] Step S110: Obtain a multi-source big data set, which includes big data units of text type, image type, audio type, and video type. Each type of big data unit carries a unique data source identifier and inherent data feature information. The inherent feature information of the text type big data unit is keyword groups and semantic tendency information. The inherent feature information of the image type big data unit is pixel distribution information and contour feature information. The inherent feature information of the audio type big data unit is frequency distribution information and tone feature information. The inherent feature information of the video type big data unit is frame sequence feature information and dynamic trajectory feature information.

[0018] In this embodiment, taking the data collection and correlation analysis scenario of different departments within an enterprise as an example, for departments such as marketing, R&D, production, and finance, the text-type big data unit can be market research reports from the marketing department, technical documents from the R&D department, production logs from the production department, and financial statements from the finance department. Keyword groups are extracted from the text to include terms related to the department's business, such as "market share," "competitors," and "consumer demand" in the market research report. Semantic bias information is analyzed using natural language processing technology, such as whether the market research report as a whole leans towards market opportunities or challenges. The image-type big data unit can be product design drawings from the R&D department, production equipment diagrams from the production department, and product promotional posters from the marketing department. Pixel distribution information is obtained by analyzing the pixel matrix of the image to determine the pixel value distribution in different areas, and contour features are also analyzed. Feature information is extracted through edge detection algorithms, such as the outline shape of a product in a product design drawing; audio-type big data units can include customer interview recordings from the marketing department, technical discussion recordings from the R&D department, and equipment operation sound recordings from the production department, etc. Frequency distribution information is obtained by processing the audio signal through Fourier transform and other methods to obtain the distribution of different frequency components, and tone feature information is obtained by analyzing the tone change patterns of the audio, such as the tone features of customers in customer interview recordings; video-type big data units can include production process videos from the production department, product demonstration videos from the marketing department, and experimental process videos from the R&D department, etc. Frame sequence feature information is obtained by analyzing the feature changes between frames through each frame image of the video, and dynamic trajectory feature information is obtained by tracking and analyzing the trajectory of moving objects in the video, such as the movement trajectory of products on the production line in production process videos.

[0019] Each type of big data unit carries a unique data source identifier, which includes information such as department identifier and data type identifier. For example, the data source identifier for market research reports from the marketing department is "Market_Report_Text", and the data source identifier for product design drawings from the R&D department is "RND_Design_Image".

[0020] Step S120: Based on the data source identifiers and inherent data feature information of big data units of different modal types, construct a cross-modal feature mapping table, establish modal association chains between big data units of different modal types through the cross-modal feature mapping table, and obtain a set of multimodal association chains.

[0021] Step S121: Classify the multi-source big data set according to the data source identifier to obtain a subset of text data sources, a subset of image data sources, a subset of audio data sources, and a subset of video data sources. Each subset of data sources contains all big data units under the corresponding type.

[0022] In the aforementioned enterprise department data scenarios, based on the data source identifier, all text-type big data units are categorized into a text data source subset. For example, market research reports from the marketing department, technical documents from the R&D department, production logs from the production department, and financial statements from the finance department are categorized into a text data source subset. Similarly, all image-type big data units are categorized into an image data source subset. For example, product design drawings from the R&D department, production equipment diagrams from the production department, and product promotional posters from the marketing department are categorized into an image data source subset. All audio-type big data units are categorized into an audio data source subset. For example, customer interview recordings from the marketing department, technical discussion recordings from the R&D department, and equipment operation sound recordings from the production department are categorized into an audio data source subset. Finally, all video-type big data units are categorized into a video data source subset. For example, production process videos from the production department, product demonstration videos from the marketing department, and experimental process videos from the R&D department are categorized into a video data source subset. Each data source subset contains all big data units of the corresponding type. For instance, the text data source subset contains all text-type big data units from the marketing, R&D, production, and finance departments, along with their corresponding data source identifiers and inherent characteristic information.

[0023] Step S122: Construct a vocabulary comparison matrix. When the core feature vocabulary groups of the text, image, audio, and video data source subsets are filled into the rows and columns of the matrix respectively, the core feature vocabulary groups corresponding to each data source subset are formed. Calculate the similarity value between the core feature vocabulary groups of different data source subsets. When the similarity value exceeds the preset value, it is determined that the corresponding data source subsets have common feature associations and are marked as related data source combinations.

[0024] In this scenario, for a subset of text data sources, core feature vocabulary groups are extracted. For example, the core feature vocabulary groups for a market research report from the marketing department might include "market share," "competitors," and "consumer demand"; for technical documents from the R&D department, they might include "technical parameters," "innovation points," and "R&D progress"; for production logs from the production department, they might include "production quantity," "equipment failure," and "production efficiency"; and for financial statements from the finance department, they might include "revenue," "cost," and "profit." For a subset of image data sources, core feature vocabulary groups are extracted. For example, the core feature vocabulary groups for product design drawings from the R&D department might include "product structure," "size specifications," and "material type"; for production equipment diagrams from the production department, they might include "equipment model," "component name," and "connection method"; and for product promotional posters from the marketing department, they might include "product..." For audio data sources, extract core feature vocabulary such as "product appearance," "promotional slogans," and "color scheme." For audio data sources, extract core feature vocabulary such as "customer feedback," "pain points," and "satisfaction," for example, customer interview recordings from the marketing department might include "customer feedback," "pain points," and "satisfaction," while technical discussion recordings from the R&D department might include "technical challenges," "solutions," and "patent applications," and equipment operation sound recordings from the production department might include "abnormal equipment noise," "operating status," and "maintenance cycle." For video data sources, extract core feature vocabulary such as "production process," "product quality," and "personnel operation," while product demonstration videos from the marketing department might include "product functions," "usage scenarios," and "advantage demonstrations," and experimental process videos from the R&D department might include "experimental steps," "experimental data," and "experimental conclusions." Then, a vocabulary comparison matrix is ​​constructed, and the aforementioned core feature vocabulary groups are filled into the rows and columns of the matrix respectively. The similarity value between the core feature vocabulary groups of different data source subsets is calculated. For example, the similarity value between the core feature vocabulary group of the market research report of the marketing department in the text data source subset and the core feature vocabulary group of the product design drawing of the R&D department in the image data source subset is calculated by counting the proportion of the number of identical words in the two core feature vocabulary groups to the total number of words (if there are semantically similar words, semantic similarity algorithms can be used to treat semantically similar words as identical words for counting). When the similarity value exceeds the preset value, it is determined that the two data source subsets have common feature association and are marked as related data source combinations.

[0025] Step S123: Using each subset of data sources in the associated data source combination as the starting node, the calculated similarity value is used as the connection weight between nodes, and common feature words are used as connection labels. A graph structure construction algorithm is used to construct the connection relationship between different subsets of data sources to form an initial association chain. The structure of the initial association chain includes the starting node identifier, the ending node identifier, the connection weight, and the connection label.

[0026] In the above scenario, assume that the market research report data subset from the marketing department in the text data subset and the product design drawing data subset from the R&D department in the image data subset constitute a related data source combination. The market research report data subset from the marketing department is taken as the starting node, and the product design drawing data subset from the R&D department is taken as the ending node. The calculated similarity value is used as the connection weight between these two nodes, and common feature words (such as common feature words related to product functional requirements in the market research report and product structure and functions in the product design drawing) are used as connection labels. A graph structure construction algorithm (such as a graph construction algorithm based on adjacency matrices, where node identifiers are used as rows and columns of the matrix, connection weights are used as values ​​of matrix elements, and common feature words are used as annotations of matrix elements) is used to construct the connection relationship between these two data source subsets, forming an initial association chain. The structure of this initial association chain includes the starting node identifier (the identifier of the market research report data subset from the marketing department, "Market_Report_Text"), the ending node identifier (the identifier of the product design drawing data subset from the R&D department, "RND_Design_Image"), the connection weight (the calculated similarity value), and the connection label (common feature words).

[0027] Step S124: Use the chain structure deduplication algorithm to traverse all initial association chains. When two initial association chains contain the same big data unit identifier, the connection weights of the two chains are weighted and summed. The common feature words in the connection tags are merged, and the connection tags of the merged chain structure are retained with different feature words. All data source identifiers and complete core feature word information of the merged chain structure are supplemented. This process is repeated until there are initial association chains without duplicate big data units, and a multimodal association chain set is obtained.

[0028] In this scenario, all initial association chains are traversed. Assuming two initial association chains both contain the identifier of a market research report big data unit, the connection weights of these two chains are weighted and summed (the weights can be determined based on factors such as the credibility of the two chains; for example, the chain with higher credibility has a larger weight. Credibility can be assessed through factors such as the reliability of the data source and the integrity of the data). Common feature words in the connection tags are merged. For example, if both chains contain "product function" in their connection tags, only one "product function" is retained. For different feature words, such as "market share" in one chain and "technical parameters" in another, both are retained, forming the connection tags of the merged chain structure. Simultaneously, all data source identifiers in the merged chain structure are supplemented (e.g., the identifiers of the market research report data source subset from the marketing department, the product design data source subset from the R&D department, and possibly the production process video data source subset from the production department) and complete core feature word information (e.g., the merged core feature words include the core feature words of both chains). This process is repeated until no initial association chains containing the same big data unit identifier remain, resulting in a multimodal association chain set.

[0029] Step S130: Calculate the strength value of each association chain in the multimodal association chain set, sort the multimodal association chain set in descending order according to the strength value, and use the sliding window mining algorithm to mine the control association points in the sorted association chains to obtain the control association point set.

[0030] Step S131: Traverse each association chain in the multimodal association chain set, extract common feature words from the chain structure connection labels, count the actual number of common feature words in each association chain, use the actual number as the initial strength value of the association chain, and record the vocabulary statistics list corresponding to the initial strength value of each association chain.

[0031] In the above-mentioned enterprise department data scenario, each association chain in the multimodal association chain set is traversed. For example, if the common feature words in the connection tags of a certain association chain are "product function", "market demand" and "technical parameters", then the actual number of common feature words in the association chain is counted, and this number is used as the initial strength value of the association chain. At the same time, the vocabulary statistics list corresponding to the initial strength value is recorded, that is, the words "product function", "market demand" and "technical parameters" are recorded.

[0032] Step S132: Construct a data update log table to record the update timestamps of the big data units contained in each association chain in the multimodal association chain set. Calculate the number of updates per unit time as the update frequency. Normalize the update frequency of all association chains to obtain the normalized update frequency value. Establish a mapping relationship table between the normalized update frequency value and the strength adjustment coefficient. The normalized update frequency value and the adjustment coefficient are positively correlated. Determine the adjustment coefficient of each association chain according to the mapping relationship table. Multiply the initial strength value by the adjustment coefficient to obtain the final strength value of each association chain. Generate a strength statistics report containing the chain identifier, initial strength value, adjustment coefficient, and final strength value.

[0033] In this scenario, a data update log table is constructed to record the update timestamps of the big data units contained in each related chain. For example, the update timestamps of the market research report big data unit contained in a related chain within a certain time period are recorded. Then, the number of updates per unit time is calculated as the update frequency. For example, if the market research report big data unit is updated a certain number of times in a month, the update frequency is the number of updates divided by the number of days in a month. The update frequencies of all related chains are normalized by scaling the update frequencies of all related chains according to the minimum and maximum values, so that the normalized update frequency value is between 0 and 1, specifically (update frequency - minimum update frequency) / (maximum update frequency - minimum update frequency). A mapping table is established between the normalized update frequency value and the intensity adjustment coefficient. For example, when the normalized update frequency value is 0.3, the intensity adjustment coefficient is 1.2; when the normalized update frequency value is 0.6, the intensity adjustment coefficient is 1.5, and so on. The larger the normalized update frequency value, the larger the adjustment coefficient. The adjustment coefficient for each associated chain is determined based on the mapping table. Then, the initial strength value is multiplied by the adjustment coefficient to obtain the final strength value of each associated chain. For example, if the initial strength value of an associated chain is 3 and the adjustment coefficient is 1.3, the final strength value is 3 × 1.3. A strength statistics report containing the chain identifier, initial strength value, adjustment coefficient, and final strength value is generated.

[0034] Step S133: Sort the multimodal association chain set in descending order of final strength value to obtain an ordered multimodal association chain set.

[0035] In the above scenario, each association chain in the multimodal association chain set is sorted in descending order of its final strength value. For example, if one association chain has a final strength value of 5, another has 4, and yet another has 3.9, then after sorting, the association chain with a final strength value of 5 is placed first, followed by the one with 4, and then the one with 3.9, resulting in an ordered multimodal association chain set.

[0036] Step S134: Set the selection ratio parameter, calculate the specific number of target association chains to be selected based on the selection ratio parameter and the total number of ordered multimodal association chain sets, select the corresponding number of association chains as target association chains starting from the starting position of the ordered multimodal association chain set, and use a chain structure parsing tool to parse the node information, connection weight and connection label of each target association chain one by one, extract the core feature words of each association chain, and record the node position and occurrence frequency of each core feature word in the association chain.

[0037] Step S1341: Traverse the ordered multimodal association chain set, use a counter to count the total number of association chains, set the selection ratio parameter, the selection ratio parameter is determined according to the control accuracy requirements through the scene accuracy level in the control scene feature library, and calculate the number of target association chains to be selected by multiplying the total number by the selection ratio parameter. If the calculation result is a decimal, round it up.

[0038] In this scenario, the ordered multimodal association chain set is traversed, and the total number of association chains is counted using a counter. A selection ratio parameter is set. Assuming that the control accuracy requirement is high, the selection ratio parameter is determined to be 0.4 based on the scene accuracy level in the control scene feature library. Then, the number of target association chains to be selected is calculated by multiplying the total number by the selection ratio parameter. If the calculation result is a decimal, it is rounded up.

[0039] Step S1342: Starting from the first association chain in the ordered multimodal association chain set, select one association chain at a time until the required number of association chains are reached. Mark the selected association chains as target association chains and generate a target association chain list. The target association chain list includes the association chain identifier, the final strength value, and the association chain structure information.

[0040] In the above scenario, selection begins with the first association chain in the ordered multimodal association chain set, and continues until the required number of association chains are reached. The selected association chains are marked as target association chains, and a target association chain list is generated. This target association chain list contains the association chain identifier, final strength value, and association chain structure information for each target association chain. For example, the association chain identifier of a target association chain is CL001, the final strength value is 5, and the association chain structure information includes the start node identifier, end node identifier, connection weight, connection label, etc.

[0041] Step S1343: Use the association chain structure parsing program to load the structural information of each target association chain, parse out the identifiers of the starting node, intermediate node and ending node in the association chain, extract the core feature vocabulary group corresponding to each node, record the node identifier to which each core feature vocabulary belongs and its position in the core feature vocabulary group, and form a vocabulary-node mapping table.

[0042] In this scenario, a relational chain structure parsing program is used to load the structural information of each target relational chain. For example, the structural information of a target relational chain includes a starting node identifier M1 (the data source subset identifier of the market research report from the marketing department), an intermediate node identifier R1 (the data source subset identifier of the product design drawings from the R&D department), and an ending node identifier P1 (the data source subset identifier of the production process video from the production department). The core feature vocabulary group corresponding to each node is extracted. The core feature vocabulary group corresponding to M1 is "market share", "consumer demand", and "product function". The core feature vocabulary group corresponding to R1 is "technical parameters", "product structure", and "innovation points". The core feature vocabulary group corresponding to P1 is "production process", "product quality", and "equipment operation". Then, the node identifier to which each core feature vocabulary belongs and its position in the core feature vocabulary group are recorded. For example, the node identifier to which "market share" belongs is M1, and its position in the core feature vocabulary group is 1; the node identifier to which "technical parameters" belongs is R1, and its position in the core feature vocabulary group is 1, etc., forming a vocabulary-node mapping table.

[0043] Step S1344: Construct a statistical table of the distribution of core feature words. The rows of the statistical table are used as word names and the columns are used as node identifiers. Traverse the word-node mapping table and record the number of occurrences in the intersection cells of the corresponding words and nodes. If the same word appears multiple times in the same node, the number of occurrences is accumulated. At the same time, add a function column for total occurrences to the statistical table to record the total number of occurrences of each word in all nodes.

[0044] In the above scenario, a statistical table of core feature word distribution is constructed. The rows of the table represent word names, and the columns represent node identifiers (e.g., M1, R1, P1, etc.). The word-node mapping table is traversed, and the frequency of occurrence is recorded in the cell where the corresponding word and node intersect. For example, if "market share" appears once in node M1, then 1 is recorded in the cell where column M1 intersects with the row "market share"; if "technical parameters" appears once in node R1, then 1 is recorded in the cell where column R1 intersects with the row "technical parameters". If the same word appears multiple times in the same node, the count is accumulated. For example, if "product function" appears twice in node M1, then 2 is recorded. Simultaneously, a column for total occurrences is added to the statistical table to record the total number of occurrences of each word across all nodes. For example, the total number of occurrences for "market share" is 1, and the total number of occurrences for "technical parameters" is 1, etc.

[0045] Step S1345: Sort the core feature words according to their frequency of occurrence, mark the core feature words with the highest frequency of occurrence as key feature words, and improve the distribution statistics table.

[0046] In this scenario, core feature words are sorted according to their frequency of occurrence. For example, "product function" appears twice, "market share" appears once, and "technical parameters" appears once. The core feature words with the highest frequency of occurrence (such as "product function" which appears most frequently) are marked as key feature words. The distribution statistics table is improved, and key feature words are highlighted or listed separately.

[0047] Step S135: Construct a word location heatmap, using the node positions of all target association chains as the horizontal axis and the core feature words as the vertical axis, with the frequency of word occurrence as the heat value. After generating the heatmap, mark the position with the highest heat value as the core overlapping position, extract the node identifier corresponding to the core overlapping position, query the corresponding data source identifier based on the node identifier, retrieve the inherent feature information of the big data unit under the data source identifier, and combine the core feature words of the core overlapping position to determine the control association point. Each control association point includes a location identifier, a data source identifier, inherent feature information, and core feature words.

[0048] In the above scenario, a word location heatmap is constructed. The positions of all target association chains (such as the positions of the starting node, intermediate node, and ending node) are used as the horizontal axis, and the core feature words are used as the vertical axis. The frequency of word occurrence is used as the heat value. After generating the heatmap, assuming that the node corresponding to the position with the highest heat value is identified as M1 (the data source subset identifier of the market research report from the marketing department), and the core feature word is "product function", the corresponding data source identifier is queried based on the node identifier M1 (for example, the identifier of the data source subset of the market research report from the marketing department is DS_M_Text). The inherent feature information of the big data unit under the data source identifier is retrieved (for example, the keyword group "market share", "consumer demand", "product function" and semantic tendency information "market opportunity" in the market research report). Combined with the core feature word "product function" of the core overlapping position, the combination of the position identifier (the identifier of the core overlapping position), the data source identifier DS_M_Text, the inherent feature information (keyword group and semantic tendency information) and the core feature word "product function" is determined as the control association point. This control association point includes the position identifier, the data source identifier, the inherent feature information and the core feature word.

[0049] Step S136: Use the deduplication algorithm for control association points, with the data source identifier and core feature words as the joint deduplication key, traverse all determined control association points, delete duplicate control association points, organize the attribute information of the deduplicated control association points, and supplement the generation time and the association chain identifier of each control association point to form a set of control association points.

[0050] Step S1361: Extract the attribute information of each determined control association point. The attribute information includes location identifier, data source identifier, inherent feature information and core feature words. Use the location identifier, data source identifier, inherent feature information and core feature words as the basis for deduplication judgment.

[0051] In this scenario, the attribute information of each specific control-related point is extracted. For example, the location identifier of a certain control-related point is P001, the data source identifier is DS_M_Text, the inherent feature information is the keyword group "market share", "consumer demand" and "product function" and the semantic tendency information "market opportunity", and the core feature word is "product function". The above information is used as the basis for deduplication judgment.

[0052] Step S1362: Construct a deduplication keyword group, and select the data source identifier and core feature words as joint deduplication keywords.

[0053] In the above scenario, a deduplication keyword group is constructed, and the data source identifier (such as DS_M_Text) and core feature words (such as "product function") are selected as joint deduplication keywords.

[0054] Step S1363: Create a temporary storage list for deduplication, initialize the temporary storage list for deduplication to be empty, traverse all determined control association points, perform hash calculation on the joint deduplication key of each control association point, and obtain the key hash value.

[0055] In this scenario, a temporary deduplication storage list is created and initialized to empty. All identified control association points are traversed, and the combined deduplication keyword (data source identifier and core feature vocabulary) of each control association point is hashed, for example, using the SHA-256 algorithm, to obtain the keyword hash value.

[0056] Step S1364: Query whether there is a duplicate keyword hash value in the deduplication temporary storage list. If not, store the attribute information and keyword hash value of the control association point into the deduplication temporary storage list. If it exists, skip the control association point and complete the initial deduplication.

[0057] In the above scenario, the system checks whether there are identical keyword hash values ​​in the deduplication temporary storage list. If the keyword hash value of a certain control association point does not exist in the deduplication temporary storage list, the attribute information and keyword hash value of the control association point are stored in the deduplication temporary storage list. If it exists, the control association point is skipped, and the initial deduplication is completed.

[0058] Step S1365: Verify the attribute information of the control association points after initial deduplication, verify the correspondence between the location identifier and the data source identifier, and verify the correspondence between the inherent feature information and the core feature words. If there is a mismatch, return to redetermine the control association points. After verification, add the generation time and the association chain identifier of each control association point. Sort all the sorted control association points according to the data source identifier to obtain the control association point set.

[0059] In this scenario, the attribute information of the control association points after initial deduplication is checked. For example, it is checked whether the location identifier P001 corresponds to the data source identifier DS_M_Text (because the location information of the big data unit under the data source identifier should match the location identifier). It is also checked whether the keyword group in the inherent feature information contains the core feature word "product function" (because the core feature word should be extracted from the keyword group of the inherent feature information). If there is a mismatch, the control association points are re-determined. After verification, the generation time (such as the current time) and the association chain identifier (such as CL001) of each control association point are added. All the sorted control association points are sorted by the data source identifier to obtain the control association point set.

[0060] Step S140: Construct a control scenario feature library, compare and match the features of each control association point in the control association point set with the preset control scenarios in the control scenario feature library, and generate a multimodal control rule set based on the matching results.

[0061] Step S141: Construct a control scenario feature library. The control scenario feature library includes data storage control scenarios, data transmission control scenarios, data usage control scenarios, and data destruction control scenarios. Each control scenario has preset scenario feature terms and corresponding scenario control requirements. The feature terms for data storage control scenarios are storage period, storage medium, and backup frequency. The control requirements are to specify the storage medium type and backup execution method. The feature terms for data transmission control scenarios are transmission protocol, encryption method, and transmission rate. The control requirements are to specify the transmission protocol and encryption algorithm. The feature terms for data usage control scenarios are usage permissions, access frequency, and operation logs. The control requirements are to set permission levels and log recording ranges. The feature terms for data destruction control scenarios are destruction methods, residual detection, and destruction logs. The control requirements are to specify the destruction methods and residual detection standards.

[0062] In the aforementioned enterprise department data scenarios, a management and control scenario feature library is constructed. The scenario feature keywords for data storage management and control scenarios include storage period (e.g., short-term, long-term), storage medium (e.g., local hard drive, cloud storage), and backup frequency (e.g., daily, weekly). Management requirements specify the storage medium type (e.g., requiring cloud storage) and backup execution method (e.g., performing backups daily at midnight). The scenario feature keywords for data transmission management and control scenarios include transmission protocol (e.g., HTTP, HTTPS), encryption method (e.g., AES, RSA), and transmission rate (e.g., high-speed, low-speed). Management requirements specify the transmission protocol (e.g., requiring HTTPS) and encryption algorithm (e.g., requiring AES). Data usage management... The key features of the control scenario are access permissions (e.g., departmental permissions, cross-departmental permissions), access frequency (e.g., high frequency, low frequency), and operation logs (e.g., recording the operator and operation time). The control requirements are to set permission levels (e.g., ordinary employees can only view, while department managers can modify) and the scope of log recording (e.g., recording all operation content). The key features of the data destruction control scenario are destruction methods (e.g., physical destruction, logical destruction), residual detection (e.g., detecting whether the amount of residual data is less than a certain threshold), and destruction logs (e.g., recording the destroyer and destruction time). The control requirements are to specify the destruction method (e.g., requiring the use of physical destruction) and residual detection standards (e.g., requiring the amount of residual data to be less than a certain percentage).

[0063] Step S142: Construct a feature comparison vector space. Convert the core feature words of each control-related point in the control-related point set into feature vectors. At the same time, convert the scene feature words of each scene in the control-related scene feature library into feature vectors. Use the Euclidean distance algorithm to calculate the distance between the feature vector of each control-related point and the feature vector of each scene. The smaller the distance value, the higher the matching degree. Count the number of feature words whose distance value is less than the preset distance threshold when each control-related point matches each scene, and use this as the feature word matching number.

[0064] In this scenario, a feature comparison vector space is constructed. The core feature words (e.g., "product function") of each control-related point in the control-related point set are transformed into feature vectors. This transformation involves mapping the core feature words into a high-dimensional vector space, where each dimension corresponds to the semantic representation of a word (this can be achieved through word embedding techniques, such as Word2Vec, which converts words into fixed-length vectors). Simultaneously, the scenario feature words of each scenario in the control-related scenario feature library (e.g., "storage cycle," "storage medium," and "backup frequency" in the data storage control scenario) are also transformed into feature vectors. The Euclidean distance algorithm is used to calculate the distance between the feature vector of each control-related point and the feature vectors of each scenario. The calculation method is the square root of the sum of the squares of the differences between the corresponding dimensions of the two vectors; a smaller distance value indicates a higher matching degree. The number of feature words whose distance value is less than a preset distance threshold (e.g., 0.5) when each control-related point matches each scenario is counted. This number is used as the feature word matching count. For example, when a control-related point matches the data storage control scenario, if the feature words with a distance value less than 0.5 are "storage cycle" and "backup frequency," then the feature word matching count is 2.

[0065] Step S143: Mark the preset control scenarios where the number of matching feature words reaches a preset threshold as the target control scenario corresponding to the control association point.

[0066] In the above scenario, the preset threshold is 2. When the number of feature words matched by a certain control-related point is 2, the data storage control scenario is marked as the target control scenario corresponding to the control-related point.

[0067] Step S144: Extract the scenario control requirements of the target control scenario, parse the constraints and execution actions in the scenario control requirements, determine the specific source of the control object based on the data source identifier of the control association point, and adjust the execution actions in combination with the inherent characteristic information of the control association point to form an adapted control requirement.

[0068] Step S1441: Use a rule parser to perform structured parsing of the scenario control requirements of the target control scenario, and split it into three core element fields: control object field, control action field, and control scope field. The control object field records the type and source of the data to be controlled, the control action field records the specific operation type to be performed, and the control scope field records the data source range covered by the control.

[0069] In this scenario, a rule parser is used to perform structured parsing of the scenario control requirements for data storage control, and to break down the data into three core elements: the control object field (which records that the type of data to be controlled is text and the source is a subset of market research reports from the marketing department), the control action field (which records that the specific operation type is storage operation, including specifying storage media and backup), and the control scope field (which records that the data source covered by the control is a subset of market research reports from the marketing department).

[0070] Step S1442: Query the data source information corresponding to the data source identifier of the control association point, determine the big data unit type under the data source, label the encoding format information for text type, label the resolution information for image type, label the sampling rate information for audio type, and label the frame rate information for video type. Fill the above information into the control object field and replace the original control object field content.

[0071] In the above scenario, query the data source information corresponding to the data source identifier (DS_M_Text) of the control association point, determine that the big data unit type under this data source is text type, and the encoding format information is UTF-8. Fill the above information into the control object field, and replace the data type and source information to be controlled recorded in the original control object field. The new control object field content is text type, source is a subset of market research report data sources from the marketing department, and the encoding format is UTF-8.

[0072] Step S1443: Extract the update frequency parameter from the inherent feature information of the control-related points. If the update frequency parameter is greater than the preset update threshold, it is determined to be a real-time update type. The operation type in the control action field is adjusted to real-time monitoring operation, which includes data inflow monitoring, operation behavior monitoring, and abnormal data capture. If the update frequency parameter is less than or equal to the preset update threshold, it is determined to be a static storage type. The operation type in the control action field is adjusted to periodic verification operation, which includes data integrity verification, storage status verification, and redundant data cleanup.

[0073] In this scenario, the update frequency parameter is extracted from the inherent feature information of the control-related points. If the update frequency parameter is greater than the preset update threshold (e.g., the preset update threshold is once a week, and the update frequency parameter is once a day), it is determined to be a real-time update type. The operation type in the control action field is then adjusted to real-time monitoring operation. Real-time monitoring operation includes data inflow monitoring (monitoring whether new market research report data is flowing in), operation behavior monitoring (monitoring operation behavior on market research report data, such as modification, deletion, etc.), and abnormal data capture (capturing market research report data that does not meet the requirements, such as data with incorrect format).

[0074] Step S1444: Query the node information of the associated chain where the control and management associated point is located, extract the data source identifiers corresponding to all nodes, form a data source identifier list, fill the data source identifier list formed by extracting the data source identifiers corresponding to all nodes into the control scope field, replace the content in the original control scope field, and record all data sources covered by control.

[0075] In the above scenario, query the node information of the associated chain where the control link is located, and extract the data source identifiers corresponding to all nodes. For example, if the node identifiers of the associated chain are M1 (a subset of market research report data sources from the marketing department), R1 (a subset of product design drawing data sources from the R&D department), and P1 (a subset of production process video data sources from the production department), the corresponding data source identifiers are DS_M_Text, DS_R_Image, and DS_P_Video, respectively, forming a data source identifier list [DS_M_Text, DS_R_Image, DS_P_Video]. Fill this data source identifier list into the control scope field, replacing the content in the original control scope field. The new control scope field content is that the data sources covered by the control are DS_M_Text, DS_R_Image, and DS_P_Video.

[0076] Step S1445: Integrate the updated control object field, the adjusted control action field, and the new control scope field. Determine the execution order based on the type of control action. Real-time monitoring operations are arranged in the order of data inflow monitoring, operation behavior monitoring, and abnormal data capture. Periodic verification operations are arranged in the order of data integrity verification, storage status verification, and redundant data cleanup. Supplement the execution time interval parameter to form an adapted control requirement that includes field information, execution order, and time interval.

[0077] In this scenario, the updated control object fields (text type, source: a subset of market research reports from the marketing department, encoding format: UTF-8), the adjusted control action fields (real-time monitoring operations, including data inflow monitoring, operational behavior monitoring, and abnormal data capture), and the new control scope fields (control coverage data sources: DS_M_Text, DS_R_Image, DS_P_Video) are integrated. Based on the type of control action (real-time monitoring operation), the execution order is determined as data inflow monitoring, operational behavior monitoring, and abnormal data capture. The additional execution time interval parameter is set to execute data inflow monitoring once per hour, operational behavior monitoring once every half hour, and abnormal data capture once per minute, forming an adapted control requirement that includes field information, execution order, and time interval.

[0078] Step S145: Associate and bind the location identifier and data source identifier of the control-related point with the scene identifier of the target control scenario, the constraints and execution actions that adapt to the control requirements, and generate a rule identifier. The rule identifier is composed of the location identifier and the scene identifier, forming a single multimodal control rule that includes the rule identifier, control object information, constraints, and execution actions.

[0079] In the above scenario, the location identifier P001 of the control-related point, the data source identifier DS_M_Text, the scenario identifier SC001 of the target control scenario (data storage control scenario), the constraints that adapt to the control requirements (such as the encoding format being UTF-8), and the execution actions (real-time monitoring operations, including data inflow monitoring, operation behavior monitoring, and abnormal data capture) are associated and bound to generate a rule identifier. The rule identifier is composed of the location identifier P001 and the scenario identifier SC001, namely P001_SC001, forming a single multimodal control rule that includes the rule identifier P001_SC001, the control object information (text type, source is a subset of market research report data sources from the marketing department, encoding format being UTF-8), the constraints (encoding format being UTF-8), and the execution actions (real-time monitoring operations, including data inflow monitoring, operation behavior monitoring, and abnormal data capture).

[0080] Step S146: Classify all single multimodal control rules according to the scenario identifier in the rule identifier, group the rules with the same scenario identifier into a group to form a data storage control rule group, a data transmission control rule group, a data usage control rule group, and a data destruction control rule group. Sort the rules in each group according to the position identifier of the rule identifier, and supplement the scope of application description of each group of rules to obtain a multimodal control rule set.

[0081] In the above scenario, all individual multimodal control rules are categorized according to the scenario identifier in the rule identifier. For example, rules with identifiers P001_SC001 and P002_SC001 are grouped into the data storage control rule group (scenario identifier SC001), and rules with identifiers P003_SC002 and P004_SC002 are grouped into the data transmission control rule group (scenario identifier SC002), thus forming data storage control rule groups, data transmission control rule groups, data usage control rule groups, and data destruction control rule groups. Each group of rules is then sorted according to the position identifier of the rule identifier. For example, rules in the data storage control rule group are sorted in the order of position identifiers P001 and P002. A scope of application description is added for each group of rules. For example, the scope of application description for the data storage control rule group is applicable to the data storage control of business data of text, images, videos, etc., in departments such as the enterprise marketing department, R&D department, and production department, resulting in a multimodal control rule set.

[0082] Step S150: Build a control rule adaptation engine, input the multimodal control rule set into the control rule adaptation engine, and use the control rule adaptation engine to adapt the rules of different modal big data units in the multi-source big data set to generate multi-source big data fusion control instructions.

[0083] Step S151: Read the data source identifier of each big data unit in the multi-source big data set, construct a data source-data unit mapping table, and group big data units with the same data source identifier according to the mapping table to obtain the text data source to be managed group, image data source to be managed group, audio data source to be managed group, and video data source to be managed group. Each to be managed group contains all big data units under the data source and their corresponding inherent feature information.

[0084] In the aforementioned enterprise department data scenario, the data source identifier of each big data unit in the multi-source big data set is read. For example, the data source identifier of the market research report big data unit in the marketing department is DS_M_Text, and the data source identifier of the product design drawing big data unit in the R&D department is DS_R_Image, etc. A data source-data unit mapping table is constructed, which records the list of big data units corresponding to each data source identifier. Based on the mapping table, big data units with the same data source identifier are grouped together to obtain the text data source to be managed group (including all big data units with data source identifiers such as DS_M_Text, DS_F_Text (financial statement data source identifier in the finance department) and their corresponding inherent feature information, such as keyword groups and semantic tendency information of market research reports) and the image data source to be managed group (including all big data units with data source identifiers such as DS_R_Image, DS_M_Image (product promotional poster data source identifier in the marketing department) and their corresponding inherent feature information, such as pixel distribution information and contour feature information of product design drawings). The audio data source to be managed group (including all data source identifiers such as DS_M_Audio (customer interview recording data source identifier of the marketing department) and DS_R_Audio (technical discussion recording data source identifier of the R&D department) and their corresponding inherent characteristic information, such as frequency distribution information and tone characteristic information of customer interview recordings) and the video data source to be managed group (including all data source identifiers such as DS_P_Video and DS_M_Video (product demonstration video data source identifier of the marketing department) and their corresponding inherent characteristic information, such as frame sequence characteristic information and dynamic trajectory characteristic information of production process videos).

[0085] Step S152: Construct a rule matching index. Using the data source type and data features as index keys, classify and store each rule in the multimodal management rule set according to the index key. Traverse each data unit group to be managed, extract the data source type and core data features of the data unit group to be managed, and query the matching rules according to the index key. If the data unit group to be managed is text type, match the rule containing the text feature index key; if it is image type, match the rule containing the image feature index key, and so on, to obtain the exclusive management rules for each data unit group to be managed.

[0086] In this scenario, a rule matching index is constructed, using data source type (text, image, audio, video) and data features (such as keyword groups in text, pixel distribution information in images, etc.) as index keys. Each rule in the multimodal management rule set is categorized and stored according to its index key. For example, rules containing text feature index keys are stored in the text rule subset, and rules containing image feature index keys are stored in the image rule subset, and so on. Each data unit group to be managed is traversed, and the data source type and core data features of the data unit group to be managed are extracted. For example, the data source type of the text data source group to be managed is text, and the core data features are keyword groups "market share," "consumer demand," "product function," and semantic tendency information "market opportunity." Based on the index key (text type and the above core data features), the matching rules are queried to obtain the exclusive management rules for the text data source group to be managed (such as the rule identified as P001_SC001).

[0087] Step S153: Parse the control action field and control scope field of the exclusive control rule, determine the execution node type according to the type of control action, assign real-time monitoring actions to real-time processing nodes, assign periodic verification actions to timed processing nodes, and record the node identifier of the execution node; extract the specific execution content from the control action field, extract the keyword filtering list and encoding verification standard for text type, extract the sensitive area coordinates and resolution standard for image type, extract the voiceprint feature library and sampling rate standard for audio type, and extract the keyframe extraction rules and frame rate standard for video type, as the execution content.

[0088] In the above scenario, the control action field (real-time monitoring operation, including data inflow monitoring, operation behavior monitoring, and abnormal data capture) and the control scope field (the data sources covered by the control are DS_M_Text, DS_R_Image, and DS_P_Video) of the exclusive control rule (P001_SC001) of the text data source to be controlled are parsed. Based on the type of control action (real-time monitoring action), the execution node type is determined to be a real-time processing node, and the node identifier of the execution node (such as RN001) is recorded. The specific execution content is extracted from the control action field, and the keyword filter list (such as filtering out keywords containing the word "confidential") and encoding verification standard (such as verifying the encoding format as UTF-8) are extracted from the text type as the execution content.

[0089] Step S154: Associate the node identifier and execution content of the execution node with the data source identifier and data unit list of the data unit group to be managed to generate an instruction number. The instruction number is composed of the node identifier and the data source identifier. It is clear that the execution object of the single data source management instruction is all the data units in the data unit group to be managed, forming a single data source management instruction containing the instruction number, execution node, execution content, and execution object.

[0090] In this scenario, the node identifier RN001 of the execution node, the execution content (keyword filtering list and encoding verification standard), and the data source identifier DS_M_Text of the data unit group to be managed are associated with the data unit list (all big data units with data source identifier DS_M_Text) to generate an instruction number. The instruction number is composed of the node identifier RN001 and the data source identifier DS_M_Text, i.e., RN001_DS_M_Text. This clarifies that the execution object of the single data source management instruction is all data units in the data unit group to be managed (all big data units with data source identifier DS_M_Text), forming a single data source management instruction containing the instruction number RN001_DS_M_Text, the execution node RN001, the execution content (keyword filtering list and encoding verification standard), and the execution object (all big data units with data source identifier DS_M_Text).

[0091] Step S155: Collect all single data source control instructions, query the exclusive control rule corresponding to each single data source control instruction, find the corresponding association chain strength value through the control association point associated with the exclusive control rule, use the association chain strength value as the basis for determining the execution priority of the single data source control instruction, and mark the corresponding priority level for each single data source control instruction.

[0092] Step S1551: Retrieve the final strength value of each association chain in the multimodal association chain set, establish a mapping relationship table between association chain identifier and final strength value, and the final strength value of the association chain is the association strength value of the control association point.

[0093] In the above scenario, the final strength value of each association chain in the multimodal association chain set is retrieved. For example, the final strength value of association chain with the identifier CL001 is 5, the final strength value of association chain with the identifier CL002 is 4, etc. A mapping table between association chain identifiers and final strength values ​​is established. The final strength value of the association chain is the association strength value of the control association point.

[0094] Step S1552: Summarize all single data source control instructions and add an association chain identifier field to each single data source control instruction. The association chain identifier field is consistent with the association chain identifier of the control association point associated with the exclusive control rule corresponding to the single data source control instruction.

[0095] In this scenario, all single data source control instructions are aggregated. For example, for a single data source control instruction with instruction number RN001_DS_M_Text, the associated control link of its corresponding exclusive control rule (P001_SC001) is identified by the association chain identifier CL001. An association chain identifier field is added to this single data source control instruction, with the field value being CL001.

[0096] Step S1553: Query the mapping relationship table based on the association chain identifier, obtain the association strength value corresponding to each single data source control instruction, and fill the association strength value into the association strength field of the single data source control instruction.

[0097] In the above scenario, the mapping relationship table is queried based on the association chain identifier CL001 to obtain the corresponding association strength value of 5. This association strength value is then filled into the association strength field of the single data source control instruction with instruction number RN001_DS_M_Text.

[0098] Step S1554: Divide the association strength value into multiple priority levels, and determine the corresponding priority level according to the range of association strength value of each single data source control instruction. Mark the priority level in the priority field of the single data source control instruction, and record the basis of the association strength value range corresponding to the priority level. Complete the priority marking of all single data source control instructions.

[0099] In this scenario, the association strength value is divided into multiple priority levels. For example, an association strength value greater than or equal to 5 is priority 1, an association strength value greater than or equal to 4 and less than 5 is priority 2, and an association strength value greater than or equal to 3 and less than 4 is priority 3, etc. The single data source control instruction with instruction number RN001_DS_M_Text has an association strength value of 5, falling within the range of greater than or equal to 5. Therefore, its corresponding priority level is determined to be 1. Priority level 1 is marked in the priority field of this single data source control instruction. Simultaneously, the association strength value range corresponding to priority level 1 is recorded as an association strength value greater than or equal to 5. This completes the priority marking for this single data source control instruction. Similarly, the priority marking for all single data source control instructions is completed.

[0100] Step S156: Sort the single data source control instructions in descending order of priority. Single data source control instructions with the same priority are arranged in alphabetical order of instruction number. Obtain the current system time as the generation time of the single data source control instruction. Extract the execution time interval parameter from the exclusive control rules as the execution cycle of the single data source control instruction. Supplement the execution result feedback address of each single data source control instruction. Integrate the sorted single data source control instructions with the generation time, execution cycle, and feedback address to obtain the multi-source big data fusion control instructions.

[0101] In the above scenario, single data source control instructions are sorted in descending order of priority. For example, instructions with priority 1 are placed first, followed by those with priority 2, and then those with priority 3. Single data source control instructions with the same priority are arranged in lexicographical order of their instruction numbers. For instance, single data source control instructions with the numbers RN001_DS_M_Text and RN002_DS_R_Image both have a priority of 1, but RN001_DS_M_Text is placed before RN002_DS_R_Image in lexicographical order. The system obtains the current system time (e.g., 10:00:00 on October 1, 2024) as the generation time of the single data source control instruction. It extracts the execution time interval parameter (e.g., once per hour) from the exclusive control rules as the execution cycle of the single data source control instruction. It supplements the execution result feedback address of each single data source control instruction (e.g., feedback to the enterprise's control platform database). It integrates the sorted single data source control instructions with the generation time, execution cycle, and feedback address to obtain the multi-source big data fusion control instruction. This multi-source big data fusion control instruction contains the relevant information of all single data source control instructions and can be used to fuse and control the enterprise's multi-source big data, break down data silos between departments, and realize the effective association and control of data from different departments.

[0102] Furthermore, Figure 2A schematic diagram of the hardware structure of a multimodal fusion management and control system 100 applied to multi-source big data for implementing the methods provided in the embodiments of this application is shown. Figure 2 As shown, the multimodal fusion management and control system 100 applied to multi-source big data may include at least one processor 102 (the processor 102 may be, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, a transmission device 106 for communication functions, and a controller 108. Those skilled in the art will understand that... Figure 2 The structure shown is for illustrative purposes only and does not limit the structure of the multimodal fusion management and control system 100 applied to multi-source big data. For example, the multimodal fusion management and control system 100 applied to multi-source big data may also include more than Figure 2 The more or fewer components shown, or having the same Figure 2 The different configurations shown.

[0103] The memory 104 can be used to store software programs and modules of application software, such as the program instructions corresponding to the method embodiments described above in this application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-described multimodal fusion management and control method applied to multi-source big data. The transmission device 106 is used to acquire or send data via a network.

[0104] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

Claims

1. A multi-modal fusion management and control method applied to multi-source big data, characterized in that, The method includes: A multi-source big data set is obtained, which includes big data units of text type, image type, audio type, and video type. Each type of big data unit carries a unique data source identifier and inherent data feature information. The inherent feature information of text type big data units is keyword groups and semantic tendency information. The inherent feature information of image type big data units is pixel distribution information and contour feature information. The inherent feature information of audio type big data units is frequency distribution information and tone feature information. The inherent feature information of video type big data units is frame sequence feature information and dynamic trajectory feature information. Based on the data source identifiers and inherent characteristics of big data units of different modal types, a cross-modal feature mapping table is constructed. Modal association chains between big data units of different modal types are established through the cross-modal feature mapping table to obtain a set of multimodal association chains. The strength value of each association chain in the multimodal association chain set is calculated. The multimodal association chain set is sorted in descending order according to the strength value. The sliding window mining algorithm is used to mine the control association points in the sorted association chains to obtain the control association point set. Construct a control scenario feature library, compare and match each control association point in the control association point set with the preset control scenarios in the control scenario feature library, and generate a multimodal control rule set based on the matching results; A management rule adaptation engine is built. The multi-modal management rule set is input into the management rule adaptation engine. The management rule adaptation engine adapts the rules to different modal big data units in the multi-source big data set and generates multi-source big data fusion management instructions. 2.The multi-modal fusion management and control method applied to multi-source big data according to claim 1, wherein, Based on the data source identifiers and inherent data feature information of big data units of different modal types, a cross-modal feature mapping table is constructed. Modal association chains between big data units of different modal types are established through this cross-modal feature mapping table, resulting in a multimodal association chain set, including: The multi-source big data collection is classified according to the data source identifier to obtain text data source subsets, image data source subsets, audio data source subsets, and video data source subsets. Each data source subset contains all big data units under the corresponding type. A vocabulary comparison matrix is ​​constructed. When the core feature vocabulary groups of the text, image, audio and video data source subsets are filled into the rows and columns of the matrix respectively, the core feature vocabulary groups corresponding to each data source subset are formed. The similarity value between the core feature vocabulary groups of different data source subsets is calculated. When the similarity value exceeds the preset value, it is determined that the corresponding data source subsets have common feature associations and are marked as related data source combinations. Using each subset of data sources in the associated data source combination as the starting node, the calculated similarity value as the connection weight between nodes, and common feature words as the connection label, a graph structure construction algorithm is used to construct the connection relationship between different subsets of data sources to form an initial association chain. The structure of the initial association chain includes the starting node identifier, the ending node identifier, the connection weight, and the connection label. A chain structure deduplication algorithm is used to traverse all initial association chains. When two initial association chains contain the same big data unit identifier, the connection weights of the two chains are weighted and summed. Common feature words in the connection tags are merged, and connection tags with different feature words are retained to form the connection tags of the merged chain structure. All data source identifiers and complete core feature word information of the merged chain structure are added. This process is repeated until an initial association chain without duplicate big data units exists, resulting in a multimodal association chain set.

3. The multimodal fusion and management method for multi-source big data according to claim 1, characterized in that, The process involves calculating the strength value of each association chain in the multimodal association chain set, sorting the multimodal association chain set in descending order based on the strength values, and using a sliding window mining algorithm to mine the control association points in the sorted association chains, resulting in a control association point set, including: Traverse each association chain in the multimodal association chain set, extract common feature words from the chain structure connection labels, count the actual number of common feature words in each association chain, use the actual number as the initial strength value of the association chain, and record the vocabulary statistics list corresponding to the initial strength value of each association chain. A data update log table is constructed to record the update timestamps of the big data units contained in each association chain in the multimodal association chain set. The number of updates per unit time is calculated as the update frequency. The update frequency of all association chains is normalized to obtain the normalized update frequency value. A mapping relationship table between the normalized update frequency value and the strength adjustment coefficient is established. The normalized update frequency value and the adjustment coefficient are positively correlated. The adjustment coefficient of each association chain is determined according to the mapping relationship table. The initial strength value is multiplied by the adjustment coefficient to obtain the final strength value of each association chain. A strength statistics report containing chain identifier, initial strength value, adjustment coefficient, and final strength value is generated. The multimodal association chain set is sorted according to the final strength value from largest to smallest to obtain an ordered multimodal association chain set; Set a selection ratio parameter, calculate the specific number of target association chains to be selected based on the selection ratio parameter and the total number of ordered multimodal association chain sets, select the corresponding number of association chains as target association chains starting from the starting position of the ordered multimodal association chain set, and use a chain structure parsing tool to parse the node information, connection weight and connection label of each target association chain one by one, extract the core feature words of each association chain, and record the node position and occurrence frequency of each core feature word in the association chain; A word location heatmap is constructed, with the node positions of all target association chains as the horizontal axis and the core feature words as the vertical axis. The frequency of word occurrence is used as the heat value. After generating the heatmap, the position with the highest heat value is marked as the core overlapping position. The node identifier corresponding to the core overlapping position is extracted. The corresponding data source identifier is queried based on the node identifier. The inherent feature information of the big data unit under the data source identifier is retrieved. Combined with the core feature words of the core overlapping position, the three are combined to determine the control association point. Each control association point includes a location identifier, a data source identifier, inherent feature information, and core feature words. A deduplication algorithm for control association points is adopted, using the data source identifier and core feature words as the joint deduplication key. All identified control association points are traversed, duplicate control association points are deleted, the attribute information of the deduplicated control association points is sorted, and the generation time and the association chain identifier of each control association point are added to form a set of control association points.

4. The multimodal fusion and management method for multi-source big data according to claim 1, characterized in that, The construction of the control scenario feature library involves comparing and matching each control-related point in the control-related point set with a preset control scenario in the control scenario feature library, and generating a multimodal control rule set based on the matching results, including: A control scenario feature library is constructed, which includes data storage control scenarios, data transmission control scenarios, data usage control scenarios, and data destruction control scenarios. Each control scenario has preset scenario feature terms and corresponding scenario control requirements. The feature terms for data storage control scenarios are storage period, storage medium, and backup frequency, and the control requirements are specifying the storage medium type and backup execution method. The feature terms for data transmission control scenarios are transmission protocol, encryption method, and transmission rate, and the control requirements are specifying the transmission protocol and encryption algorithm. The feature terms for data usage control scenarios are usage permissions, access frequency, and operation logs, and the control requirements are setting permission levels and log recording ranges. The feature terms for data destruction control scenarios are destruction method, residual detection, and destruction logs, and the control requirements are specifying the destruction method and residual detection standards. A feature comparison vector space is constructed. The core feature words of each control-related point in the control-related point set are transformed into feature vectors. At the same time, the scene feature words of each scene in the control scene feature library are also transformed into feature vectors. The Euclidean distance algorithm is used to calculate the distance between the feature vector of each control-related point and the feature vector of each scene. The smaller the distance value, the higher the matching degree. The number of feature words whose distance value is less than the preset distance threshold when each control-related point matches each scene is counted as the feature word matching number. Pre-defined control scenarios where the number of matching feature words reaches a preset threshold are marked as target control scenarios corresponding to control association points. Extract the scenario control requirements of the target control scenario, analyze the constraints and execution actions in the scenario control requirements, determine the specific source of the control object based on the data source identifier of the control association point, and adjust the execution actions in combination with the inherent characteristic information of the control association point to form an adapted control requirement; The location identifier and data source identifier of the control-related point are associated and bound with the scene identifier of the target control scenario, the constraints and execution actions that adapt to the control requirements, to generate a rule identifier. The rule identifier is composed of the location identifier and the scene identifier, forming a single multimodal control rule that includes the rule identifier, control object information, constraints and execution actions. All individual multimodal control rules are classified according to the scenario identifier in the rule identifier. Rules with the same scenario identifier are grouped together to form a data storage control rule group, a data transmission control rule group, a data usage control rule group, and a data destruction control rule group. The rules in each group are sorted according to the position identifier of the rule identifier, and the scope of application of each group of rules is supplemented to obtain a multimodal control rule set.

5. The multimodal fusion and management method for multi-source big data according to claim 4, characterized in that, The process involves extracting the scenario control requirements of the target control scenario, parsing the constraints and execution actions within these requirements, determining the specific source of the control object based on the data source identifier of the control-related points, and adjusting the execution actions in conjunction with the inherent characteristic information of the control-related points to form an adapted control requirement, including: A rule parser is used to perform structured parsing of the scenario control requirements of the target control scenario, and to break down the three core element fields: control object field, control action field, and control scope field. The control object field records the type and source of the data to be controlled, the control action field records the specific operation type to be performed, and the control scope field records the range of data sources covered by the control. Query the data source information corresponding to the data source identifier of the control and management associated point, determine the big data unit type under the data source, label the encoding format information for text type, label the resolution information for image type, label the sampling rate information for audio type, label the frame rate information for video type, fill the above information into the control and management object field, and replace the original control and management object field content. Extract the update frequency parameter from the inherent feature information of the control-related points. If the update frequency parameter is greater than the preset update threshold, it is determined to be a real-time update type, and the operation type in the control action field is adjusted to real-time monitoring operation, which includes data inflow monitoring, operation behavior monitoring, and abnormal data capture. If the update frequency parameter is less than or equal to the preset update threshold, it is determined to be a static storage type, and the operation type in the control action field is adjusted to periodic verification operation, which includes data integrity verification, storage status verification, and redundant data cleanup. Query the node information of the associated chain where the control and management points are located, extract the data source identifiers corresponding to all nodes, form a list of data source identifiers, fill the list of data source identifiers formed by extracting the data source identifiers corresponding to all nodes into the control scope field, replace the content in the original control scope field, and record all data sources covered by control. The updated control object field, the adjusted control action field, and the new control scope field are integrated. The execution order is determined according to the type of control action. Real-time monitoring operations are arranged in the order of data inflow monitoring, operation behavior monitoring, and abnormal data capture. Periodic verification operations are arranged in the order of data integrity verification, storage status verification, and redundant data cleanup. The execution time interval parameter is added to form an adapted control requirement that includes field information, execution order, and time interval.

6. The multimodal fusion management and control method for multi-source big data according to claim 1, characterized in that, The process of building a control rule adaptation engine involves inputting a multimodal control rule set into the engine, and then using the engine to adapt rules to different modal big data units within the multi-source big data set to generate multi-source big data fusion control instructions, including: Read the data source identifier of each big data unit in the multi-source big data set, construct a data source-data unit mapping table, and group big data units with the same data source identifier according to the mapping table to obtain the text data source to be managed group, image data source to be managed group, audio data source to be managed group, and video data source to be managed group. Each to be managed group contains all big data units under the data source and their corresponding inherent feature information. A rule matching index is constructed, using data source type and data features as index keys. Each rule in the multimodal management rule set is classified and stored according to the index key. Each data unit group to be managed is traversed, and the data source type and core data features of the data unit group to be managed are extracted. The matching rules are queried according to the index key. If the data unit group to be managed is text type, the rule containing the text feature index key is matched. If it is image type, the rule containing the image feature index key is matched. And so on, to obtain the exclusive management rules for each data unit group to be managed. The system analyzes the control action field and control scope field of the exclusive control rules, determines the execution node type according to the type of control action, assigns real-time monitoring actions to real-time processing nodes, and assigns periodic verification actions to timed processing nodes, and records the node identifier of the execution node; it extracts the specific execution content from the control action field, extracts keyword filtering list and encoding verification standards from text type, extracts sensitive area coordinates and resolution standards from image type, extracts voiceprint feature library and sampling rate standards from audio type, and extracts keyframe extraction rules and frame rate standards from video type, as the execution content; The node identifier and execution content of the execution node are associated with the data source identifier and data unit list of the data unit group to be managed to generate an instruction number. The instruction number is composed of the node identifier and the data source identifier. It is clear that the execution object of the single data source management instruction is all the data units in the data unit group to be managed, forming a single data source management instruction that includes the instruction number, execution node, execution content, and execution object. Collect all single data source control instructions, query the exclusive control rules corresponding to each single data source control instruction, find the corresponding association chain strength value through the control association points associated with the exclusive control rules, use the association chain strength value as the basis for determining the execution priority of single data source control instructions, and mark the corresponding priority level for each single data source control instruction; Single data source control instructions are sorted in descending order of priority. Single data source control instructions with the same priority are arranged in alphabetical order of instruction number. The current system time is obtained as the generation time of the single data source control instruction. The execution time interval parameter is extracted from the exclusive control rules as the execution cycle of the single data source control instruction. The execution result feedback address of each single data source control instruction is supplemented. The sorted single data source control instructions are integrated with the generation time, execution cycle, and feedback address to obtain multi-source big data fusion control instructions.

7. The multimodal fusion management and control method for multi-source big data according to claim 3, characterized in that, The setting of the selection ratio parameter, calculating the specific number of target association chains based on the selection ratio parameter and the total number of ordered multimodal association chain sets, and selecting the corresponding number of association chains as target association chains starting from the beginning of the ordered multimodal association chain set, includes: Traverse the ordered multimodal association chain set, use a counter to count the total number of association chains, set the selection ratio parameter, the selection ratio parameter is determined by the scene accuracy level in the control scene feature library according to the control accuracy requirements, and calculate the number of target association chains to be selected by multiplying the total number by the selection ratio parameter. If the calculation result is a decimal, round it up. Starting from the first association chain in the ordered multimodal association chain set, selection is carried out sequentially until the selection quantity is reached. The selected association chains are marked as target association chains, and a target association chain list is generated. The target association chain list includes association chain identifiers, final strength values, and association chain structure information. The structure information of each target association chain is loaded using an association chain structure parsing program. The identifiers of the starting node, intermediate node, and ending node in the association chain are parsed out, the core feature vocabulary group corresponding to each node is extracted, and the node identifier to which each core feature vocabulary belongs and its position in the core feature vocabulary group are recorded to form a vocabulary-node mapping table. Construct a statistical table of the distribution of core feature words. The rows of the statistical table are word names and the columns are node identifiers. Traverse the word-node mapping table and record the number of occurrences in the intersection cell of the corresponding word and node. If the same word appears multiple times in the same node, the number of occurrences is accumulated. At the same time, add a function column for total occurrences to the statistical table to record the total number of occurrences of each word in all nodes. The core feature words are sorted according to their frequency of occurrence, and the core feature words with the highest frequency of occurrence are marked as key feature words, thus improving the distribution statistics table.

8. The multimodal fusion management and control method for multi-source big data according to claim 3, characterized in that, The algorithm employs a deduplication mechanism for control association points. Using the data source identifier and core feature words as the joint deduplication key, it iterates through all identified control association points, deletes duplicates, and then organizes the attribute information of the deduplicated control association points, supplementing each point with its generation time and associated chain identifier, forming a set of control association points, including: Extract the attribute information of each identified control and management link point. The attribute information includes location identifier, data source identifier, inherent feature information and core feature words. The location identifier, data source identifier, inherent feature information and core feature words are used as the basis for deduplication judgment. Construct a deduplication keyword group, and select data source identifiers and core feature words as joint deduplication keywords; Create a temporary storage list for deduplication, initialize the temporary storage list for deduplication to be empty, traverse all determined control association points, perform hash calculation on the joint deduplication key of each control association point, and obtain the key hash value; Check if there is a duplicate keyword hash value in the deduplication temporary storage list. If not, store the attribute information and keyword hash value of the control association point into the deduplication temporary storage list. If it exists, skip the control association point and complete the initial deduplication. The attribute information of the control association points after initial deduplication is checked. The correspondence between the location identifier and the data source identifier is checked, and the correspondence between the inherent feature information and the core feature words is checked. If there is a mismatch, the control association points are re-determined. After the verification is correct, the generation time and the association chain identifier of each control association point are added. All the sorted control association points are sorted by the data source identifier to obtain the control association point set.

9. The multimodal fusion management and control method for multi-source big data according to claim 6, characterized in that, The process involves collecting all single-data source control instructions, querying the specific control rules corresponding to each single-data source control instruction, finding the corresponding association chain strength value through the control association points associated with the specific control rules, and using the association chain strength value as the basis for determining the execution priority of the single-data source control instruction. Each single-data source control instruction is then labeled with a corresponding priority level, including: Retrieve the final strength value of each association chain in the multimodal association chain set, establish a mapping relationship table between association chain identifiers and final strength values, and the final strength value of the association chain is the association strength value of the control association point; Summarize all single data source control instructions and add an association chain identifier field to each single data source control instruction. The association chain identifier field is consistent with the association chain identifier of the control association point associated with the exclusive control rule corresponding to the single data source control instruction. Query the mapping relationship table based on the association chain identifier, obtain the association strength value corresponding to each single data source control instruction, and fill the association strength value into the association strength field of the single data source control instruction; The association strength value is divided into multiple priority levels, and the corresponding priority level is determined according to the range of association strength value of each single data source control instruction. The priority level is marked in the priority field of the single data source control instruction, and the basis of the association strength value range corresponding to the priority level is recorded to complete the priority marking of all single data source control instructions.

10. A multimodal fusion management and control system applied to multi-source big data, characterized in that, It includes a processor and a readable storage medium storing a program that, when executed by the processor, implements the multimodal fusion management method for multi-source big data as described in any one of claims 1-9.

Images