An information processing method and system based on multiple information sources

Abstract

The present invention discloses an information processing method and system based on multiple information sources, which is used in the field of information processing. The method includes the following steps: obtaining the source and content of information; building an information classification framework for information types, designing label categories for information types, and classifying them by preset classification rules; processing non-text information using multimodal feature fusion technology, and extracting entity relationship knowledge of non-text information through knowledge graphs; mining the connection between information based on spatiotemporal entity information, performing cross-source information fusion, and constructing information association graphs; providing intelligent application services based on association analysis results. The present invention constructs a multimodal learning framework and a common discriminant subspace to better integrate and utilize image features and audio features, improve the processing capability of non-text information, and apply the optimized knowledge graphs to actual business scenarios to meet the actual needs of users.

Description

Technical Field

[0001] The present invention relates to the field of information processing, and in particular, to an information processing method and system based on multiple information sources. Background Art

[0002] With the rapid development of the Internet and the advancement of global informatization, we are facing the realistic challenge of information explosion and encountering a large amount of multimodal information, including text, images, audio, video and other types. Processing the above information and classifying and parsing it according to the personalized needs of each user has become an important demand in today's society. At the same time, real-time processing of newly collected information, especially non-text information, and extracting practical knowledge and information from it is also an important direction of current technological development.

[0003] In order to improve the accuracy of classifying and processing information, it is also crucial to evaluate and optimize the performance of information classification models. However, in this context, existing technologies may not be able to fully meet these requirements, which is an issue that needs further research and resolution.

[0004] Existing information classification methods may not be customized according to the specific needs of users, and may not be able to effectively classify newly collected data in real-time scenarios. In addition, existing methods may lack effective model performance evaluation and optimization mechanisms, resulting in insufficient classification accuracy. At the same time, existing information processing technologies may mainly focus on text information, and have limited processing capabilities for non-text information such as images, audio and video. They may not be able to extract and integrate multimodal features in these non-text information, and may not be able to effectively utilize the entity relationship knowledge in these information. They may not be able to meet the actual needs of users through effective optimization methods.

[0005] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0006] In order to overcome the above problems, the present invention aims to propose an information processing method and system based on multiple information sources, with the aim of solving the problem that existing information classification methods may not be customized according to the specific needs of users and may not be able to effectively classify newly collected data in real-time scenarios. In addition, existing methods may lack effective model performance evaluation and optimization mechanisms, resulting in insufficient classification accuracy.

[0007] To this end, the specific technical solution adopted by the present invention is as follows: According to one aspect of the present invention, there is provided an information processing method based on multiple information sources, the information processing method comprising the following steps: S1. Obtain the source and content of information, determine the type of information, and extract time, location and entity information; S2. Build an information classification framework based on the use requirements of information, design label categories for information types, and classify them using preset classification rules; S3. Use multimodal feature fusion technology to process non-text information, and extract entity relationship knowledge of non-text information through knowledge graph, and extract spatiotemporal entity information at the same time; S4. Mining the connections between information based on spatiotemporal entity information, integrating cross-source information, building an information association graph, and performing association analysis based on the information association graph, while providing a visual display interface for association analysis; S5. Provide intelligent application services and optimize information processing procedures based on the results of correlation analysis.

[0008] Optionally, according to the use requirements of the information, building an information classification framework for the information types, designing label categories for the information types, and classifying by preset classification rules includes the following steps: S21. Communicate with users, obtain their needs, analyze their feedback, and determine the purpose of information; S22. Create an information classification framework based on information usage requirements and compare it with user needs; S23. Adjust the information classification framework based on user feedback and design label categories for each category; S24. Classify information types using natural language processing technology according to preset classification rules and label categories.

[0009] Optionally, classifying the information types using natural language processing technology according to preset classification rules and label categories includes the following steps: S241. Formulate classification rules and label categories based on information usage requirements and domain knowledge; S242, using a word embedding model to convert text data in the information category into an initial feature vector; S243, using information gain on the initial feature vocabulary to determine the final feature word, and then calculating the value of the feature word to form a final feature vector; S244, dividing the final feature vector data set into a training set and a test set; S245, selecting a machine learning algorithm, and training a classification model using the feature vectors and label categories of the training set; S246, using the test set to evaluate the performance of the classification model, using the evaluation index to evaluate the performance of the classification model, and optimizing the parameters of the classification model according to the evaluation results; S247. Apply the optimized classification model to actual scenarios and classify the collected new text data into label categories.

[0010] Optionally, the formula for calculating the value of the feature word is: ; In the formula, Represents the feature words in text data The characteristic value of represents the weighting factor; Represents the data point in the feature word The set of all connected data points; Represents a data point in text data With data points The weight of the connecting line; represents the denominator of the normalized formula; represents the weighted value; Represents the feature words in text data Medium data point Frequency of occurrence.

[0011] Optionally, the non-text information is processed using multimodal feature fusion technology, and entity relationship knowledge of the non-text information is extracted through the knowledge graph, and the spatiotemporal entity information is extracted, including the following steps: S31, preprocessing non-text data, the non-text data including images, audio and video; S32, extracting visual features of the image, sound features of the audio, and visual and audio features of the video; S33, integrating the visual features of the image, the sound features of the audio, and the visual and audio features of the video through multimodal feature fusion technology; S34, constructing a knowledge graph using the fused features, and annotating the spatiotemporal entity information in the features during the construction process; S35, fusing the entity information extracted from the non-text data with the spatiotemporal entity information in the knowledge graph to obtain the final knowledge graph; S36. Verify the final knowledge graph and optimize the knowledge graph according to the verification results; S37. Apply the optimized knowledge graph to business scenarios.

[0012] Optionally, integrating the visual features of the image, the sound features of the audio, and the visual and audio features of the video by a multimodal feature fusion technique comprises the following steps: S331, extracting texture and edge features of the image using a Gabor filter; S332, collecting audio signals and extracting audio features through Mel-frequency cepstral coefficients; S333, combining the texture and edge features of the image with the features of the audio, and extracting dynamic texture features and audio-visual synchronization features; S334, construct a multimodal learning framework, integrate image features and audio features, and create a comprehensive feature vector, merging different types of features according to a preset ratio; S335, constructing a common discriminant subspace using a statistical learning method, and solving an optimal projection matrix according to the eigenvalues ​​and eigenvectors of the samples; S336, using a classifier in the constructed common discriminant subspace to classify the integrated features; S337, training the classifier, optimizing the classification parameters using the cross-validation method, and testing the classifier using the data set to evaluate its classification performance; S338. Analyze the impact of different modal features on classification results, perform feature selection and classifier optimization, and analyze the applicability of multimodal fusion models.

[0013] Optionally, constructing a multimodal learning framework, integrating image features and audio features, and creating a comprehensive feature vector, merging different types of features according to a preset ratio includes the following steps: S3341. Allocate weights to the image features and the audio features according to their contribution degrees. S3342, integrating the weighted image features and audio features by weighted summation to obtain a comprehensive feature vector; S3343, constructing a multimodal learning framework using comprehensive feature vectors; S3344. Determine the number of nodes in the input layer, the hidden layer, and the output layer according to the multimodal learning framework, and select a loss function for the task; S3345. Use the data in the training set to train the multimodal learning framework, update the parameters of the multimodal learning framework through the back-propagation algorithm, evaluate the performance indicators of the multimodal learning framework on the validation set, and adjust the structure of the multimodal learning framework according to the evaluation results.

[0014] Optionally, constructing a common discriminant subspace using a statistical learning method and solving an optimal projection matrix according to the eigenvalues ​​and eigenvectors of the samples includes the following steps: S3351, collecting image feature and audio feature data after weight allocation; S3352, preprocessing image features and audio features; S3353, projecting the preprocessed image features and audio features into their respective subspaces, and calculating the intra-class scatter matrix and the inter-class scatter matrix according to the category information; S3354, transforming the intra-class scatter matrix and the inter-class scatter matrix to obtain a new discriminant matrix; S3355. Calculate the eigenvalues ​​and eigenvectors of the new discriminant matrix; S3356. Select the optimal projection matrix according to the eigenvalues ​​and eigenvectors of the discriminant matrix.

[0015] Optionally, mining the connection between information according to the spatiotemporal entity information, performing cross-source information fusion, constructing an information association graph, performing association analysis based on the information association graph, and providing a visual display interface for the association analysis include the following steps: S41. Collecting spatiotemporal entity information from different sources, where the spatiotemporal entity information at least includes time, place, person, and event; S42, preprocessing the time, place, person, and event of the spatiotemporal entity information, extracting the time, place, person, and event of the spatiotemporal entity information and the relationship between the time, place, person, and event of the spatiotemporal entity information; S43. Use knowledge graph technology to connect the same spatiotemporal entity information in different information sources to form a unified view; S44, constructing a heterogeneous information network based on the time, place, person, event of the empty entity information and the relationship between the time, place, person, event of the spatiotemporal entity information, wherein the nodes of the heterogeneous information network represent entities and the edges of the heterogeneous information network represent the relationship between the entities; S45, performing association analysis on heterogeneous information networks, finding paths in the network according to the analysis target, and obtaining associations between different information sources; S46. Use graph analysis algorithms to find entities and paths in heterogeneous information networks; S47. Provide a visual display interface to visualize the topological structure, key entities and key paths of heterogeneous information networks.

[0016] According to another aspect of the present invention, there is also provided an information processing system based on multiple information sources, the system comprising: an information acquisition and content analysis module, an information classification and label design module, a non-text information processing and entity relationship extraction module, a correlation analysis module and an intelligent application service module; The information acquisition and content analysis module is connected with the non-text information processing and entity relationship extraction module through the information classification and label design module, and the non-text information processing and entity relationship extraction module is connected with the intelligent application service module through the association analysis module; Information acquisition and content analysis module, used to obtain the source and content of information, determine the type of information, and extract time, location and entity information; The information classification and label design module is used to build an information classification framework for information types according to the use requirements of the information, design label categories for information types, and classify them according to preset classification rules; The non-text information processing and entity relationship extraction module is used to process non-text information using multimodal feature fusion technology, extract entity relationship knowledge from non-text information through knowledge graphs, and extract spatiotemporal entity information at the same time; The association analysis module is used to mine the relationship between information based on spatiotemporal entity information, perform cross-source information fusion, build an information association graph, and perform association analysis based on the information association graph, while providing a visual display interface for association analysis; The intelligent application service module is used to provide intelligent application services and optimize the information processing process based on the correlation analysis results.

[0017] Compared with the prior art, this application has the following beneficial effects: 1. The present invention can create and adjust the information classification architecture and design label categories according to user needs. By using natural language processing technology and machine learning algorithms, information can be effectively classified. In addition, the performance of the classification model will be evaluated and the model will be optimized based on the evaluation results to improve the accuracy of classification. Finally, by applying the optimized model to actual scenarios, real-time classification of newly collected text data can be achieved to meet the real-time needs of users.

[0018] 2. The present invention utilizes multimodal feature fusion technology to process non-text information. This method can better utilize various types of information such as images, audio and video, and improve the processing capability of non-text information. It not only extracts the visual features of images, the sound features of audio, and the visual and audio features of videos, but also integrates these features to create a comprehensive feature vector, which can better represent and utilize non-text information. The fused features are used to construct a knowledge graph, and the spatiotemporal entity information in the features is annotated during the construction process, which can better utilize the entity relationship knowledge in the non-text information.

[0019] 3. The present invention also trains the classifier, uses the cross-validation method to optimize the classification parameters, and tests the classifier through the data set to evaluate its classification performance. This can better optimize the model, improve the classification accuracy of non-text information, and construct a multimodal learning framework and a common discriminant subspace. These two construction steps can better integrate and utilize image features and audio features, improve the processing capabilities of non-text information, and apply the optimized knowledge graph to actual business scenarios. This can directly meet the actual needs of users and improve the practicality of non-text information. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above characteristics, features and advantages of the present invention and their implementation methods and methods will become more clearly understood in conjunction with the following description of the embodiments, which are described in detail in conjunction with the accompanying drawings. Herein, a schematic diagram is shown: Figure 1 is a flow chart of an information processing method based on multiple information sources according to an embodiment of the present invention; Figure 2 It is a principle block diagram of an information processing system based on multiple information sources according to an embodiment of the present invention.

[0021] In the figure: 1. Information acquisition and content analysis module; 2. Information classification and label design module; 3. Non-text information processing and entity relationship extraction module; 4. Association analysis module; 5. Intelligent application service module. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] According to an embodiment of the present invention, a method and system for information processing based on multiple information sources are provided.

[0024] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to an embodiment of the present invention, a method for information processing based on multiple information sources is provided, and the information processing method comprises the following steps: S1. Obtain the source and content of information, determine the type of information, and extract time, location and entity information.

[0025] It needs to be explained that first, information is collected from various sources, including at least the Internet, databases, API interfaces, sensor data, etc. The content of the information needs to cover multiple forms such as text, images, audio, video, etc. After determining the type of information, such as news, social media posts, scientific research reports, etc., key elements such as time, place and entity information are extracted. Natural language processing technology, such as named entity recognition, needs to be applied to identify timestamps, geographic locations and specific entities such as people, organizations or events in the text, so as to process and analyze data more specifically.

[0026] S2. Build an information classification framework based on the information usage requirements, design label categories for information types, and classify them using preset classification rules.

[0027] Preferably, according to the use requirements of the information, an information classification framework of the information types is constructed, label categories of the information types are designed, and classification is performed according to preset classification rules, including the following steps: S21. Communicate with users, obtain their needs, analyze their feedback, and determine the purpose of information; S22. Create an information classification framework based on information usage requirements and compare it with user needs; S23. Adjust the information classification framework based on user feedback and design label categories for each category; S24. Classify information types using natural language processing technology according to preset classification rules and label categories.

[0028] Preferably, according to preset classification rules and label categories, using natural language processing technology to classify information types includes the following steps: S241. Formulate classification rules and label categories based on information usage requirements and domain knowledge; S242, using a word embedding model to convert text data in the information category into an initial feature vector; S243, using information gain on the initial feature vocabulary to determine the final feature word, and then calculating the value of the feature word to form a final feature vector; S244, dividing the final feature vector data set into a training set and a test set; S245, selecting a machine learning algorithm, and training a classification model using the feature vectors and label categories of the training set; S246, using the test set to evaluate the performance of the classification model, using the evaluation index to evaluate the performance of the classification model, and optimizing the parameters of the classification model according to the evaluation results; S247. Apply the optimized classification model to actual scenarios and classify the collected new text data into label categories.

[0029] Preferably, the formula for calculating the value of the feature word is: ; In the formula, Represents the feature words in text data The characteristic value of represents the weighting factor; Represents the data point in the feature word The set of all connected data points; Represents a data point in text data With data points The weight of the connecting line; represents the denominator of the normalized formula; represents the weighted value; Represents the feature words in text data Medium data point Frequency of occurrence.

[0030] In addition, according to different document parts in the above formula, The weights will vary: in the title, the weight of the word frequency is usually set to 2, in the abstract, the weight of the word frequency is usually set to 1.8, and at the beginning or end of each paragraph, the weight of the word frequency is usually set to 1.5.

[0031] It should be explained that the purpose of information is first determined by communicating with users and analyzing feedback. Then, a preliminary information classification architecture is constructed according to the needs, and fine-tuned according to user feedback. After designing label categories for each category, natural language processing technology and preset classification rules are used to classify information, and classification rules and label categories are formulated. The word embedding model is used to convert text data into feature vectors, and then the information gain is used to determine the final feature words, and the value of the feature words is calculated to form the final feature vector. Next, the data set is divided into a training set and a test set. A machine learning algorithm, such as a support vector machine or a random forest, is selected to train the classification model with the feature vector and label category of the training set, and then the test set is used to evaluate the performance of the model. Some evaluation indicators, such as precision, recall rate and F1 score, are used to evaluate the performance of the model, and the parameters of the model are optimized according to the evaluation results. Finally, the optimized model is applied to the actual scenario to classify the newly collected text data to meet the specific needs of users, providing an effective basis for subsequent information processing and analysis.

[0032] S3. Use multimodal feature fusion technology to process non-text information, and extract entity relationship knowledge of non-text information through knowledge graph, and extract spatiotemporal entity information at the same time.

[0033] Preferably, the non-text information is processed using multimodal feature fusion technology, and the entity relationship knowledge of the non-text information is extracted through the knowledge graph, and the spatiotemporal entity information is extracted, including the following steps: S31, preprocessing non-text data, the non-text data including images, audio and video; S32, extracting visual features of the image, sound features of the audio, and visual and audio features of the video; S33, integrating the visual features of the image, the sound features of the audio, and the visual and audio features of the video through multimodal feature fusion technology; S34, constructing a knowledge graph using the fused features, and annotating the spatiotemporal entity information in the features during the construction process; S35, fusing the entity information extracted from the non-text data with the spatiotemporal entity information in the knowledge graph to obtain the final knowledge graph; S36. Verify the final knowledge graph and optimize the knowledge graph according to the verification results; S37. Apply the optimized knowledge graph to business scenarios.

[0034] In addition, it should be noted that the non-text data, including images, audio and video, are first preprocessed, and then the features of these data are extracted, such as the visual features of the image, the sound features of the audio, and the visual and audio features of the video. The visual and audio features of the video are integrated through multimodal feature fusion technology, and some deep learning technologies are involved, such as convolutional neural networks or recurrent neural networks. The fused features are then used to construct a knowledge graph, and the spatiotemporal entity information in the features is annotated during the construction process. Some knowledge graph construction technologies will be used, such as graph embedding or relational learning. The entity information extracted from the non-text data is then fused with the spatiotemporal entity information in the knowledge graph to obtain the final knowledge graph, which is then verified and optimized based on the verification results. Finally, the optimized knowledge graph is applied to business scenarios.

[0035] Preferably, integrating the visual features of the image, the sound features of the audio, and the visual and audio features of the video by the multimodal feature fusion technology comprises the following steps: S331, extracting texture and edge features of the image using a Gabor filter; S332, collecting audio signals and extracting audio features through Mel-frequency cepstral coefficients; S333, combining the texture and edge features of the image with the features of the audio, and extracting dynamic texture features and audio-visual synchronization features; S334, construct a multimodal learning framework, integrate image features and audio features, and create a comprehensive feature vector, merging different types of features according to a preset ratio; S335, constructing a common discriminant subspace using a statistical learning method, and solving an optimal projection matrix according to the eigenvalues ​​and eigenvectors of the samples; S336, using a classifier in the constructed common discriminant subspace to classify the integrated features; S337, training the classifier, optimizing the classification parameters using the cross-validation method, and testing the classifier using the data set to evaluate its classification performance; S338. Analyze the impact of different modal features on classification results, perform feature selection and classifier optimization, and analyze the applicability of multimodal fusion models.

[0036] Preferably, constructing a multimodal learning framework, integrating image features and audio features, and creating a comprehensive feature vector, merging different types of features according to a preset ratio includes the following steps: S3341. Allocate weights to the image features and the audio features according to their contribution degrees. S3342, integrating the weighted image features and audio features by weighted summation to obtain a comprehensive feature vector; S3343, constructing a multimodal learning framework using comprehensive feature vectors; S3344. Determine the number of nodes in the input layer, the hidden layer, and the output layer according to the multimodal learning framework, and select a loss function for the task; S3345. Use the data in the training set to train the multimodal learning framework, update the parameters of the multimodal learning framework through the back-propagation algorithm, evaluate the performance indicators of the multimodal learning framework on the validation set, and adjust the structure of the multimodal learning framework according to the evaluation results.

[0037] Preferably, constructing a common discriminant subspace using a statistical learning method and solving the optimal projection matrix according to the eigenvalues ​​and eigenvectors of the samples comprises the following steps: S3351, collecting image feature and audio feature data after weight allocation; S3352, preprocessing image features and audio features; S3353, projecting the preprocessed image features and audio features into their respective subspaces, and calculating the intra-class scatter matrix and the inter-class scatter matrix according to the category information; S3354, transforming the intra-class scatter matrix and the inter-class scatter matrix to obtain a new discriminant matrix; S3355. Calculate the eigenvalues ​​and eigenvectors of the new discriminant matrix; S3356. Select the optimal projection matrix according to the eigenvalues ​​and eigenvectors of the discriminant matrix.

[0038] It should be explained that Gabor filters are first used to extract the texture and edge features of the image. At the same time, audio signals are also collected to extract the audio features through the Mel-frequency cepstral coefficient method. The audio features are then merged to extract dynamic texture features and audio-visual synchronization features. When constructing a multimodal learning framework, weights are assigned to each feature according to its contribution. After that, all the features are integrated through weighted summation to obtain a comprehensive feature vector. Using this comprehensive feature vector, a multimodal learning framework is constructed, and the number of nodes in the input layer, hidden layer, and output layer is determined, as well as the loss function of the task. The multimodal learning framework is trained using the data from the training set. A multimodal learning framework was proposed, which used the back-propagation algorithm to update its parameters and evaluated its performance indicators on the validation set. According to the evaluation results, the structure of the multimodal learning framework was adjusted, and the image feature and audio feature data after weight assignment were collected. The feature data were preprocessed and the preprocessed image features and audio features were projected into their respective subspaces. According to the category information, the intra-class scatter matrix and the inter-class scatter matrix were calculated. Then, the scatter matrix was transformed to obtain a new discriminant matrix, and the eigenvalues ​​and eigenvectors of the discriminant matrix were calculated. Finally, the optimal projection matrix was selected according to the eigenvalues ​​and eigenvectors, which provided key technical support for subsequent classification and prediction.

[0039] S4. Mining the connections between information based on spatiotemporal entity information, performing cross-source information fusion, constructing an information association graph, and performing association analysis based on the information association graph, while providing a visual display interface for association analysis.

[0040] Preferably, mining the connection between information based on the spatiotemporal entity information, performing cross-source information fusion, constructing an information association graph, and performing association analysis based on the information association graph, while providing a visual display interface for the association analysis, includes the following steps: S41. Collecting spatiotemporal entity information from different sources, where the spatiotemporal entity information at least includes time, place, person, and event; S42, preprocessing the time, place, person, and event of the spatiotemporal entity information, extracting the time, place, person, and event of the spatiotemporal entity information and the relationship between the time, place, person, and event of the spatiotemporal entity information; S43. Use knowledge graph technology to connect the same spatiotemporal entity information in different information sources to form a unified view; S44, constructing a heterogeneous information network based on the time, place, person, event of the empty entity information and the relationship between the time, place, person, event of the spatiotemporal entity information, wherein the nodes of the heterogeneous information network represent entities and the edges of the heterogeneous information network represent the relationship between the entities; S45, performing association analysis on heterogeneous information networks, finding paths in the network according to the analysis target, and obtaining associations between different information sources; S46. Use graph analysis algorithms to find entities and paths in heterogeneous information networks; S47. Provide a visual display interface to visualize the topological structure, key entities and key paths of heterogeneous information networks.

[0041] It needs to be explained that in the process of cross-source information fusion, we first collect spatiotemporal entity information from different sources. The spatiotemporal entity information includes at least time, place, people and events. Then we preprocess the spatiotemporal entity information to extract the time, place, people, events and the relationship between them. Then we use knowledge graph technology to connect the same spatiotemporal entity information in different information sources to form a unified view. A heterogeneous information network is constructed based on the spatiotemporal entity information, in which the nodes of the network represent entities and the edges represent the relationship between entities. Then we perform association analysis on this heterogeneous information network, find paths in the network according to the analysis objectives, and obtain the associations between different information sources. We use graph analysis algorithms to find entities and paths in the heterogeneous information network. Finally, we provide a visual display interface to visualize the topological structure, key entities and key paths of the heterogeneous information network, providing comprehensive and in-depth information support for decision makers.

[0042] S5. Provide intelligent application services and optimize information processing procedures based on the results of correlation analysis.

[0043] It should be explained that, based on the user's behavior and preferences, we provide personalized information recommendation and search services, as well as other related data-driven intelligent services, which may include but are not limited to intelligent query, intelligent analysis, intelligent prediction, etc. In addition, we continue to learn and improve, and through machine learning and artificial intelligence technology, we continuously optimize its service quality and information processing process to meet the growing needs of users and enhance user experience.

[0044] According to another embodiment of the present invention, Figure 2 As shown, there is also provided an information processing system based on multiple information sources, the system comprising: an information acquisition and content analysis module 1, an information classification and label design module 2, a non-text information processing and entity relationship extraction module 3, a correlation analysis module 4 and an intelligent application service module 5; The information acquisition and content analysis module 1 is connected to the non-text information processing and entity relationship extraction module 3 through the information classification and label design module 2, and the non-text information processing and entity relationship extraction module 3 is connected to the intelligent application service module 5 through the association analysis module 4; Information acquisition and content analysis module 1, used to obtain the source and content of information, determine the type of information, and extract time, location and entity information; Information classification and label design module 2, used to build an information classification framework for information types according to information usage requirements, design label categories for information types, and classify them according to preset classification rules; Non-text information processing and entity relationship extraction module 3, used to process non-text information using multimodal feature fusion technology, and extract entity relationship knowledge of non-text information through knowledge graph, and extract spatiotemporal entity information at the same time; The association analysis module 4 is used to mine the relationship between information based on the spatiotemporal entity information, perform cross-source information fusion, construct an information association graph, perform association analysis based on the information association graph, and provide a visual display interface for association analysis; The intelligent application service module 5 is used to provide intelligent application services and optimize the information processing flow according to the correlation analysis results.

[0045] In summary, with the help of the above-mentioned technical scheme of the present invention, the present invention can create and adjust the information classification architecture according to the needs of users, and design label categories. By using natural language processing technology and machine learning algorithms, the information can be effectively classified. In addition, the performance of the classification model will be evaluated, and the model will be optimized according to the evaluation results to improve the accuracy of classification. Finally, by applying the optimized model to actual scenarios, real-time classification of newly collected text data is achieved to meet the real-time needs of users; the present invention uses multimodal feature fusion technology to process non-text information. This method can better utilize various types of information such as images, audio and video, and improve the processing ability of non-text information; it not only extracts the visual features of the image, the sound features of the audio, and the visual and audio features of the video, but also these features The present invention integrates and creates a comprehensive feature vector, which can better represent and utilize non-text information; constructs a knowledge graph using the fused features, and annotates the spatiotemporal entity information in the features during the construction process, which can better utilize the entity relationship knowledge in the non-text information; the present invention also trains the classifier, uses the cross-validation method to optimize the classification parameters, and tests the classifier through the data set to evaluate its classification performance, which can better optimize the model and improve the classification accuracy of non-text information, and constructs a multimodal learning framework and a common discriminant subspace. These two construction steps can better integrate and utilize image features and audio features, improve the processing capability of non-text information, and apply the optimized knowledge graph to actual business scenarios, which can directly meet the actual needs of users and improve the practicality of non-text information.

[0046] Although the present invention has been disclosed as above with preferred embodiments, the embodiments are merely examples for the convenience of description and are not intended to limit the present invention. Those skilled in the art may make several changes and modifications without departing from the spirit and scope of the present invention. The scope of protection claimed by the present invention shall be based on the claims.

Claims

1. An information processing method based on multiple information sources, characterized in that: The information processing method comprises the following steps: S1. Obtain the source and content of information, determine the type of information, and extract time, location and entity information; S2. Build an information classification framework based on the use requirements of information, design label categories for information types, and classify them using preset classification rules; S3. Use multimodal feature fusion technology to process non-text information, and extract entity relationship knowledge of non-text information through knowledge graph, and extract spatiotemporal entity information at the same time; S4. Mining the connections between information based on spatiotemporal entity information, integrating cross-source information, building an information association graph, and performing association analysis based on the information association graph, while providing a visual display interface for association analysis; S5. Provide intelligent application services and optimize information processing procedures based on the results of correlation analysis.

2. The information processing method based on multiple information sources according to claim 1, characterized in that: The steps of building an information classification framework for information types based on information usage requirements, designing label categories for information types, and classifying information types using preset classification rules include the following steps: S21. Communicate with users, obtain their needs, analyze their feedback, and determine the purpose of information; S22. Create an information classification framework based on information usage requirements and compare it with user needs; S23. Adjust the information classification framework based on user feedback and design label categories for each category; S24. Classify information types using natural language processing technology according to preset classification rules and label categories.

3. The information processing method based on multiple information sources according to claim 2, characterized in that: The method of classifying information types by using natural language processing technology according to preset classification rules and label categories includes the following steps: S241. Formulate classification rules and label categories based on information usage requirements and domain knowledge; S242, using a word embedding model to convert text data in the information category into an initial feature vector; S243, using information gain on the initial feature vocabulary to determine the final feature word, and then calculating the value of the feature word to form a final feature vector; S244, dividing the final feature vector data set into a training set and a test set; S245, selecting a machine learning algorithm, and training a classification model using the feature vectors and label categories of the training set; S246, using the test set to evaluate the performance of the classification model, using the evaluation index to evaluate the performance of the classification model, and optimizing the parameters of the classification model according to the evaluation results; S247. Apply the optimized classification model to actual scenarios and classify the collected new text data into label categories.

4. The information processing method based on multiple information sources according to claim 3, characterized in that: The formula for calculating the value of the feature word is: ； In the formula, Represents the feature words in text data The characteristic value of represents the weighting factor; Represents the data point in the feature word The set of all connected data points; Represents a data point in text data With data points Weights of connecting lines; represents the denominator of the normalized formula; represents the weighted value; Represents the feature words in text data Medium data point Frequency of occurrence.

5. The information processing method based on multiple information sources according to claim 1, characterized in that: The method of processing non-text information by using multimodal feature fusion technology, extracting entity relationship knowledge of non-text information through knowledge graph, and extracting spatiotemporal entity information includes the following steps: S31, preprocessing non-text data, wherein the non-text data includes images, audio and video; S32, extracting visual features of the image, sound features of the audio, and visual and audio features of the video; S33, integrating the visual features of the image, the sound features of the audio, and the visual and audio features of the video through multimodal feature fusion technology; S34, constructing a knowledge graph using the fused features, and annotating the spatiotemporal entity information in the features during the construction process; S35, fusing the entity information extracted from the non-text data with the spatiotemporal entity information in the knowledge graph to obtain the final knowledge graph; S36. Verify the final knowledge graph and optimize the knowledge graph according to the verification results; S37. Apply the optimized knowledge graph to business scenarios.

6. The information processing method based on multiple information sources according to claim 5, characterized in that: The method of integrating the visual features of the image, the sound features of the audio, and the visual and audio features of the video by the multimodal feature fusion technology comprises the following steps: S331, extracting texture and edge features of the image using a Gabor filter; S332, collecting audio signals and extracting audio features through Mel-frequency cepstral coefficients; S333, combining the texture and edge features of the image with the features of the audio, and extracting dynamic texture features and audio-visual synchronization features; S334, construct a multimodal learning framework, integrate image features and audio features, and create a comprehensive feature vector, merging different types of features according to a preset ratio; S335, constructing a common discriminant subspace using a statistical learning method, and solving an optimal projection matrix according to the eigenvalues ​​and eigenvectors of the samples; S336, using a classifier in the constructed common discriminant subspace to classify the integrated features; S337, training the classifier, optimizing the classification parameters using the cross-validation method, and testing the classifier using the data set to evaluate its classification performance; S338. Analyze the impact of different modal features on classification results, perform feature selection and classifier optimization, and analyze the applicability of multimodal fusion models.

7. The information processing method based on multiple information sources according to claim 6, characterized in that: The multimodal learning framework is constructed to integrate image features and audio features, create a comprehensive feature vector, and merge different types of features according to a preset ratio, including the following steps: S3341. Allocate weights to the image features and the audio features according to their contribution degrees. S3342, integrating the weighted image features and audio features by weighted summation to obtain a comprehensive feature vector; S3343, constructing a multimodal learning framework using comprehensive feature vectors; S3344. Determine the number of nodes in the input layer, the hidden layer, and the output layer according to the multimodal learning framework, and select a loss function for the task; S3345. Use the data in the training set to train the multimodal learning framework, update the parameters of the multimodal learning framework through the back-propagation algorithm, evaluate the performance indicators of the multimodal learning framework on the validation set, and adjust the structure of the multimodal learning framework according to the evaluation results.

8. The information processing method based on multiple information sources according to claim 7, characterized in that: The method of constructing a common discriminant subspace by using a statistical learning method and solving an optimal projection matrix according to the eigenvalues ​​and eigenvectors of the samples comprises the following steps: S3351, collecting image feature and audio feature data after weight allocation; S3352, preprocessing image features and audio features; S3353, projecting the preprocessed image features and audio features into their respective subspaces, and calculating the intra-class scatter matrix and the inter-class scatter matrix according to the category information; S3354, transforming the intra-class scatter matrix and the inter-class scatter matrix to obtain a new discriminant matrix; S3355. Calculate the eigenvalues ​​and eigenvectors of the new discriminant matrix; S3356. Select the optimal projection matrix according to the eigenvalues ​​and eigenvectors of the discriminant matrix.

9. The information processing method based on multiple information sources according to claim 1, characterized in that: The method of mining the connection between information according to the spatiotemporal entity information, fusing cross-source information, building an information association graph, performing association analysis based on the information association graph, and providing a visual display interface for the association analysis includes the following steps: S41, collecting spatiotemporal entity information from different sources, wherein the spatiotemporal entity information at least includes time, place, person, and event; S42, preprocessing the time, place, person, and event of the spatiotemporal entity information, extracting the time, place, person, and event of the spatiotemporal entity information and the relationship between the time, place, person, and event of the spatiotemporal entity information; S43. Use knowledge graph technology to connect the same spatiotemporal entity information in different information sources to form a unified view; S44, constructing a heterogeneous information network based on the time, place, person, event of the empty entity information and the relationship between the time, place, person, event of the spatiotemporal entity information, wherein the nodes of the heterogeneous information network represent entities and the edges of the heterogeneous information network represent the relationship between the entities; S45, performing association analysis on heterogeneous information networks, finding paths in the network according to the analysis target, and obtaining associations between different information sources; S46. Use graph analysis algorithms to find entities and paths in heterogeneous information networks; S47. Provide a visual display interface to visualize the topological structure, key entities and key paths of heterogeneous information networks.

10. An information processing system based on multiple information sources, used to implement the information processing method based on multiple information sources according to any one of claims 1 to 9, characterized in that: The system includes: information acquisition and content analysis module, information classification and label design module, non-text information processing and entity relationship extraction module, association analysis module and intelligent application service module; The information acquisition and content analysis module is connected with the non-text information processing and entity relationship extraction module through the information classification and label design module, and the non-text information processing and entity relationship extraction module is connected with the intelligent application service module through the association analysis module; Information acquisition and content analysis module, used to obtain the source and content of information, determine the type of information, and extract time, location and entity information; The information classification and label design module is used to build an information classification framework for information types according to the use requirements of the information, design label categories for information types, and classify them according to preset classification rules; The non-text information processing and entity relationship extraction module is used to process non-text information using multimodal feature fusion technology, extract entity relationship knowledge from non-text information through knowledge graphs, and extract spatiotemporal entity information at the same time; The association analysis module is used to mine the relationship between information based on spatiotemporal entity information, perform cross-source information fusion, build an information association graph, and perform association analysis based on the information association graph, while providing a visual display interface for association analysis; The intelligent application service module is used to provide intelligent application services and optimize the information processing process based on the correlation analysis results.

Images