Multi-modal reasoning chain generation method fusing industry knowledge base

Abstract

The invention provides a multi-modal reasoning chain generation method fusing an industry knowledge base, and relates to the crossing field of artificial intelligence and coal industry application. The method comprises the following steps: firstly, receiving multi-modal data of a coal mine site, carrying out feature extraction on each modal data to obtain a first feature corresponding to each modal data, then carrying out time sequence alignment and semantic alignment on the first feature corresponding to each modal data to obtain an aligned second feature of each modal data, and then fusing the second features of each modal data to obtain a multi-modal data fusion model; and finally, based on the scene representation features, retrieving knowledge related to the scene representation features from a preset knowledge base, and based on the scene representation features and the related knowledge, generating a reasoning chain. Therefore, by integrating the multi-modal data and professional knowledge of the coal industry, the accuracy and reliability of the generated reasoning chain are improved, and reliable intelligent decision support is provided for safety production of the coal industry.

Description

Technical Field

[0001] The present disclosure relates to the intersection of artificial intelligence and coal industry applications, and in particular to a method for generating a multimodal reasoning chain that integrates an industry knowledge base. Background Art

[0002] With the deep integration of information technology and the coal industry, intelligent coal mine construction has become a key focus of industry development. Currently, coal mines use multi-source, heterogeneous data collection methods such as video surveillance, sensor monitoring, and inspection reports to accumulate a vast amount of safety production standards, accident case knowledge bases, and real-time monitoring data. However, this multimodal data is typically stored and used independently, making it difficult to form a complete understanding of the situation. Furthermore, the reasoning process based on this data lacks industry knowledge support, resulting in low accuracy and reliability of the inference results. Summary of the Invention

[0003] The present disclosure provides a method for generating a multimodal reasoning chain by integrating an industry knowledge base.

[0004] According to one aspect of the present disclosure, a method for generating a multimodal reasoning chain integrating an industry knowledge base is provided, comprising: Receiving multimodal data from a coal mine site, performing feature extraction on each modal data, and obtaining a first feature corresponding to each modal data; Performing temporal alignment and semantic alignment on the first features corresponding to each of the modal data to obtain aligned second features of each of the modal data; fusing the second features of the modal data to generate a scene representation feature; Based on the scene representation feature, retrieving knowledge related to the scene representation feature from a preset knowledge base; An inference chain is generated based on the scene representation features and the related knowledge.

[0005] A method for generating a multi-modal inference chain integrating an industry knowledge base provided by the present disclosure. First, multi-modal data at a coal mine site is received, feature extraction is performed on each modal data to obtain a first feature corresponding to each modal data, then the first features corresponding to each modal data are temporally aligned and semantically aligned to obtain second features after alignment of each modal data, and then the second features of each modal data are fused to generate a scene representation feature. Finally, based on the scene representation feature, knowledge related to the scene representation feature is retrieved from a preset knowledge base, and an inference chain is generated based on the scene representation feature and the related knowledge. Thus, by temporally and semantically aligning the multi-modal data at the coal mine site and generating a scene representation feature, retrieving the industry knowledge base based on the scene representation feature to obtain relevant knowledge, and finally generating an inference chain based on the scene representation feature and the relevant knowledge, the multi-modal data and professional knowledge in the coal industry are effectively integrated, the accuracy and reliability of the generated inference chain are improved, and reliable intelligent decision-making support is provided for the safe production of the coal industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0007] Figure 1 It is a schematic flowchart of a method for generating a multi-modal inference chain integrating an industry knowledge base provided by an embodiment of the present disclosure; Figure 2 It is a schematic flowchart of a method for generating a multi-modal inference chain integrating an industry knowledge base provided by an embodiment of the present disclosure; Figure 3 It is a schematic flowchart of a method for generating a multi-modal inference chain integrating an industry knowledge base provided by an embodiment of the present disclosure; Figure 4 It is a schematic flowchart of a method for generating a multi-modal inference chain integrating an industry knowledge base provided by an embodiment of the present disclosure; Figure 5 It is a schematic flowchart of a method for generating a multi-modal inference chain integrating an industry knowledge base provided by an embodiment of the present disclosure; Figure 6 It is a schematic flowchart of a method for generating a multi-modal inference chain integrating an industry knowledge base provided by an embodiment of the present disclosure; Figure 7 It is a schematic system architecture diagram of a method for generating a multi-modal inference chain integrating an industry knowledge base proposed by an embodiment of the present disclosure.

[0008] Through the above-mentioned drawings, specific embodiments of the present disclosure have been shown and will be described in more detail hereinafter. These drawings and the textual description are not intended to limit the scope of the concept of the present disclosure in any way, but to illustrate the concept of the present disclosure to those skilled in the art by referring to specific embodiments. Detailed Description of the Embodiments

[0009] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the multi-modal inference chain generation method that integrates industry knowledge bases proposed by the present disclosure.

[0010] It should be noted that in the technical solution of the present disclosure, the acquisition, transmission, storage, use, processing, etc. of data all comply with the relevant regulations of laws and regulations.

[0011] It should be noted that in the embodiments of the present disclosure, some industry-existing solutions such as certain software, components, models, etc. may be mentioned. They should be regarded as exemplary, and their purpose is only to illustrate the feasibility in the implementation of the technical solution of the present disclosure, but it does not mean that the present disclosure has already or necessarily used this solution.

[0012] Next, with reference to the drawings, the multi-modal inference chain generation method that integrates industry knowledge bases in the embodiments of the present disclosure will be described in detail.

[0013] <http: / / www.sxpc.edu.cn / info / 1021 / 3048.htm> Figure 1 It is a schematic flowchart of a multi-modal inference chain generation method that integrates industry knowledge bases provided for the embodiments of the present disclosure.

[0014] As Figure 1 shown, the method includes: Step 101, receiving multi-modal data from the coal mine site, extracting features from each modal data, and obtaining the first feature corresponding to each modal data.

[0015] Among them, the multi-modal data may include modal data such as video data, audio data, image data, sensor data, and text data. For example, the multi-modal data may include surveillance videos of the coal mine site, vibration sensor data, equipment operation sounds, equipment images, and text reports recorded by operators, etc. The present disclosure does not limit this.

[0016] In the present disclosure, when extracting features from each modal data, a pre-trained modal-specific encoder can be used to extract the features of each modal data.

[0017] For example, for video data, a spatio-temporal convolutional network (3D-CNN) or a video Transformer architecture can be used to extract dynamic features , which is the first feature of the video data, as shown in the following formula (1): , (1) where is the feature dimension, is the time step, is the modal data.

[0018] Among them, 3D-CNN is the abbreviation of 3D Convolutional Neural Networks.

[0019] For example, for image data, a convolutional neural network (ResNet / EfficientNet) can be used to extract spatial features , which is the first feature of the image data, as shown in the following formula (2): , (2) Among them, ResNet is the abbreviation of Residual Network; EfficientNet is the abbreviation of Efficient Network.

[0020] For example, for sensor data, a temporal convolutional network (TCN) or an LSTM network can be used to extract temporal features , which is the first feature of the sensor data, as shown in the following formula (3): , (3) Among them, TCN is the abbreviation of Temporal Convolutional Network; LSTM is the abbreviation of Long Short-Term Memory.

[0021] For example, for text data, a pre-trained language model (such as BERT) can be used to extract semantic features , which is the first feature of the text data, as shown in the following formula (4): , (4) Among them, BERT is the abbreviation of Bidirectional Encoder Representations from Transformers.

[0022] That is to say, when extracting features from each modality data, the mathematical expression can be as shown in the following formula (5): , (5) Among them, represents the encoder corresponding to the modality ; represents the data of the modality ; represents the first feature corresponding to the modality . respectively represent video, image, sensor, and text.

[0023] Step 102: Perform temporal alignment and semantic alignment on the first features corresponding to each modality data to obtain the aligned second features of each modality data.

[0024] Among them, the second feature is the feature with both temporal and semantic alignment of each modality data.

[0025] In the present disclosure, after obtaining the first features of each modality data, in order to effectively avoid the problems of isolation and fragmentation of single modality data, the multi-modal data can be fused. Before fusion, in order to improve the accuracy of fusion, the first features corresponding to each modality data can be temporally aligned and semantically aligned, so as to improve the perception accuracy and integrity of the coal mine site conditions, enabling the staff to comprehensively consider various data factors when analyzing coal mine production, safety, management and other issues, and improving the accuracy and reliability of decision-making.

[0026] Step 103: Fuse the second features of each modality data to generate a scene representation feature.

[0027] Among them, the scene representation feature can be a feature representing semantics, context, and relationships between different elements in the scene, etc. For example, the scene representation feature can include information features such as environmental parameters and equipment status, and the present disclosure does not limit this.

[0028] In the present disclosure, after obtaining the aligned second features of each modality data, the second features of each modality data can be fused to generate a scene representation feature, thereby providing a data basis for the safe production and intelligent management of the coal mine.

[0029] Step 104: Retrieve knowledge related to the scene representation feature from a preset knowledge base based on the scene representation feature.

[0030] Among them, the knowledge base can be a pre-set knowledge base for the coal industry, which may include relevant knowledge such as safety standards, equipment specifications, and fault cases in the coal industry. The present disclosure does not limit this.

[0031] In the present disclosure, after generating the scene representation features, relevant knowledge related to the scene representation features can be retrieved from the pre-set knowledge base, so as to provide a basis for ensuring that the generated inference chain conforms to industry standards and specifications.

[0032] Optionally, when retrieving knowledge related to the scene representation features from the knowledge base, the scene representation features can be determined first and the semantic similarity with each piece of knowledge in the knowledge base, as shown in formula (6): , (6) wherein, and are respectively the encoding functions of the scene and the knowledge.

[0033] Then, sort the semantic similarity between the scene representation features and each piece of knowledge , and take the piece of knowledge with the highest similarity as the knowledge related to the scene representation features , as shown in formula (7): , (7) wherein, n can be determined according to actual needs, and the present disclosure does not limit this.

[0034] Step 105: Generate an inference chain based on the scene representation features and the relevant knowledge.

[0035] It should be noted that the specific structure of the inference chain can be determined according to needs. For example, the structure of the inference chain generated in the present disclosure can be a quadruple, including four links: observation, analysis, inference, and conclusion. For example, the generated inference chain can be "observing that the vibration value of the belt conveyor exceeds the standard and the temperature rises → analyzing that the bearing is severely worn → inferring that it is caused by insufficient lubrication → concluding that immediate shutdown inspection and bearing replacement are required", etc. The present disclosure does not limit this.

[0036] In the present disclosure, after retrieving the relevant knowledge, in order to improve the accuracy and reliability of the generated inference chain, an inference chain can be generated based on the scene representation features and the relevant knowledge.

[0037] In the embodiments of the present disclosure, first, multi-modal data at the coal mine site is received, feature extraction is performed on each modal data to obtain a first feature corresponding to each modal data, then the first features corresponding to each modal data are aligned in time sequence and semantic alignment to obtain a second feature after alignment of each modal data, and then the second features of each modal data are fused to generate a scene representation feature. Finally, based on the scene representation feature, knowledge related to the scene representation feature is retrieved from a preset knowledge base, and an inference chain is generated based on the scene representation feature and the related knowledge. Thus, by aligning the time sequence and semantics of the multi-modal data at the coal mine site, generating a scene representation feature, retrieving the industry knowledge base based on the scene representation feature, obtaining relevant knowledge, and finally generating an inference chain based on the scene representation feature and the relevant knowledge, the multi-modal data and professional knowledge in the coal industry are effectively integrated, the accuracy and reliability of the generated inference chain are improved, and reliable intelligent decision-making support is provided for the safe production of the coal industry.

[0038] Figure 2 It is a schematic flowchart of a multi-modal inference chain generation method integrating an industry knowledge base provided by an embodiment of the present disclosure.

[0039] As Figure 2 shown, the method includes: Step 201: Receive multi-modal data at the coal mine site, perform feature extraction on each modal data, and obtain a first feature corresponding to each modal data.

[0040] Among them, for the specific implementation form of step 201, reference can be made to the detailed descriptions in other embodiments of the present disclosure, and no specific elaboration will be provided here.

[0041] Step 202: Convert the timestamps of each modal data to a unified time reference, perform time sequence alignment on each first feature, and obtain a third feature after time sequence alignment of each modal data.

[0042] Among them, the time reference is a time standard commonly referred to by the timestamps of each modal data, which can be Coordinated Universal Time (UTC), local time, etc., and the present disclosure does not limit this.

[0043] Optionally, after converting the timestamps of each modal data to a unified time reference, when performing time sequence alignment on each first feature, it may include at least one of the following: Identify key event points in each modal data, and based on the key event points, perform time sequence alignment on each first feature; for modal data with different sampling frequencies, through interpolation or resampling, align the time sequences of the first features of each modal data.

[0044] Among them, the key event point can be pre-set, or can also be determined according to actual needs. For example, the key event point can be an abnormal event point, etc., which is not limited in this disclosure.

[0045] In the present disclosure, after converting the timestamps of each modal data into a unified time base, the key event points in each modal data can be identified, and the key event points can be used as the alignment points for each modal data, so that the first features of each modal data can be time-series aligned based on the key event points.

[0046] In the present disclosure, after the timestamps of each modal data are converted into a unified time base, since the sampling frequencies of the data in the multimodal data may be different, such as the sampling frequency of the video data may be higher than the sampling frequencies of other modal data, therefore, for modal data with different sampling frequencies, the timing of the first features of each modal data can be aligned by difference or resampling. For example, in order to time-align each modal data with the video data, each modal data can be interpolated or resampled to achieve time-alignment with the video data. The present disclosure does not limit this.

[0047] It should be noted that when timing alignment is performed on the first feature of each modal data, timing alignment can be performed through key event points, or by interpolating or resampling data with different sampling frequencies, or by key time points, and by interpolating or resampling data with different sampling frequencies, etc. The present disclosure does not limit this.

[0048] In this disclosure, the timing alignment process can be expressed as follows: , (8) in, Indicates modality The corresponding feature representation is the first feature, Indicates modality time series, represents the timing alignment function, The feature representation after time alignment is the third feature.

[0049] Step 203 : Based on each modality data, all possible ordered modality pairs are generated, wherein the ordered modality pairs include different modalities, the first modality in the modality pair is the source modality, and the second modality is the target modality.

[0050] Among them, the source modality can be the modality that provides features in the feature fusion process of multimodal data; the target modality can be the modality that receives and converts the source modality information in the feature fusion process of multimodal data.

[0051] In the present disclosure, after temporal alignment of each modal data, in order to reliably semantically align each modal data, all possible ordered modal pairs can be generated based on each modal data.

[0052] Step 204: Determine the attention weight corresponding to each modality pair based on the third feature of the modality data contained in each modality pair.

[0053] In the present disclosure, after generating all possible modality pairs, the attention weight corresponding to each modality pair can be determined based on the third feature of the modality data contained in each modality pair. For example, the source modality in the modality pair is , the target mode is Take this as an example to illustrate how to calculate the attention weight. The formula can be shown as follows (9): , (9) in, and are the weight matrices for query and key, respectively, and is the parameter matrix used to map the features after alignment of the corresponding modalities into the vector space; is the normalization function; is the vector dimension.

[0054] In step 205 , based on the attention weights of each modality pair, the third feature of the target modality in each modality pair is fused with the third feature of the corresponding source modality to generate a fused semantic feature corresponding to each modality pair.

[0055] In the present disclosure, after determining the attention weights of each modality pair, the third feature of the target modality in each modality pair can be fused with the third feature of the corresponding source modality based on the attention weights of each modality pair to generate the corresponding fused semantic features of each modality pair, thereby completing the semantic association of the modality data in each modality pair. Taking the above example as an example, the fused semantic features of the modality pair are generated. The formula can be shown as follows (10): , (10) in, is the weight matrix of value.

[0056] In step 206 , for each modal data, the fused semantic features of all modal pairs of the target modality are integrated through a multi-head attention mechanism to obtain the second feature of each modal data.

[0057] In the present disclosure, after generating the fused semantic features corresponding to each modality pair, for each modality data, through the multi-head attention mechanism, the fused semantic features of all modality pairs of the target modality can be integrated as the second feature of each modality data. Taking the above example, the formula for integrating the fused semantic features of modality data pairs through the multi-head attention mechanism can be shown as the following formula (11): , (11) where is a linear transformation matrix for compressing the integrated feature matrix to a preset dimension. , is the second feature of modality .

[0058] where the calculation expression of each attention head is shown as the following formula (12):

[0059] , (12) Step 207: Fuse the second features of each modality data to generate a scene representation feature.

[0060] Step 208: Based on the scene representation feature, retrieve knowledge related to the scene representation feature from a preset knowledge base.

[0061] Step 209: Generate an inference chain based on the scene representation feature and the related knowledge.

[0062] Among them, the specific implementation forms of Step 207 to Step 209 can refer to the detailed descriptions in other embodiments of the present disclosure and will not be specifically elaborated here.

[0063] In the embodiments of the present disclosure, first, multi-modal data at the coal mine site is received, feature extraction is performed on each modal data to obtain a first feature corresponding to each modal data, and the timestamps of each modal data are converted into a unified time reference. Then, temporal alignment is performed on each first feature to obtain a third feature after temporal alignment of each modal data. Next, all possible ordered modal pairs are generated based on each modal data, and the attention weight corresponding to each modal pair is determined based on the third features of the modal data included in each modal pair. After that, based on the attention weights of each modal pair, the third feature of the target modal in each modal pair is respectively fused with the third feature of the corresponding source modal to generate a fused semantic feature corresponding to each modal pair. For each modal data, through a multi-head attention mechanism, the fused semantic features of all modal pairs with it as the target modal are integrated to obtain a second feature of each modal data. Finally, the second features of each modal data are fused to generate a scene representation feature, and based on the scene representation feature, knowledge related to the scene representation feature is retrieved from a preset knowledge base, and an inference chain is generated based on the scene representation feature and the related knowledge. Thus, after feature extraction of multi-modal data, by converting the timestamps of multi-modal data into a unified time reference, temporal alignment of the features of multi-modal data is performed. Then, all possible modal pairs are generated, the attention weight of each modal pair is determined, based on the attention weight, the fused semantic feature of the corresponding modal pair is determined, and through a multi-head attention mechanism, the fused semantic features of each modal fused with other modalities are integrated to obtain features with both temporal and semantic alignment of each modal data, effectively avoiding the problems of isolation and fragmentation of single-modal data, improving the perception accuracy and integrity of the coal mine site conditions, and providing conditions for improving the accuracy and reliability of multi-modal inference chain generation.

[0064] Figure 3 It is a schematic flowchart of a method for generating a multi-modal inference chain integrating an industry knowledge base provided by an embodiment of the present disclosure.

[0065] As Figure 3 shown, the method includes: Step 301, receive multi-modal data at the coal mine site, perform feature extraction on each modal data, and obtain a first feature corresponding to each modal data.

[0066] Step 302, perform temporal alignment and semantic alignment on the first feature corresponding to each modal data to obtain a second feature after alignment of each modal data.

[0067] Among them, for the specific implementation forms of steps 301 to 302, reference can be made to the detailed descriptions in other embodiments of the present disclosure, and details will not be elaborated here.

[0068] Step 303, obtain the scene associated with the coal mine site.

[0069] It should be noted that the scenarios associated with the coal mine site can be determined according to the actual situation. For example, the scenarios associated with the coal mine site can be production scenarios, transportation scenarios, personnel operation scenarios, etc., and the present disclosure does not limit this.

[0070] In the present disclosure, after obtaining the second feature with temporal and semantic alignment of multi-modal data, before generating the scene representation feature, the scenarios associated with the coal mine site can be first obtained, so as to provide conditions for improving the perception accuracy and integrity of the coal mine site.

[0071] Step 304, based on the scenario, determine the importance of each modal data.

[0072] Among them, the importance can be used to represent the importance degree of each modal data in the scenario.

[0073] Optionally, before determining the importance of each modal data based on the scenario, the second feature of each modality can be normalized to the same feature space through a feature normalization operation.

[0074] Step 305, based on the importance of each modal data, fuse the second features of each modal data to generate a scene representation feature.

[0075] In the present disclosure, by fusing the second features of each modal data based on the importance of each modal data to generate a scene representation feature, the generated scene representation feature can be made more in line with the scenario, thereby improving the accuracy and reliability of retrieving relevant knowledge from the knowledge base. Based on the importance of each modal data , fuse the second features of each modal data , generate a scene representation feature The formula can be as shown in the following formula (13): , (13) Step 306, based on the scene representation feature, retrieve knowledge related to the scene representation feature from a preset knowledge base.

[0076] Among them, for the specific implementation form of step 306, reference can be made to the detailed descriptions in other embodiments of the present disclosure, and details will not be elaborated here.

[0077] Step 307, fuse the scene representation feature with the relevant knowledge to obtain a context feature.

[0078] In the present disclosure, after retrieving the knowledge related to the scene representation feature from the knowledge base, in order to improve the accuracy and reliability of the generated inference chain, before generating the inference chain, the scene representation feature can be first fused with the relevant knowledge to obtain a context feature , which can be as shown in the following formula (14): , (14) Among them, represents the feature splicing operation.

[0079] Step 308, determine the structure of the inference chain.

[0080] Among them, the structure of the inference chain can be determined according to actual needs. For example, the structure of the inference chain generated in this disclosure can be a quadruple , which respectively represent observation, analysis, inference, and conclusion. This disclosure does not make any limitations on this.

[0081] Step 309, generate an inference chain based on the context features and the structure.

[0082] In this disclosure, after determining the structure of the inference chain, based on the context features and the structure, a conditional generation model based on Transformer can be used to sequentially generate each component of the inference chain, as shown in the following formula (15): , (15) Optionally, after generating the inference chain, professional term recognition and replacement can also be performed to enhance the professionalism of the inference chain in the industry.

[0083] In the embodiment of this disclosure, first, multi-modal data of the coal mine site is received, feature extraction is performed on each modal data to obtain the first feature corresponding to each modal data, and the first features corresponding to each modal data are subjected to temporal alignment and semantic alignment to obtain the aligned second features of each modal data. Then, the scenes associated with the coal mine site are obtained, and based on the scenes, the importance of each modal data is determined. After that, based on the importance of each modal data, the second features of each modal data are fused to generate a scene representation feature, and based on the scene representation feature, knowledge related to the scene representation feature is retrieved from a preset knowledge base. Finally, the scene representation feature and the related knowledge are fused to obtain context features, the structure of the inference chain is determined, and an inference chain is generated based on the context features and the structure. Thus, after performing temporal and semantic alignment on the multi-modal data of the coal mine site, based on the scenes associated with the coal mine site, the importance of each modal data is determined, and based on the importance of each modal data, the aligned features of the multi-modal data are fused to generate a scene representation feature, and the scene representation feature is fused with the related knowledge in the knowledge base to generate context features, and an inference chain is generated based on the context features and the structure of the inference chain, thereby improving the accuracy and reliability of the generated inference chain, and further providing accurate and reliable decision support for the safe production of the coal industry.

[0084] Figure 4 It is a schematic flowchart of a multi-modal inference chain generation method integrating an industry knowledge base provided by an embodiment of this disclosure.

[0085] As Figure 4 shown, the method includes: Step 401: Receive multi-modal data from the coal mine site, extract features from each modal data, and obtain the first feature corresponding to each modal data.

[0086] Step 402: Align the first features corresponding to each modal data in time series and semantically to obtain the aligned second features of each modal data.

[0087] Step 403: Fuse the second features of each modal data to generate a scene representation feature.

[0088] Step 404: Based on the scene representation feature, retrieve knowledge related to the scene representation feature from a preset knowledge base.

[0089] Step 405: Generate an inference chain based on the scene representation feature and the related knowledge.

[0090] Among them, for the specific implementation forms of steps 401 to 405, reference may be made to the detailed descriptions in other embodiments of the present disclosure, and details are not described herein again.

[0091] Step 406: Evaluate at least one of the following items of the inference chain: logical integrity, specification consistency, and uncertainty quantification.

[0092] Among them, logical integrity is used to verify whether the causal relationship in the inference chain is complete.

[0093] Among them, specification consistency is industry specification consistency, which is used to check whether the inference chain complies with industry safety standards and operation specifications.

[0094] Among them, uncertainty quantification is used to quantitatively evaluate the possible uncertainties in the inference chain, as shown in the following formula (16): , (16) Among them, represents the number of knowledge evidences in the knowledge base that support the inference , that is, the number of knowledge related to the inference in the knowledge base.

[0095] Step 407: In the case that any of the evaluations fails, return to execute the operation of retrieving knowledge related to the scene representation feature from the preset knowledge base based on the scene representation feature until all evaluations pass to obtain the target inference chain.

[0096] Among them, the target inference chain is the final inference chain corresponding to the multi-modal data at the coal mine site, and it is an inference chain for which all evaluations pass.

[0097] In the present disclosure, in the case where any one of the evaluations fails, it can be determined that the quality of the currently generated inference chain is low. At this time, in order to obtain a higher-quality inference chain, the operation of retrieving knowledge related to the scenario representation feature from the preset knowledge base based on the scenario representation feature can be returned for execution until all evaluations pass, and it can be determined that the quality of the currently generated inference chain is high, thereby obtaining the target inference chain.

[0098] In an embodiment of the present disclosure, first, multi-modal data of a coal mine site is received, feature extraction is performed on each modal data to obtain a first feature corresponding to each modal data, and the first features corresponding to each modal data are subjected to temporal alignment and semantic alignment to obtain a second feature after alignment of each modal data. Then, the second features of each modal data are fused to generate a scenario representation feature, and based on the scenario representation feature, knowledge related to the scenario representation feature is retrieved from the preset knowledge base. After that, based on the scenario representation feature and the related knowledge, an inference chain is generated, and at least one of the following of the inference chain is evaluated: logical integrity, specification consistency, and uncertainty quantification. Finally, in the case where any one of the evaluations fails, the operation of retrieving knowledge related to the scenario representation feature from the preset knowledge base based on the scenario representation feature is returned for execution until all verifications pass, and the target inference chain is obtained. Thus, after the inference chain is generated, in order to determine the reliability and accuracy of the inference chain, the logical integrity, specification consistency, and uncertainty quantification of the inference chain can be evaluated, and in the case where any one of the evaluations fails, the inference chain is regenerated and evaluated again until all evaluations pass, and it can be determined that the quality of the currently generated inference chain is high. At this time, the final inference chain of the coal mine site is obtained, thereby improving the accuracy and quality of the generated inference chain.

[0099] Figure 5 It is a schematic flowchart of a multi-modal inference chain generation method integrating an industry knowledge base provided by an embodiment of the present disclosure.

[0100] As Figure 5 shown, the method includes: Step 501, receiving multi-modal data of a coal mine site, performing feature extraction on each modal data, and obtaining a first feature corresponding to each modal data.

[0101] Step 502, performing temporal alignment and semantic alignment on the first features corresponding to each modal data to obtain a second feature after alignment of each modal data.

[0102] Step 503, fusing the second features of each modal data to generate a scenario representation feature.

[0103] Step 504, retrieving knowledge related to the scenario representation feature from the preset knowledge base based on the scenario representation feature.

[0104] Step 505: Generate an inference chain based on the scenario representation features and related knowledge.

[0105] For the specific implementation forms of steps 501 to 505, reference may be made to the detailed descriptions in other embodiments of the present disclosure, and details are not described herein again.

[0106] Step 506: Send the inference chain to the user's terminal and obtain the user's feedback, where the feedback includes the knowledge corrected by the user.

[0107] It should be noted that the user's feedback may also include the user's satisfaction score for the inference chain, the reason for correction, and the feedback type, such as knowledge missing type, inference error type, and inaccurate expression type, etc. The present disclosure does not make any limitations in this regard.

[0108] In the present disclosure, after generating the inference chain, the inference chain can be sent to the supporting applications of the inference chain in the user's terminal, such as applications for safety production monitoring, fault diagnosis, and early warning disposal, etc., for the user's reference. The present disclosure does not make any limitations in this regard.

[0109] Step 507: Determine the first knowledge vector corresponding to the corrected knowledge.

[0110] The first knowledge vector may be a high-dimensional knowledge vector corresponding to the corrected knowledge.

[0111] In the present disclosure, after obtaining the user's feedback on the inference chain, the relevant knowledge in the knowledge base can be updated based on the user's feedback, so as to realize the dynamic update of the knowledge base, improve the timeliness and reliability of the knowledge base. At this time, the first knowledge vector corresponding to the knowledge corrected by the user can be determined first.

[0112] Step 508: Locate the corresponding knowledge in the knowledge base based on the first knowledge vector and perform content replacement to complete the update of the knowledge base.

[0113] In the present disclosure, after determining the first knowledge vector, the corresponding knowledge in the knowledge base can be located first based on the first knowledge vector, and then this part of the knowledge can be replaced with the first knowledge vector, so as to complete the update of the knowledge base to ensure the timeliness and accuracy of the knowledge in the knowledge base.

[0114] In the embodiments of the present disclosure, first, multi-modal data at the coal mine site is received, feature extraction is performed on each modal data to obtain a first feature corresponding to each modal data, and the first features corresponding to each modal data are subjected to temporal alignment and semantic alignment to obtain a second feature after alignment of each modal data. Then, the second features of each modal data are fused to generate a scene representation feature, and based on the scene representation feature, knowledge related to the scene representation feature is retrieved from a preset knowledge base. After that, based on the scene representation feature and the related knowledge, an inference chain is generated, and the inference chain is sent to the user's terminal, and user feedback is obtained. Finally, a first knowledge vector corresponding to the corrected knowledge is determined, and based on the first knowledge vector, the corresponding knowledge in the knowledge base is located and content replacement is performed to complete the update of the knowledge base. Thus, after the inference chain is generated, the inference chain can be sent to the user, and user feedback is collected, and the relevant knowledge in the knowledge base is updated based on the user feedback, thereby realizing the dynamic update of the knowledge base and improving the reliability and timeliness of the knowledge base.

[0115] Figure 6 It is a schematic flowchart of a multi-modal inference chain generation method integrating an industry knowledge base provided by an embodiment of the present disclosure.

[0116] As Figure 6 shown, the method includes: Step 601, when new knowledge related to the coal mine site is received through a preset interface, determine a second knowledge vector corresponding to the new knowledge.

[0117] Among them, the new knowledge may include knowledge such as newly added coal industry cases and standards annotated by experts related to the coal industry, and the present disclosure does not limit this.

[0118] Among them, the second knowledge vector may be a high-dimensional knowledge vector corresponding to the new knowledge.

[0119] Among them, the preset interface may be a knowledge standardization interface, which can be used to determine a unified knowledge import protocol and data format, and the present disclosure does not limit this.

[0120] In the present disclosure, when determining the second knowledge vector corresponding to the new knowledge , it can be determined by the following formula (17): , (17) Among them, is a vector embedding function, is the standardized knowledge, is a pre-trained multi-modal encoding model, generating a vector with a dimension of (the vector dimension is usually determined by the semantic richness and Decision).

[0121] Step 602: Determine the retrieval weight of the second knowledge vector.

[0122] In the present disclosure, after determining the second knowledge vector corresponding to the newly added knowledge and before adding the second knowledge vector to the knowledge base, the retrieval weight of the second knowledge vector can be determined using formula (18) according to knowledge timeliness and authority The formula (18) is as follows: , (18) wherein, represents knowledge authority,[[]] is the knowledge age,[[]] and are weighting coefficients and can be set as needed.

[0123] Step 603: Associatively add the second knowledge vector and the retrieval weight to the knowledge base and identify the partition in the knowledge base to which the second knowledge vector belongs.

[0124] It should be noted that the knowledge partitions in the knowledge base can be divided according to actual needs. For example, the partitions in the knowledge base can include industry norms and standards partitions, work safety partitions, environmental protection partitions, technology partitions, and fault case partitions, etc. The present disclosure does not limit this.

[0125] In the present disclosure, when associatively adding the second knowledge vector and the retrieval weight to the knowledge base, the integration and real-time availability of the newly added knowledge can be achieved through the incremental vector update method, as shown in formula (19): , (19) wherein, is the vector database instance,[[]] is the set of second knowledge vectors,[[]] is the timeliness decay factor and can be set as needed.

[0126] In the present disclosure, since the knowledge index in the knowledge base is a partition index, a scalable vector search structure is constructed using the hierarchical quantization technique as shown in formula (20): , (20) wherein, is the Hierarchical Navigable Small Worldgraphs algorithm,[[]] is the set of all knowledge vectors in the knowledge base,[[]] controls the maximum number of connections per node,[[]] ​Control the quality of index construction.

[0127] Therefore, after adding the second knowledge vector associated with the retrieval weight to the knowledge base, it is also necessary to identify the partition of the second knowledge vector in the knowledge base to reconstruct the knowledge index in the corresponding partition.

[0128] Step 604, reconstruct the index of the knowledge vectors in the affiliated partition to complete the update of the knowledge base.

[0129] In the present disclosure, by reconstructing the index of the knowledge vectors in the partition of the second knowledge vector, it effectively avoids reconstructing the index of each knowledge vector in the knowledge base and improves the efficiency of the dynamic update of the knowledge base.

[0130] In the embodiments of the present disclosure, first, when new knowledge related to the coal mine site is received through a preset interface, the second knowledge vector corresponding to the new knowledge is determined, then the retrieval weight of the second knowledge vector is determined, and then the second knowledge vector is associated with the retrieval weight and added to the knowledge base, and the partition of the second knowledge vector in the knowledge base is identified. Finally, the index of the knowledge vectors in the affiliated partition is reconstructed to complete the update of the knowledge base. Thus, the knowledge vector and the retrieval weight of the new knowledge related to the coal mine site are determined and associated and added to the knowledge base, and at the same time, the knowledge index in the partition to which the newly added knowledge vector belongs is reconstructed to complete the update of the knowledge base, thereby improving the efficiency of the dynamic update of the knowledge base and ensuring the accuracy and timeliness of the knowledge base.

[0131] Next, in combination with Figure 7 , an example is given for the system architecture of the multi-modal inference chain generation method integrating the industry knowledge base proposed in the present disclosure. Figure 7 FIG. is a schematic diagram of the system architecture of the multi-modal inference chain generation method integrating the industry knowledge base proposed in the embodiments of the present disclosure.

[0132] As Figure 7 shown, the system architecture of the multi-modal inference chain generation method integrating the industry knowledge base includes: a multi-modal alignment layer, a knowledge enhancement inference layer, and a dynamic iteration layer.

[0133] Among them, the multi-modal alignment layer can be used to perform temporal semantic alignment on multi-modal data and generate scene representation features. It includes a multi-modal feature extraction layer for extracting features from multi-modal data, a temporal aligner for temporally aligning the features of multi-modal data, a semantic alignment network for semantically aligning the temporally aligned features, and a scene semantic fusion device for fusing the temporally semantically aligned features to generate scene representation features.

[0134] The knowledge-enhanced reasoning layer can be used to generate reasoning chains based on scenario representation features and industry knowledge bases. This layer includes a retrieval-enhanced generation module for retrieving relevant knowledge from the industry knowledge base based on scenario representation features, a reasoning chain generator for generating reasoning chains based on scenario representation features and relevant knowledge, and an evaluation module for evaluating the generated reasoning chains. This module can also be an interpretability verification module for verifying the reasoning chains.

[0135] The dynamic iteration layer can be used to dynamically update the knowledge base. It includes a user feedback collector for collecting user feedback on the reasoning chain, a knowledge annotation docking module for receiving new knowledge received by the knowledge annotation tool, and a knowledge base update mechanism module for updating the knowledge base.

[0136] When using a system of the multimodal reasoning chain generation method for integrating an industry knowledge base proposed in the present disclosure, the scene representation features of the coal mine site can be first generated through a multimodal alignment layer. The specific steps are: using a multimodal feature extractor to extract features from the multimodal data (such as video data, image data, sensor data, and text data) of the coal mine site, and then using a temporal aligner and a semantic alignment network in sequence to perform temporal and semantic alignment on the features of the multimodal data to obtain the aligned features, and then using a scene semantic fusion device to fuse the aligned features of the multimodal data to generate scene representation features.

[0137] Then, the knowledge-enhanced reasoning layer is used to generate an inference chain based on the scene representation features. The specific steps are: use the retrieval enhancement generation module to retrieve knowledge related to the scene representation features from the industry knowledge base, and then use the inference chain generator to generate an inference chain based on the scene representation features and related knowledge. After that, the inference chain is evaluated by the evaluation module. If the inference chain evaluation fails, return to the step of using the retrieval enhancement module to retrieve relevant knowledge and regenerate the inference chain until the inference chain evaluation passes and the inference chain is obtained.

[0138] After the inference chain is generated, it can be sent to an application supported by the inference chain in the user terminal, such as production safety monitoring, fault diagnosis, or early warning and disposal applications.

[0139] Furthermore, after the reasoning chain is sent to the user, the user feedback collector in the dynamic iteration layer can be used to collect user feedback, and the knowledge base can be updated based on the user feedback using the knowledge base update mechanism module.

[0140] Furthermore, when the knowledge annotation docking module receives the newly added knowledge sent by the knowledge annotation tool, the knowledge base is updated based on the newly added knowledge using the knowledge base update mechanism module.

[0141] It should be noted that the knowledge annotation tool can be pre-set and used to obtain knowledge such as newly added cases and standards annotated by industry-related experts. It can be an interface tool, and the present disclosure does not limit this.

[0142] Therefore, by using the system of the multi-modal inference chain generation method integrating the industry knowledge base proposed in the present disclosure, not only can accurate and reliable inference chains be generated to ensure the professionalism of system decisions, but also on the basis that the inference structure conforms to industry standards and specifications, the efficiency and accuracy of fault diagnosis are greatly improved. It can also realize the dynamic update of the knowledge base, enabling the system performance to continuously grow with the development of industry production practices, continuously enhancing its own intelligence level and adaptability, better meeting the increasing intelligent needs of the coal industry, and providing technical support for the sustainable development of the coal industry.

[0143] After considering the specification and the invention disclosed herein in practice, those skilled in the art will readily think of other embodiments of the present disclosure. The present disclosure aims to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and examples are only regarded as exemplary.

[0144] It should be understood that the present disclosure is not limited to the exact structures already described and shown in the drawings, and various modifications and changes can be made without departing from its scope.

[0145] It should be noted that in the description of the present disclosure, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present disclosure, unless otherwise specified, the meaning of "a plurality" is two or more.

[0146] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of the code including one or more executable instructions for implementing a specific logical function or process. The scope of the preferred embodiments of the present disclosure includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in the reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art of the embodiments of the present disclosure.

[0147] It should be understood that various parts of the present disclosure can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and the like.

[0148] Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0149] In addition, in each embodiment of the present disclosure, the functional units can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into a module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0150] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disk, or the like.

[0151] In the description of this specification, the descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms are not necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0152] Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A multi-modal inference chain generation method integrating an industry knowledge base, characterized in that, Including: Receiving multimodal data from the coal mine site, extracting features from each of the modal data, and obtaining a first feature corresponding to each of the modal data; Performing temporal alignment and semantic alignment on the first feature corresponding to each of the modal data to obtain a second aligned feature of each of the modal data; Fusing the second features of each of the modal data to generate a scene representation feature; Based on the scene representation feature, retrieving knowledge related to the scene representation feature from a preset knowledge base; Generating an inference chain based on the scene representation feature and the related knowledge.

2. The method according to claim 1, wherein The performing temporal alignment and semantic alignment on the first feature corresponding to each of the modal data to obtain a second aligned feature of each of the modal data includes: Converting the timestamps of each of the modal data to a unified time base, performing temporal alignment on each of the first features, and obtaining a third feature after temporal alignment of each of the modal data; Generating all possible ordered modal pairs based on each of the modal data, where the ordered modal pairs include different modalities, the first modality in the modal pair is the source modality, and the second modality is the target modality; Determining an attention weight corresponding to each of the modal pairs based on the third features of the modal data included in each of the modal pairs; Based on the attention weights of each of the modal pairs, respectively making the third feature of the target modality in each of the modal pairs fuse the corresponding third feature of the source modality to generate a fused semantic feature corresponding to each of the modal pairs; For each of the modal data, through a multi-head attention mechanism, integrating the fused semantic features of all the modal pairs taking it as the target modality to obtain a second feature of each of the modal data.

3. The method according to claim 2, wherein The performing temporal alignment on each of the first features includes at least one of the following: Identifying key event points in each of the modal data, and performing temporal alignment on each of the first features based on the key event points; For modal data with different sampling frequencies, performing temporal alignment on the first features of each of the modal data by interpolation or resampling.

4. The method according to claim 1, wherein The fusing the second features of each of the modal data to generate a scene representation feature includes: Obtaining the scene associated with the coal mine site; Determining the importance of each of the modal data based on the scene; Based on the importance of each of the modal data, fusing the second features of each of the modal data to generate the scene representation feature.

5. The method according to claim 1, wherein The generating an inference chain based on the scene representation feature and the related knowledge includes: Fusing the scene representation feature and the related knowledge to obtain a context feature; Determining the structure of the inference chain; Generating the inference chain based on the context feature and the structure.

6. The method according to claim 1, wherein After generating the inference chain, it further includes: Evaluating at least one of the following for the inference chain: logical integrity, specification consistency, and uncertainty quantification; In the case where any of the evaluations fails, returning to perform the operation of retrieving knowledge related to the scene representation feature from a preset knowledge base based on the scene representation feature until all evaluations pass to obtain a target inference chain.

7. The method according to claim 1, characterized in that, After generating the inference chain, it further includes: Send the inference chain to the user's terminal and obtain the user's feedback, where the feedback includes the knowledge corrected by the user; Determine the first knowledge vector corresponding to the corrected knowledge; Based on the first knowledge vector, locate the corresponding knowledge in the knowledge base and perform content replacement to complete the update of the knowledge base.

8. The method according to claim 1, characterized in that, The method further includes: When new knowledge related to the coal mine site is received through a preset interface, determine the second knowledge vector corresponding to the new knowledge; Determine the retrieval weight of the second knowledge vector; Associate and add the second knowledge vector and the retrieval weight to the knowledge base, and identify the partition to which the second knowledge vector belongs in the knowledge base; Rebuild the index of the knowledge vectors in the partition to complete the update of the knowledge base.

Images