Product full life cycle multi-mode industrial knowledge software method

Abstract

The invention discloses a product full-life-cycle multi-modal industrial knowledge software method, which comprises the following steps of: 1, acquiring a source and a demand of product full-life-cycle multi-modal industrial data, and performing clustering analysis on the multi-modal industrial data; 2, multiplexing and mapping existing ontology and standards, and constructing a hierarchical multi-modal industrial knowledge ontology system; 3, converting the multi-mode industrial data into a text mode; multi-modal industrial data knowledge mining is realized by using a text knowledge mining method, and an industrial knowledge triple cluster of each modal is formed; step 4, global coding is carried out on each modal industrial knowledge triple cluster, modal fusion and cross-modal alignment are carried out, and cross-modal industrial knowledge fusion is realized; and 5, supporting personalized configuration research and development of multi-scene intelligent business ubiquitous deployment industrial software by taking the product full-life-cycle multi-modal industrial knowledge graph as a knowledge reuse base, and realizing a software standard process of industrial knowledge from acquisition, mining to reuse.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal data processing, and specifically is a method for softwareizing multimodal industrial knowledge throughout the product life cycle. Background Art

[0002] At present, information technology, intelligent technology and industrial technology are accelerating their integration. As a carrier for realizing the continuous accumulation, system transformation and collective intelligence application of industrial knowledge, the research and development of industrial software has an urgent need for the acquisition and mining of industrial knowledge. However, the industrial knowledge implicit in the massive, multimodal and dynamic data of the product's entire life cycle is difficult to express, accumulate, integrate and reuse, making it difficult to turn industrial knowledge into software and industrial intelligent business lacks software support. As a result, most knowledge-based work on industrial sites still relies on people, with low decision-making efficiency and poor stability.

[0003] From the perspective of the theoretical system, at present, the form, characteristics, utility, and software-based path of product life cycle knowledge are unclear. It is necessary to provide basic theoretical support for the software-based expression of multi-modal, dynamic, and multi-related product life cycle knowledge through research on multi-source and multi-modal industrial knowledge clustering models for the product life cycle, product life cycle industrial knowledge engineering theory, and basic theories of industrial knowledge software expression.

[0004] From the perspective of knowledge mining, "multimodal" industrial knowledge with cross-domains and multiple carriers has problems such as difficulty in representation and difficulty in mining and integration. Therefore, it is necessary to study industrial knowledge ontology construction technology, multimodal mining technology of industrial knowledge throughout the product life cycle, cross-modal industrial knowledge fusion and graph construction technology, so as to provide algorithm models and technical support for the core links of building a multimodal industrial knowledge system. Summary of the invention

[0005] In view of this, in order to solve the shortcomings of the existing technology in the softwareization of multimodal industrial knowledge data throughout the product life cycle, the purpose of the present invention is to provide a method for softwareization of multimodal industrial knowledge throughout the product life cycle.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A method for softwareizing multimodal industrial knowledge in the entire product life cycle includes the following steps:

[0008] Step 1: Obtain the source and demand of multimodal industrial data throughout the product life cycle, and perform cluster analysis on the multimodal industrial data;

[0009] Step 2: Based on the clustering results of multimodal industrial data, reuse and map the existing ontologies and standards to build a hierarchical multimodal industrial knowledge ontology system;

[0010] Step 3: Under the guidance of the multimodal industrial knowledge ontology system, use text as a modal bridge to convert multimodal industrial data into text modality; use text knowledge mining methods to realize multimodal industrial data knowledge mining and form triple clusters of industrial knowledge in each modality;

[0011] Step 4: Globally encode the triplet clusters of industrial knowledge in each modality, and perform modal fusion and cross-modal alignment to achieve cross-modal industrial knowledge fusion and form a multi-modal industrial knowledge graph for the entire product life cycle;

[0012] Step 5: Use the multimodal industrial knowledge graph of the entire product life cycle as the foundation for knowledge reuse, support the personalized configuration and development of industrial software for ubiquitous deployment of multi-scenario intelligent business, and realize the software-based standard process of industrial knowledge acquisition, mining, and reuse.

[0013] Furthermore, in step 1, the method steps for clustering analysis of multimodal industrial data are as follows:

[0014] 11) Analyze the sources of multimodal industrial data, knowledge carriers and application requirements, perform data cleaning, relationship definition, combination induction and integration processing on multimodal industrial knowledge sources, build a full-cycle multimodal industrial database, and realize abnormal data cleaning and multi-source heterogeneous data integration of multimodal industrial knowledge sources;

[0015] 12) According to the multimodal industrial data types and the adapted similarity metric standards, combined with the full-cycle multimodal industrial knowledge software requirements, a bottom-up agglomerative multimodal clustering algorithm is selected;

[0016] 13) Perform data initialization on multimodal industrial data, treat each data point as an independent cluster, and globally measure the distance or similarity matrix between all clusters;

[0017] 14) Determine whether the distance or similarity between adjacent clusters meets the inter-cluster merging condition: if yes, execute step 15); if not, the inter-cluster merging is terminated, the clustering hierarchy is constructed, and execute step 16);

[0018] 15) Select two clusters with the smallest distance or the smallest similarity to merge, update the distance or similarity matrix between clusters, and repeat step 14);

[0019] 16) The hierarchical clustering results of full-cycle multimodal industrial data are presented using a dendrogram to enhance the understanding of the gradual aggregation process of the data and the hierarchical structure of the clusters.

[0020] Furthermore, in step 14), distance is used to judge whether adjacent clusters meet the inter-cluster merging condition, and the distance includes the nearest neighbor distance, the farthest neighbor distance and the average distance.

[0021] Furthermore, in step 2, the method steps for constructing a hierarchical multimodal industrial knowledge ontology system are:

[0022] 21) The ontology domain is clearly defined as the industrial domain, and the ontology application purpose is clearly defined as providing top-level guidance for the mining and softwareization of industrial knowledge;

[0023] 22) Check whether there are reusable concepts: if there are reusable ontologies, modify the reusable ontologies; for reusable standards, map the required content into standard ontologies according to the requirements of industrial knowledge softwareization;

[0024] 23) Evaluate the reused ontology and standard ontology, introduce the reused ontology and standard ontology that meet the requirements into the reused ontology library, and re-revise or re-map the ontologies that do not meet the requirements;

[0025] 24) Match the concepts in the reused ontology library with the full-cycle multimodal industrial data clustering results, and determine the important terms of the industrial knowledge ontology system for the clustering results and related concepts without reused ontology;

[0026] 25) Based on the hierarchical clustering results, define the hierarchical structure of the ontology class and evaluate the ontology hierarchy;

[0027] 26) Determine the ontology attributes and their properties, and define and facet the attributes in detail;

[0028] 27) Build a bridging ontology to integrate ontologies of different levels, modes, and types into a multimodal industrial knowledge ontology system for the entire product life cycle;

[0029] 28) Create instances based on the defined ontology and attributes.

[0030] Furthermore, in step 27), the multimodal industrial knowledge ontology system of the entire product life cycle is a hierarchical ontology system, which is divided into three layers, among which: the top layer includes the broadest and most abstract industrial concepts, involving the core concepts and universal principles of the entire industrial field; the second layer is close to the actual industrial scene, but still maintains a certain degree of abstraction; the bottom layer is aimed at specific industrial application scenarios.

[0031] Furthermore, in step 3, the method steps of using the text knowledge mining method to mine multimodal industrial data knowledge are as follows:

[0032] 31) Explore the natural language description method of multimodal industrial data and establish an industrial knowledge mining framework for modal feature-natural language description consistency generation;

[0033] 32) The industrial knowledge mining framework generated by modal feature-natural language description consistency converts non-text modal data into text or graph form, and realizes multimodal knowledge mining through text extraction methods, including:

[0034] For text modality: First, the PL-Marker strategy is used to encode text sentences or paragraphs, and then Albert is used for embedding representation. Finally, a hypergraph is used to model the embedded representation, and a hypergraph neural network is constructed to extract text modality knowledge and form a text modality triple group.

[0035] For image modality: select several multi-image description models, perform image description tasks on the same image, obtain multiple image description texts, embed and represent the description texts and images, perform feature recognition, and sort the description text quality. Input the top-k description texts into a combined prompt and input it into the large language model to generate a fused image description. Use the text modality knowledge extraction method to extract image modality industrial knowledge and form an image modality triple group.

[0036] For the graphic modality: first, all 3D models are converted into a standard format, and then the PCL library is introduced to process the 3D models into point clouds to obtain the 3D point clouds of the models; then the 3D object detector is used to obtain the relevant features of the point cloud, and the point cloud description text is generated based on multi-head cross attention; finally, the text modality knowledge extraction method is used to realize the graphic modality industrial knowledge extraction and form a graphic modality triple group;

[0037] For signal modality: first, the signal data is stored in a standard database, then the signal is processed and the typical time-frequency domain features of the signal are extracted; under the guidance of prior knowledge such as sensors, signal-text association rules are constructed; finally, the text modality knowledge extraction method is used to extract industrial knowledge of signal modality and form a signal modality triple group;

[0038] Regarding expression modality: according to the different forms of expressions, it is divided into unstructured expression data and semi-structured expression data; for semi-structured expression data, the discretized information is directly extracted and a graph structure is formed; for unstructured expression data, the expression in the image is first recognized through OCR, and then it is discretized to realize the construction of the scene graph of discrete elements, and finally form an expression modality triple group.

[0039] Furthermore, in step 4, the method for forming a multimodal industrial knowledge graph of the entire product life cycle is:

[0040] According to the correlation between modal triple groups, the modal original data features are introduced, and the modal triple group graph features of each triple cluster are combined for global encoding. The weights of each part of the features are assigned through dynamic cross-modal weighting, and then modal fusion and cross-modal alignment operations are performed to achieve cross-modal knowledge fusion and form a multi-modal industrial knowledge graph for the entire product life cycle.

[0041] The multimodal industrial knowledge graph of the product's entire life cycle uses the URI number as the core node and is loaded with text attribute descriptions, modal attribute descriptions, standardized data attributes and feature attributes.

[0042] Furthermore, in step 5, the method steps for softwareizing multimodal industrial knowledge of the entire product life cycle are:

[0043] Embed the multimodal industrial knowledge graph of the product's entire life cycle into existing industrial software, or implement personalized configuration and development of scenario-based intelligent business industrial software based on the multimodal industrial knowledge graph of the product's entire life cycle;

[0044] Taking the multimodal industrial knowledge graph of the entire product life cycle as the industrial knowledge foundation, we provide intelligent industrial knowledge services for the actual industrial site, break down the barriers between the steps of industrial data collection, industrial knowledge mining and fusion, and industrial software knowledge reuse in the softwareization of industrial knowledge, and realize the full process of industrial knowledge software.

[0045] The beneficial effects of the present invention are:

[0046] The method for converting product life cycle multimodal industrial knowledge into software has the following advantages:

[0047] (1) Starting from the foundation of industrial data collection and clustering, the present invention constructs an industrial knowledge ontology system, establishes a multimodal mining and cross-modal fusion framework for industrial knowledge, and uses the industrial knowledge graph as a carrier to support industrial software, realize industrial knowledge reuse, and finally connects the entire process of industrial knowledge softwareization.

[0048] (2) The multimodal industrial knowledge mining framework proposed in the present invention is a standardized mining method. In the present invention, the method processes five types of modal data: text, image, graphic, signal and expression. However, based on the standardized mining ideas contained in the framework, the framework can be extended to other more extensive modal types. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0050] Figure 1 It is a flow chart of the method for converting multimodal industrial knowledge into software for the entire product life cycle of the present invention;

[0051] Figure 2 Flowchart for constructing hierarchical multimodal industrial knowledge ontology system;

[0052] Figure 3 It is a structural diagram of the hierarchical ontology system;

[0053] Figure 4 Schematic diagram of the industrial knowledge mining framework for modal feature-natural language description generation;

[0054] Figure 5 A schematic diagram for the cross-modal industrial knowledge fusion and industrial knowledge graph formation throughout the product life cycle. DETAILED DESCRIPTION

[0055] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0056] like Figure 1 As shown, the method for softwareizing multimodal industrial knowledge throughout the product life cycle of this embodiment includes the following steps.

[0057] Step 1: Obtain the sources and needs of multimodal industrial data throughout the product life cycle, and perform cluster analysis on multimodal industrial data.

[0058] Specifically, in this embodiment, the method steps for clustering analysis of multimodal industrial data are as follows:

[0059] 11) Analyze the sources of multimodal industrial data, knowledge carriers and application requirements, perform data cleaning, relationship definition, combination induction and integration processing on multimodal industrial knowledge sources, build a full-cycle multimodal industrial database, and realize abnormal data cleaning and multi-source heterogeneous data integration of multimodal industrial knowledge sources.

[0060] 12) According to the multimodal industrial data types and the adapted similarity metrics, combined with the full-cycle multimodal industrial knowledge software requirements, a bottom-up agglomerative multimodal clustering algorithm is selected.

[0061] 13) Perform data initialization on multimodal industrial data, treat each data point as an independent cluster, and globally measure the distance or similarity matrix between all clusters.

[0062] 14) Determine whether the distance or similarity between adjacent clusters meets the inter-cluster merging condition: if so, execute step 15); if not, the inter-cluster merging is terminated, the clustering hierarchy is constructed, and execute step 16).

[0063] In this embodiment, the inter-cluster merging condition is less than or equal to a preset distance threshold or similarity threshold. In this embodiment, the distance is used to judge whether the adjacent clusters meet the inter-cluster merging condition, and the distance includes the nearest neighbor distance (single chain method), the farthest neighbor distance (full chain method) and the average distance (average chain method).

[0064] 15) Select two clusters with the smallest distance or the smallest similarity to merge, and update the distance or similarity matrix between clusters to reflect the relationship between the newly formed cluster and other clusters; and execute step 14) repeatedly.

[0065] 16) The hierarchical clustering results of full-cycle multimodal industrial data are presented using a dendrogram to enhance the understanding of the gradual aggregation process of the data and the hierarchical structure of the clusters.

[0066] Step 2: Based on the clustering results of multimodal industrial data, reuse and map the existing ontologies and standards to build a hierarchical multimodal industrial knowledge ontology system.

[0067] In this embodiment, the product life cycle multimodal industrial knowledge ontology system is based on the hierarchical clustering results. The specific ontology system construction process is as follows: Figure 2 As shown. The ontology system construction method of this embodiment is an improved seven-step method, which is an improvement based on the Stanford seven-step method. Specifically, in this embodiment, the method steps for constructing a hierarchical multimodal industrial knowledge ontology system are:

[0068] 21) Clarify the ontology domain as the industrial domain, and clarify the application purpose of ontology to provide top-level guidance for the mining and softwareization of industrial knowledge.

[0069] 22) Check whether there are any reusable concepts, mainly involving existing ontologies and standards: if there are reusable ontologies, modify the reusable ontology; for reusable standards, map the required content to the standard ontology based on the needs of industrial knowledge software.

[0070] 23) Domain experts evaluate the reusable ontologies and standard ontologies based on the evaluation criteria, and introduce the reusable ontologies and standard ontologies that meet the requirements into the reusable ontology library. For ontologies that do not meet the requirements, they are revised or mapped again until they meet the requirements or are eliminated.

[0071] 24) Match the concepts in the reused ontology library with the full-cycle multimodal industrial data clustering results. For the clustering results and related concepts without reused ontology, determine the important terms of the industrial knowledge ontology system with the guidance of domain experts.

[0072] 25) Based on the hierarchical clustering results, define the hierarchy of ontology classes and evaluate the ontology hierarchy.

[0073] 26) Determine the ontology attributes and their properties, and define and facet the attributes in detail.

[0074] 27) Build a bridging ontology to integrate ontologies of different levels, modes, and types into a multimodal industrial knowledge ontology system for the entire product life cycle.

[0075] like Figure 3 As shown, in this embodiment, the multimodal industrial knowledge ontology system of the product's entire life cycle is a hierarchical ontology system, which is divided into three layers, among which: the top layer includes the broadest and most abstract industrial concepts, such as "productivity", "automation", "sustainable development", etc., involving the core concepts and universal principles of the entire industrial field; the second layer is close to the actual industrial scene, but still maintains a certain degree of abstraction, including specific production processes, industrial machine types, production efficiency and other concepts; the bottom layer is aimed at specific industrial application scenarios, including the operating details of specific machines, specific maintenance guidelines, troubleshooting procedures, etc.

[0076] 28) Create instances based on the defined ontology and attributes.

[0077] Step 3: Under the guidance of the multimodal industrial knowledge ontology system, use text as a modal bridge to convert multimodal industrial data into text modality; use text knowledge mining methods to realize multimodal industrial data knowledge mining and form triple clusters of industrial knowledge in each modality.

[0078] like Figure 4 As shown, in this embodiment, the method steps of using the text knowledge mining method to perform multimodal industrial data knowledge mining are as follows:

[0079] 31) In view of the problem that industrial data of the entire product life cycle is presented in many ways, types, and carriers, and knowledge mining methods of different modal data are difficult to standardize, explore the natural language description method of multimodal industrial data, and establish an industrial knowledge mining framework for modal feature-natural language description consistency generation. Specifically, the industrial knowledge mining framework for modal feature-natural language description consistency generation constructed in this embodiment is mainly aimed at the five common modalities in industrial sites, namely text, image, graphic, signal and formula, but the idea of ​​the mining framework can be extended to other modal data.

[0080] 32) The industrial knowledge mining framework generated by modal feature-natural language description consistency converts non-text modal data into text or graph form, and realizes multimodal knowledge mining through text extraction methods, including:

[0081] For text modality: First, the PL-Marker strategy is used to encode text sentences or paragraphs, and then Albert is used for embedding representation. Finally, a hypergraph is used to model the embedded representation, and a hypergraph neural network is constructed to extract text modality knowledge and form a text modality triple group.

[0082] Note: There are various methods for text modality extraction. This embodiment is only an example. In this framework, the text modality knowledge extraction method is the most basic method and is applied to other modalities.

[0083] For image modality: select several multi-image description models, perform image description tasks on the same image, obtain multiple image description texts, embed and represent the description texts and images, perform feature recognition, and sort the description text quality. The top-k description texts are combined and prompted into the large language model to generate a fused image description. Use text modality knowledge extraction methods to extract image modality industrial knowledge and form image modality triple groups.

[0084] For graphic modality: first convert all 3D models into standard formats such as OBJ or STEP, then introduce the PCL library to perform point cloud processing on the 3D model to obtain the 3D point cloud of the model; then use 3D object detectors such as Pointnet++ to obtain point cloud related features, and generate point cloud description text based on multi-head cross attention; use text modality knowledge extraction method to realize graphic modality industrial knowledge extraction and form graphic modality triple group.

[0085] For signal modality: first, store the signal data in standard databases such as csv, then call time-frequency domain statistical methods and signal processing methods such as Fourier transform and wavelet transform to process the signal, and extract the typical time-frequency domain features of the signal; under the guidance of prior knowledge such as sensors, construct signal-text association rules; use text modality knowledge extraction methods to realize signal modality industrial knowledge extraction and form signal modality triple groups.

[0086] Regarding expression modality: according to the different forms of expressions, it is divided into unstructured expression data and semi-structured expression data; among them, unstructured mainly refers to data in the form of pictures, and semi-structured refers to data in the form of expressions such as latex and XML. Specifically, for semi-structured expression data, the discretized information is directly extracted and a graph structure is formed; for unstructured expression data, the expression in the picture is first identified through OCR, then discretized, and the scene graph of discrete elements is constructed with human participation, and finally an expression modality triple group is formed.

[0087] Note: All text knowledge extraction is carried out under the guidance of the ontology system, and the parts that require human participation are also operated in accordance with the system specified by the ontology.

[0088] Step 4: Globally encode the triplet clusters of industrial knowledge in each modality, and perform modal fusion and cross-modal alignment to achieve cross-modal industrial knowledge fusion and form a multimodal industrial knowledge graph for the entire product life cycle.

[0089] After the multimodal industrial data is processed by the industrial knowledge mining framework generated by multimodal features and natural language description, each modality forms its own modal triple group. In order to better realize the complementarity of each modality's industrial knowledge, cross-modal industrial knowledge fusion is required. Figure 5 As shown. In this embodiment, the method for forming a multimodal industrial knowledge graph for the entire life cycle of a product is as follows: according to the association relationship of the modal triple group, the modal original data features are introduced, and the modal triple group graph features such as the graph topology information, relationship information and attribute information of each triple cluster are combined and globally encoded, and weights are assigned to each part of the features through dynamic cross-modal weighting, and then modal fusion and cross-modal alignment operations are performed to achieve cross-modal knowledge fusion and form a multimodal industrial knowledge graph for the entire life cycle of the product. The multimodal industrial knowledge graph for the entire life cycle of a product takes the URI number as the core node, and loads text attribute descriptions (i.e., each modal triple cluster information), modal attribute descriptions (i.e., the number of modalities involved under the node and related information), standardized data attributes (i.e., the standardized data that can represent the node, and non-text data can be represented by URL mounting) and feature attributes (i.e., abstract features extracted by various algorithm models in the mining framework).

[0090] Step 5: Use the multimodal industrial knowledge graph of the entire product life cycle as the foundation for knowledge reuse, support the personalized configuration and development of industrial software for ubiquitous deployment of multi-scenario intelligent business, and realize the software-based standard process of industrial knowledge acquisition, mining, and reuse.

[0091] In this embodiment, the method of software-based multimodal industrial knowledge of the entire life cycle of a product is to embed the multimodal industrial knowledge graph of the entire life cycle of a product into existing industrial software, or to realize personalized configuration and development of scenario-based intelligent business industrial software based on the multimodal industrial knowledge graph of the entire life cycle of a product.

[0092] In this embodiment, the multimodal industrial knowledge graph of the entire life cycle of the product is used as the industrial knowledge base, and is combined with functions such as semantic parsers, intelligent searchers, and report statistics to provide powerful intelligent industrial knowledge services for the actual industrial site, thereby breaking through the barriers between the various steps of industrial knowledge softwareization, such as industrial data collection, industrial knowledge mining and fusion, and industrial software knowledge reuse, and achieving full process integration of industrial knowledge software.

[0093] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A method for converting multimodal industrial knowledge into software for the entire product life cycle, characterized by: The steps include: Step 1: Obtain the source and demand of multimodal industrial data throughout the product life cycle, and perform cluster analysis on the multimodal industrial data; Step 2: Based on the clustering results of multimodal industrial data, reuse and map the existing ontologies and standards to build a hierarchical multimodal industrial knowledge ontology system; Step 3: Under the guidance of the multimodal industrial knowledge ontology system, use text as a modal bridge to convert multimodal industrial data into text modality; use text knowledge mining methods to realize multimodal industrial data knowledge mining and form triple clusters of industrial knowledge in each modality; Step 4: Globally encode the triplet clusters of industrial knowledge in each modality, and perform modal fusion and cross-modal alignment to achieve cross-modal industrial knowledge fusion and form a multi-modal industrial knowledge graph for the entire product life cycle; Step 5: Use the multimodal industrial knowledge graph of the entire product life cycle as the foundation for knowledge reuse, support the personalized configuration and development of industrial software for ubiquitous deployment of multi-scenario intelligent business, and realize the software-based standard process of industrial knowledge acquisition, mining, and reuse.

2. The method for converting product life cycle multimodal industrial knowledge into software according to claim 1, characterized in that: In step 1, the method steps for clustering analysis of multimodal industrial data are as follows: 11) Analyze the sources of multimodal industrial data, knowledge carriers and application requirements, perform data cleaning, relationship definition, combination induction and integration processing on multimodal industrial knowledge sources, build a full-cycle multimodal industrial database, and realize abnormal data cleaning and multi-source heterogeneous data integration of multimodal industrial knowledge sources; 12) According to the multimodal industrial data types and the adapted similarity metric standards, combined with the full-cycle multimodal industrial knowledge software requirements, a bottom-up agglomerative multimodal clustering algorithm is selected; 13) Perform data initialization on multimodal industrial data, treat each data point as an independent cluster, and globally measure the distance or similarity matrix between all clusters; 14) Determine whether the distance or similarity between adjacent clusters meets the inter-cluster merging condition: if yes, execute step 15); if not, the inter-cluster merging is terminated, the clustering hierarchy is constructed, and execute step 16); 15) Select two clusters with the smallest distance or the smallest similarity to merge, update the distance or similarity matrix between clusters, and repeat step 14); 16) The hierarchical clustering results of full-cycle multimodal industrial data are presented using a dendrogram to enhance the understanding of the gradual aggregation process of the data and the hierarchical structure of the clusters.

3. The method for converting product life cycle multimodal industrial knowledge into software according to claim 2, characterized in that: In step 14), distance is used to judge whether adjacent clusters meet the inter-cluster merging condition, and the distance includes the nearest neighbor distance, the farthest neighbor distance and the average distance.

4. The method for converting product life cycle multimodal industrial knowledge into software according to claim 1, characterized in that: In the second step, the method steps for constructing a hierarchical multimodal industrial knowledge ontology system are as follows: 21) The ontology domain is clearly defined as the industrial domain, and the ontology application purpose is clearly defined as providing top-level guidance for the mining and softwareization of industrial knowledge; 22) Check whether there are reusable concepts: if there are reusable ontologies, modify the reusable ontologies; for reusable standards, map the required content into standard ontologies according to the requirements of industrial knowledge softwareization; 23) Evaluate the reused ontology and standard ontology, introduce the reused ontology and standard ontology that meet the requirements into the reused ontology library, and re-revise or re-map the ontologies that do not meet the requirements; 24) Match the concepts in the reused ontology library with the full-cycle multimodal industrial data clustering results, and determine the important terms of the industrial knowledge ontology system for the clustering results and related concepts without reused ontology; 25) Based on the hierarchical clustering results, define the hierarchical structure of the ontology class and evaluate the ontology hierarchy; 26) Determine the ontology attributes and their properties, and define and facet the attributes in detail; 27) Build a bridging ontology to integrate ontologies of different levels, modes, and types into a multimodal industrial knowledge ontology system for the entire product life cycle; 28) Create instances based on the defined ontology and attributes.

5. The method for converting product life cycle multimodal industrial knowledge into software according to claim 4, characterized in that: In the step 27), the multimodal industrial knowledge ontology system of the product's entire life cycle is a hierarchical ontology system, which is divided into three layers, among which: the top layer includes the broadest and most abstract industrial concepts, involving the core concepts and universal principles of the entire industrial field; the second layer is close to the actual industrial scene, but still maintains a certain degree of abstraction; the bottom layer is aimed at specific industrial application scenarios.

6. The method for converting product life cycle multimodal industrial knowledge into software according to claim 1, characterized in that: In step 3, the method steps for multimodal industrial data knowledge mining using text knowledge mining method are as follows: 31) Explore the natural language description method of multimodal industrial data and establish an industrial knowledge mining framework for modal feature-natural language description consistency generation; 32) The industrial knowledge mining framework generated by modal feature-natural language description consistency converts non-text modal data into text or graph form, and realizes multimodal knowledge mining through text extraction methods, including: For text modality: First, the PL-Marker strategy is used to encode text sentences or paragraphs, and then Albert is used for embedding representation. Finally, a hypergraph is used to model the embedded representation, and a hypergraph neural network is constructed to extract text modality knowledge and form a text modality triple group. For image modality: select several multi-image description models, perform image description tasks on the same image, obtain multiple image description texts, embed and represent the description texts and images, perform feature recognition, and sort the description text quality. Input the top-k description texts into a combined prompt and input it into the large language model to generate a fused image description. Use the text modality knowledge extraction method to extract image modality industrial knowledge and form an image modality triple group. For the graphic modality: first, all 3D models are converted into a standard format, and then the PCL library is introduced to process the 3D models into point clouds to obtain the 3D point clouds of the models; then the 3D object detector is used to obtain the relevant features of the point cloud, and the point cloud description text is generated based on multi-head cross attention; finally, the text modality knowledge extraction method is used to realize the graphic modality industrial knowledge extraction and form a graphic modality triple group; For signal modality: first, the signal data is stored in a standard database, then the signal is processed and the typical time-frequency domain features of the signal are extracted; under the guidance of prior knowledge such as sensors, signal-text association rules are constructed; finally, the text modality knowledge extraction method is used to extract industrial knowledge of signal modality and form a signal modality triple group; Regarding expression modality: according to the different forms of expressions, it is divided into unstructured expression data and semi-structured expression data; for semi-structured expression data, the discretized information is directly extracted and a graph structure is formed; for unstructured expression data, the expression in the image is first recognized through OCR, and then it is discretized to realize the construction of the scene graph of discrete elements, and finally form an expression modality triple group.

7. The method for converting product life cycle multimodal industrial knowledge into software according to claim 1, characterized in that: In step 4, the method for forming a multimodal industrial knowledge graph for the entire product life cycle is: According to the correlation between modal triple groups, the modal original data features are introduced, and the modal triple group graph features of each triple cluster are combined for global encoding. The weights of each part of the features are assigned through dynamic cross-modal weighting, and then modal fusion and cross-modal alignment operations are performed to achieve cross-modal knowledge fusion and form a multi-modal industrial knowledge graph for the entire product life cycle. The multimodal industrial knowledge graph of the product's entire life cycle uses the URI number as the core node and is loaded with text attribute descriptions, modal attribute descriptions, standardized data attributes and feature attributes.

8. The method for converting product life cycle multimodal industrial knowledge into software according to claim 1, characterized in that: In step 5, the method steps for softwareization of multimodal industrial knowledge throughout the product life cycle are as follows: Embed the multimodal industrial knowledge graph of the product's entire life cycle into existing industrial software, or implement personalized configuration and development of scenario-based intelligent business industrial software based on the multimodal industrial knowledge graph of the product's entire life cycle; Taking the multimodal industrial knowledge graph of the entire product life cycle as the industrial knowledge foundation, we provide intelligent industrial knowledge services for the actual industrial site, break down the barriers between the steps of industrial data collection, industrial knowledge mining and fusion, and industrial software knowledge reuse in the softwareization of industrial knowledge, and realize the full process of industrial knowledge software.

Images