A method for recommending labels and label instances

Abstract

The present invention relates to a method for recommending labels and label instances, and belongs to the field of computer data processing technology. In response to the problems of label management and intelligent label management of massive test data, the present invention proposes a method for recommending labels and label instances. The method mainly includes: collecting structured, unstructured and semi-structured data, constructing ontology concepts according to the types of test data, and forming a label library; then, using different methods to extract equipment entities and entity relationships from multimodal data such as images, texts, audio, video, and paper, and constructing a label instance library; using methods such as rule mapping and natural language processing to map the relationship between labels and label instances; finally, mining user personal information and label usage information, combining personal information and label information, and forming intelligent recommendations based on labels and label instances.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and specifically discloses a tag and tag instance recommendation method, which aims to construct a tag and tag instance library for experimental data and realize user-related tag recommendations. Background Art

[0002] As the requirements for equipment big data development continue to increase and progress continues, test data has gradually become a crucial component of equipment data. Test and appraisal data development focuses on the needs of data organization and management, acquisition and compilation, analysis and mining, shared applications, operations and maintenance management, and infrastructure support, providing a unified data collection and compilation tool for historical test data. Different test and appraisal institutions generate enormous amounts of test data annually. Existing data management tools only manage collected and compiled data and are unable to intuitively display the entities and relationships within the data. Furthermore, traditional data management tools are unable to quickly, accurately, and intelligently recommend relevant information within the data. Summary of the Invention

[0003] To address the challenges of managing and intelligently managing the tags of massive amounts of experimental data, this paper proposes a method for recommending tags and tag instances. This method primarily involves collecting structured, unstructured, and semi-structured data from experiments, constructing ontology concepts based on the types of experimental data, and forming a tag library. Next, it uses various methods to extract equipment entities and entity relationships from multimodal data, such as images, text, audio, video, and paper, to construct a tag instance library. It then uses rule mapping and natural language processing to map the relationships between tags and tag instances. Finally, it mines user personal information and tag usage information, combining this information to form intelligent recommendations based on tags and tag instances.

[0004] The present invention specifically adopts the following technical solution: a method for recommending tags and tag instances, comprising the following steps:

[0005] Step SS1: Label ontology concept construction, including: collecting structured data, unstructured data and semi-structured data of the experiment, constructing ontology concepts according to the experimental data type, and forming a label library;

[0006] Step SS2: label entity feature extraction, including: using different methods to extract equipment entities and entity relationships from multimodal data, and building a label instance library, wherein the multimodal data includes image data, text data, audio data, video data, and paper data;

[0007] Step SS3: mapping labels and label instances, including: mapping the relationship between labels and label instances using rule mapping and natural language processing methods;

[0008] Step SS4: Generate tag and tag instance recommendations based on user information, including: mining user personal information and using tag information, combining personal information and tag information to form intelligent recommendations based on tags and tag instances.

[0009] As a preferred embodiment, step SS1 specifically includes the following steps:

[0010] Step S11: collecting test data, including: mainly using paper data, electronic files, and structured data, designing different data collection methods according to different data sources, extracting structured data, unstructured data, and semi-structured data; extracting data elements from various data, the data elements including equipment identification, equipment name, test data collection time, and test task related information;

[0011] Step S12: constructing an ontology tag library, including: when constructing the ontology library, using a tree diagram to classify the types and attributes of the collected test data, constructing the ontology concepts of test equipment, test equipment, accompanying test equipment, test tasks and test personnel, and creating a tag library.

[0012] As a preferred embodiment, step SS2 specifically includes: for the image data, using an image target recognition algorithm based on an improved convolutional neural network model, first marking the equipment target in the original image and converting it into a 224*224 standard image, inputting it into the convolutional neural network model, performing image feature extraction, adding an SE module at the exit of each convolution layer and Inception layer to update the weight of the image feature parameters, then connecting the pooling layer to perform dimensionality reduction processing on the features output by the convolution layer and the Inception layer, inputting the result of the pooling operation output into the fully connected layer to convert it into a one-dimensional vector, and finally inputting it into the softMax classifier to output the recognition result, and using the image recognition model of the improved convolutional neural network model to identify the equipment type and the position of the equipment in the picture, and perform entity marking on the picture.

[0013] As a preferred embodiment, the step SS2 specifically includes: using a deep learning method to identify entities in the text data, first, organizing a dictionary containing nouns, wherein the nouns include names of countries, important figures, and equipment names, to form a domain dictionary; then, selecting some texts containing ontology concepts for annotation, annotating the nouns of countries, equipment, and important figures, using a Chinese dictionary and a deep learning network LSTM-CRF model to train a model on the annotated data, and using the trained model to perform entity recognition on large-scale text information; using a keyword extraction model and a relationship extraction model with set rules to obtain the relationship features between the entities extracted from the text, thereby obtaining the entity and relationship information in the text and constructing a label instance library.

[0014] As a preferred embodiment, step SS2 specifically includes: for the audio data, obtaining the time length, bit rate and sampling rate of the audio from the acquired test audio data, using a convolutional neural network to capture the characteristics of the audio loudness changing over time in different frequency domains, and using the SVM support vector machine classification method to classify the audio data, thereby identifying the labeled entity information.

[0015] As a preferred embodiment, the step SS2 specifically includes: for the video data, obtaining the time length, encoding format, encoding standard and bit rate characteristics of the acquired video data; at the same time, using a clustering-based method to extract key frames from the video data, extracting entities in the video; for key frame extraction in general videos, using a clustering-based method, grouping video frames using the similarity between video frames, so that similar video frames form a cluster, thereby dividing the video data into clusters one by one, the video frames within the cluster are similar in content, while the video frames between clusters are very different in content; finally, the video frame closest to the cluster center in each cluster is added to the result set;

[0016] For the key frame extraction of motion videos, a reinforcement learning key frame extraction algorithm that integrates multi-channel features and attention mechanism is used, so that the extraction process does not require shot segmentation and manual feature extraction. The reinforcement learning key frame extraction algorithm starts with the fusion of features. First, a deep neural network is used to extract motion features and static features in the video frame and fuse them. Then, a self-attention model is used to extract the global features of the video sequence. Then, the importance of each video frame is calculated based on the global features to obtain the key frames in the video.

[0017] As a preferred embodiment, the step SS2 specifically includes: for the data source being the paper data, using the optical character recognition (OCR) algorithm, analyzing and identifying the image file of the text material, obtaining the text and layout information, identifying the text in the image, and returning it in the form of text, using a deep learning model based on CRNN, utilizing the potential of the convolutional neural network (CNN) for image feature engineering and combining it with the potential of the long short-term memory network (LSTM) for serialization recognition, extracting robust features and avoiding the extremely difficult single-character segmentation and single-character recognition in traditional algorithms through serialization recognition. At the same time, serialization recognition also embeds temporal dependency, and can convert paper text into text data through OCR technology, and then perform entity recognition, keyword extraction, and entity relationship extraction on the recognized text through text recognition technology to obtain entity and entity relationship information in the paper data.

[0018] As a preferred embodiment, step SS3 specifically includes: constructing ontology concepts and label concepts, obtaining entity information through multiple pairs of extracted image, video, text, audio, and paper data features, thereby constructing label instances, and based on the ontology concept model, adopting a multi-strategy integrated approach, including a mapping method based on rule mapping and a mapping method based on semantics, mapping labels and label instances from various forms of data, obtaining triples of <label 1-relationship-label 2> and <label 1-relationship-label instance>, and filling them into the current ontology mapping;

[0019] The mapping method based on rule mapping includes: using an association rule mining method to screen label instances with a high degree of relationship between label ontology concepts and label instances, filling them into a rule model, using a supervised learning support vector machine model to predict the mapping relationship between the mined labels and label instances, and finally, through manual verification of the prediction results, enriching the label instances corresponding to each correctly predicted label into the attribute set;

[0020] The semantic-based mapping method includes: for attributes that cannot be extracted using association rules, a method of extracting syntactic relations between sentences and semantic relations between words is used, and a RoBERTa and entity boundary prediction model is used. The prediction model includes a text encoding layer, an entity boundary prediction layer, and a BiLSTM_CRF attribute prediction layer. First, the input text is encoded by RoBERTa to obtain its hidden layer state vector, and then the hidden layer state vector is input into the entity boundary prediction layer and the BiLSTM_CRF attribute prediction layer respectively. In the entity boundary prediction layer, the entity header and the entity boundary are respectively encoded by 0 / 1 encoding. The labeling of the tail of the entity, and then calculate the loss values ​​start_loss and end_loss of the two sequence labels. In the BiLSTM_CRF attribute prediction layer, the output result of the entity boundary prediction layer is used as a feature and concatenated with the text vector, input into BiLSTM_CRF to predict the text attribute label, and then calculate its loss value att_loss. Finally, when optimizing the model, the loss value start_loss, loss value end_loss and loss value att_loss are comprehensively considered, and a weighted sum is performed. The overall optimization of the model is achieved through back propagation, and a list of candidate attributes is obtained to finally form the attributes.

[0021] As a preferred embodiment, step SS4 includes: extracting feature representation of user information from the extracted user information, including the user's organization, level, tasks, responsible resources, user usage habits, and most recently used tag information, fusing the feature representation with the extracted tag entity feature representation, obtaining a fusion of user features and tag entity features through a fusion attention mechanism, training a network model, recommending tags related to the user, recommending relevant information of the tag instance through tag library query technology, and visually displaying it.

[0022] As a preferred embodiment, step SS4 specifically includes:

[0023] Step SS41: extracting user features, including: selecting user-related information from the database, including the user's organization, level, assigned tasks, responsible resources, user usage habits, and most recent usage tag information; first, obtaining a vector value of the user information through pre-trained word vectors; then, using a convolutional neural network (CNN), extracting user-related information features to form a user feature matrix;

[0024] Step SS42: Extract tag library tag features, including: obtaining tag entities from the tag library, recorded as Represents the jth entity in the set of labeled entities corresponding to the kth relationship. Entities corresponding to different relationships are mapped to low-dimensional vector spaces to obtain the label-entity embedding matrix, which is then fed into the KGCNN model to obtain the label-entity feature matrix S. r ;

[0025] In the convolution layer, the label-entity features are extracted by different window sizes, and each possible position in the embedding matrix is ​​filtered. Finally, the maximum pooling method is used for feature selection to obtain the movie entity matrix.

[0026]

[0027] Among them, E r Represents the embedding matrix of label-entity, G represents the filter kernel for feature extraction, ⊙ represents the dot product operation, relu represents the nonlinear activation function, b represents the bias parameter of relu, and max represents the maximum pooling operation;

[0028] Step SS43: Fusion of user features and tag entity features, including: transforming the tag-entity feature matrix S r Together with the user-movie tag embedding matrix T, it is used as the input of the hybrid attention model to calculate the hybrid attention weight α of different relationship tags-entities T ;

[0029]

[0030] Among them, α T Represents the attention weight, the matmul function is a matrix multiplication function, T u and After multiplication, the attention weight is obtained by normalization through the softmax function;

[0031] The output of the KGCNN model and the rest of the label feature matrix are fed into the multi-layer perceptron MLP. After training through multiple fully connected neural network layers in the MLP, they are multiplied by the hybrid attention weights to obtain the label entity feature matrix I.

[0032] Step SS44 recommends tags and tag instances, including: fusing user features and tag entity features, passing through a fully connected layer, and finally obtaining tags related to the user; obtaining tags and tag instances related to the user through query technology in the established tag and tag instance knowledge graph, visually displaying them, and forming recommendations.

[0033] The beneficial effects of the present invention are as follows: the present invention collects and organizes experimental data to form structured data, and uses a variety of methods to effectively extract entity and entity relationship information from multimodal data for the multimodality of data. A rule-based mapping method and a semantic-based mapping method are used to map the label ontology and the label instance, and finally recommend labels based on user characteristics and behavioral characteristics, thereby solving the technical requirements of existing data management tools that only manage the collected and organized data and cannot intuitively display the entities and relationships in the data, and traditional data management tools cannot quickly, accurately and intelligently recommend relevant information of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is the overall design diagram of the present invention;

[0035] Figure 2 This is a schematic diagram of the structure of the equipment body of the present invention;

[0036] Figure 3 This is a mapping relationship diagram between labels and label instances of the present invention;

[0037] Figure 4 This is the label recommendation model diagram of the present invention. DETAILED DESCRIPTION

[0038] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to specific embodiments and drawings.

[0039] Figure 1 This is the overall design diagram of the method presented in this invention. It consists of four parts: tag ontology construction, tag entity feature extraction, tag-to-tag instance mapping, and user intelligent recommendation. This method, by constructing a library of tags and tag instances for multi-structured data collected from experimental data, can intelligently and accurately recommend tags based on user preferences and tag features from massive amounts of experimental data.

[0040] The present invention proposes a tag and tag instance recommendation method, which mainly includes the following four parts.

[0041] 1. Label ontology construction.

[0042] 1.1 Equipment Data Collection. Test data is collected primarily through paper, electronic, and structured data sources. Different data collection methods are designed based on the data source: extracting structured, unstructured, and semi-structured data. Furthermore, data elements are extracted from each data source, such as equipment identification, equipment name, test data collection time, test mission, and other related information.

[0043] 1.2 Construct an ontology tag library. When constructing the ontology library, a tree diagram is mainly used to classify the types and attributes of the collected test data, construct ontology concepts such as test equipment, test equipment, accompanying test equipment, test tasks and test personnel, and create a tag library. Figure 2 Build a schematic for the device itself.

[0044] 2. Multimodal feature extraction and label entity construction: The data collected in step 1 is used to extract entity features from multimodal data, thereby constructing entity label instances. The main multimodal data includes: image data, text data, audio data, video data, and paper data.

[0045] 2.1 Image Data. For image data, an image target recognition algorithm based on an improved convolutional neural network model is used. The equipment targets in the original image are labeled and converted into a standard 224*224 image. This image is then fed into the convolutional neural network model for image feature extraction. An SE module is added at the exit of each convolutional and Inception layer to update the weights of the image feature parameters. A pooling layer is then connected to reduce the dimensionality of the features output by the convolutional and Inception layers. The output of the pooling operation is fed into a fully connected layer to be converted into a vector, which is finally fed into a softmax classifier to output the recognition result. Using the image recognition model of the improved convolutional neural network model, the equipment type and location in the image are identified, and the image entities are labeled.

[0046] 2.2 Text Data. Deep learning methods were used to identify entities within Chinese text test data. First, a dictionary containing nouns, including names of countries, important figures, and equipment, was compiled to form a domain dictionary. Then, some text containing ontological concepts was manually annotated, with nouns for countries, equipment, and important figures labeled. A Chinese dictionary and a deep learning network LSTM-CRF model were used to train the annotated data. The trained model was used to perform entity recognition on large-scale text information. A keyword extraction model and a rule-based relationship extraction model were used to obtain the relationship features between the extracted entities in the text, thereby obtaining entity and relationship information in the text and constructing a label instance library.

[0047] 2.3 Audio Data. The obtained experimental audio data is used to obtain the audio duration, bit rate, and sampling rate. A convolutional neural network is used to capture the characteristics of the audio loudness changes over time in different frequency domains. The audio data is classified using the SVM (support vector machine) classification method to identify the labeled entity information.

[0048] 2.4 Video Data. The acquired video data is used to obtain features such as the video's duration, encoding format, encoding standard, and bitrate. Clustering-based methods are then used to extract keyframes from the video to extract entities within the video. For keyframe extraction in general videos, a clustering-based method is used to group frame objects using the similarity between video frames. Similar frame objects form a cluster, thus dividing the video into clusters. Videos within a cluster are similar in content, while videos between clusters differ significantly in content. Finally, the frame closest to the cluster center within each cluster is added to the result set.

[0049] Specifically, for keyframe extraction in motion videos, a reinforcement learning keyframe extraction algorithm that fuses multi-channel features with an attention mechanism is used, eliminating the need for shot segmentation and manual feature extraction. This algorithm begins by fusing features, first using a deep neural network to extract motion and static features from video frames and fusing them. A self-attention model is then used to extract global features from the video sequence. The importance of each frame is then calculated based on these global features, ultimately resulting in the keyframes in the video.

[0050] 2.5 Paper Data. For paper data sources, optical character recognition (OCR) is used to analyze and identify image files of text materials, obtain text and layout information, identify the text in the image, and return it as text. Using a CRNN-based deep learning model, the potential of convolutional neural networks (CNN) for image feature engineering is combined with the potential of long short-term memory networks (LSTM) for serialization recognition. This not only extracts robust features, but also avoids the extremely difficult single-character segmentation and recognition in traditional algorithms through sequence recognition. Serialization recognition also embeds temporal dependencies. OCR technology can convert paper text into text data. Then, using text recognition technology, entity recognition, keyword extraction, and entity relationship extraction are performed on the recognized text to obtain entity and entity relationship information in the paper data.

[0051] 3. Label and label instance mapping. The present invention constructs ontology concepts and label concepts, obtains entity information through multiple pairs of extracted image, video, text, audio, paper and other data features, and thus constructs label instances. Based on the ontology concept model, a multi-strategy comprehensive method is adopted, including a mapping method based on rule mapping and a mapping method based on semantics, to map labels and label instances from various forms of data, obtain the triple information of <label1-relationship-label2> and <label1-relationship-label instance>, and fill it into the current ontology mapping.

[0052] Rule-based mapping: Using association rule mining methods, we screen label instances with a high degree of relationship between label ontology concepts and label instances, populate these into the rule model, and then use the supervised learning support vector machine model to predict the mapping relationship between the mined labels and label instances. Finally, through manual verification of the prediction results, we enrich the attribute set with the corresponding label instances for each correctly predicted label.

[0053] Semantic-Based Mapping: For attributes that cannot be extracted using association rules, we leverage the syntactic relationships between sentences and the semantic relationships between words. This approach, based on RoBERTa and entity boundary prediction, comprises a text encoding layer, an entity boundary prediction layer, and a BiLSTM_CRF attribute prediction layer. First, the input text is encoded using RoBERTa to obtain its hidden layer state vector. This is then fed into the entity boundary prediction layer and the BiLSTM_CRF attribute prediction layer, respectively. In the entity boundary prediction layer, we use 0 / 1 encoding to annotate the entity head and tail, respectively, and then calculate the loss values ​​start_loss and end_loss for these two sequence annotations. In the BiLSTM_CRF attribute prediction layer, we concatenate the output of the entity boundary prediction layer with the text vector as features and feed it into the BiLSTM_CRF layer to predict the text attribute labels. The loss value att_loss is then calculated. Finally, during model optimization, we comprehensively consider these three loss values, perform a weighted sum, and optimize the overall model through backpropagation. This results in a list of candidate attributes, ultimately forming the attribute set.

[0054] The RoBERTa model is an improved version of BERT (A Robustly Optimized BERT, which is simply called the robustly optimized BERT method).

[0055] 5. Intelligent recommendation. This method extracts user information, such as the user's organization, level, tasks, resources, usage habits, and the most recently used tags, and extracts the feature representation of the user information. It then fuses the extracted tag entity feature representation with the fusion attention mechanism to obtain the fusion of user features and tag entity features. The network model is trained to recommend tags related to the user. The tag library query technology is used to recommend relevant information of the tag instance and visualize it. Figure 4 This is the model diagram of the recommended method.

[0056] The specific steps of intelligent recommendation are as follows:

[0057] Step 1: Extract user features: Select relevant user information from the database: the user's organization, level, tasks, resources responsible, user usage habits, and the most recently used tags, etc. First, obtain the vector value of the user information through pre-trained word vectors, and then use a convolutional neural network (CNN) to extract information features related to the user to form a user feature matrix.

[0058] Step 2: Extract tag library features: Get the tag entity from the tag library and record it as Represents the jth entity in the set of labeled entities corresponding to the kth relationship. Entities corresponding to different relationships are mapped to low-dimensional vector spaces to obtain the label-entity embedding matrix, which is then fed into the KGCNN model to obtain the label-entity feature matrix S. r .

[0059] In the convolution layer, the label-entity features are extracted by different window sizes, and each possible position in the embedding matrix is ​​filtered. Finally, the maximum pooling method is used for feature selection to obtain the movie entity matrix.

[0060]

[0061] Among them E r Represents the embedding matrix of label-entity, G represents the filter kernel for feature extraction, ⊙ represents the dot product operation, relu represents the nonlinear activation function, b represents the bias parameter of relu, and max represents the maximum pooling operation.

[0062] Step 3: Fusion of user features and label entity features: transform the label-entity feature matrix S r Together with the user-movie tag embedding matrix T, it is used as the input of the hybrid attention model to calculate the hybrid attention weight α of different relationship tags-entities T ,

[0063]

[0064] Among them, α T Represents the attention weight; the matmul function is a matrix multiplication function, T u and After multiplication, the attention weight is obtained by normalization through the softmax function.

[0065] The output of the KGCNN model and the rest of the label feature matrix are sent to the multi-layer perceptron MLP. After training through multiple fully connected neural network layers in the MLP, they are multiplied by the hybrid attention weight to obtain the label entity feature matrix I.

[0066] Step 4: Tag and Tag Instance Recommendation: User features and tag entity features are fused and passed through a fully connected layer to ultimately obtain user-related tags. These tags are then searched against the established tag and tag instance library using query technology to obtain user-related tags and tag instances, which are then visualized and recommended.

[0067] This recommendation method based on the special fusion of user features and label entities has the advantage of taking into account the user's own characteristics, using label records and user label preferences. At the same time, it uses the KGCNN network model based on the attention mechanism to learn the impact of the semantic features of labels and the complex behavioral characteristics of users on recommendations, making the recommendation results more accurate and effective.

[0068] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0069] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1A step that specifies a function in one or more boxes.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A method for recommending tags and tag instances, characterized in that: The following steps are involved: Step SS1: Label ontology concept construction, including: collecting structured data, unstructured data and semi-structured data of the experiment, constructing ontology concepts according to the experimental data type, and forming a label library; Step SS2: label entity feature extraction, including: using different methods to extract equipment entities and entity relationships from multimodal data, and building a label instance library, wherein the multimodal data includes image data, text data, audio data, video data, and paper data; Step SS3: mapping labels and label instances, including: mapping the relationship between labels and label instances using rule mapping and natural language processing methods; Step SS4: Generate recommendations for tags and tag instances based on user information, including: mining user personal information and tag usage information, combining personal information and tag information to form intelligent recommendations based on tags and tag instances; the step SS4 includes: extracting user information, including the user's organization, level, tasks, responsible resources, user usage habits, and the most recent tag usage information, extracting the feature representation of the user information, fusing it with the extracted tag entity feature representation, obtaining a fusion of user features and tag entity features through a fusion attention mechanism, training a network model, recommending tags related to the user, recommending relevant information of the tag instance through tag library query technology, and visually displaying it.

2. A tag and tag instance recommendation method according to claim 1, characterized in that: The step SS1 specifically includes the following steps: Step S11: collecting test data, including: mainly using paper data, electronic files, and structured data, designing different data collection methods according to different data sources, extracting structured data, unstructured data, and semi-structured data; extracting data elements from various data, the data elements including equipment identification, equipment name, test data collection time, and test task related information; Step S12: constructing an ontology tag library, including: when constructing the ontology library, using a tree diagram to classify the types and attributes of the collected test data, constructing the ontology concepts of test equipment, test equipment, accompanying test equipment, test tasks and test personnel, and creating a tag library.

3. The method for recommending tags and tag instances according to claim 1, wherein: The step SS2 specifically includes: for the image data, using an image target recognition algorithm based on an improved convolutional neural network model, first marking the equipment target in the original image and converting it into a 224*224 standard image, inputting it into the convolutional neural network model, performing image feature extraction, adding an SE module at the exit of each convolutional layer and Inception layer to update the weight of the image feature parameters, then connecting to the pooling layer to perform dimensionality reduction processing on the features output by the convolutional layer and the Inception layer, inputting the result of the pooling operation output into a fully connected layer to convert it into a one-dimensional vector, and finally inputting it into a softMax classifier to output the recognition result, and using the image recognition model of the improved convolutional neural network model to identify the type of equipment in the picture and the position of the equipment in the picture, and perform entity marking on the picture.

4. The method for recommending tags and tag instances according to claim 1, wherein: The step SS2 specifically includes: using a deep learning method to identify entities in the text data, first, organizing a dictionary containing nouns, wherein the nouns include names of countries, important figures, and equipment names, to form a domain dictionary; then, selecting some texts containing ontology concepts for annotation, annotating the nouns of countries, equipment, and important figures, using a Chinese dictionary and a deep learning network LSTM-CRF model to train a model on the annotated data, and using the trained model to perform entity recognition on large-scale text information; using a keyword extraction model and a relationship extraction model with set rules to obtain the relationship features between the entities extracted from the text, thereby obtaining the entity and relationship information in the text and constructing a label instance library.

5. The method for recommending tags and tag instances according to claim 1, wherein: The step SS2 specifically includes: for the audio data, obtaining the time length, bit rate and sampling rate of the audio from the obtained experimental audio data, using a convolutional neural network to capture the characteristics of the audio loudness changing over time in different frequency domains, and using the SVM support vector machine classification method to classify the audio data, thereby identifying label entity information.

6. A tag and tag instance recommendation method according to claim 1, characterized in that: The step SS2 specifically includes: for the video data, obtaining the time length, encoding format, encoding standard and bit rate characteristics of the acquired video data; at the same time, using a clustering-based method to extract key frames from the video data, extracting entities in the video; for key frame extraction in general videos, using a clustering-based method, grouping video frames using the similarity between video frames, so that similar video frames form a cluster, thereby dividing the video data into clusters one by one, the video frames within the cluster are similar in content, and the video frames between clusters are very different in content; finally, the video frame closest to the cluster center in each cluster is added to the result set; For the key frame extraction of motion videos, a reinforcement learning key frame extraction algorithm that integrates multi-channel features and attention mechanism is used, so that the extraction process does not require shot segmentation and manual feature extraction. The reinforcement learning key frame extraction algorithm starts with the fusion of features. First, a deep neural network is used to extract motion features and static features in the video frame and fuse them. Then, a self-attention model is used to extract the global features of the video sequence. Then, the importance of each video frame is calculated based on the global features to obtain the key frames in the video.

7. The method for recommending tags and tag instances according to claim 1, wherein: The step SS2 specifically includes: for the data source being the paper data, using the optical character recognition (OCR) algorithm, analyzing and identifying the image file of the text material, obtaining the text and layout information, identifying the text in the image, and returning it in the form of text, using a deep learning model based on CRNN, combining the potential of the convolutional neural network (CNN) for image feature engineering with the potential of the long short-term memory network (LSTM) for serialization recognition, extracting robust features, and avoiding the extremely difficult single-character segmentation and single-character recognition in traditional algorithms through serialization recognition. At the same time, serialization recognition also embeds temporal dependency, and can convert paper text into text data through OCR technology, and then perform entity recognition, keyword extraction, and entity relationship extraction on the recognized text through text recognition technology to obtain entity and entity relationship information in the paper data.

8. The method for recommending tags and tag instances according to claim 1, wherein: The step SS3 specifically includes: constructing an ontology concept and a label concept, obtaining entity information through the extracted image, video, text, audio, and paper data features, thereby constructing a label instance, and based on the ontology concept model, adopting a multi-strategy comprehensive method, including a mapping method based on rule mapping and a mapping method based on semantics, mapping labels and label instances from various forms of data, obtaining triples of <label 1-relationship-label 2> and <label 1-relationship-label instance>, and filling them into the current ontology mapping; The mapping method based on rule mapping includes: using an association rule mining method to screen label instances with a high degree of relationship between label ontology concepts and label instances, filling them into a rule model, using a supervised learning support vector machine model to predict the mapping relationship between the mined labels and label instances, and finally, through manual verification of the prediction results, enriching the label instances corresponding to each correctly predicted label into the attribute set; The semantic-based mapping method includes: for attributes that cannot be extracted using association rules, a method of extracting syntactic relations between sentences and semantic relations between words is used, and a RoBERTa and entity boundary prediction model is used. The prediction model includes a text encoding layer, an entity boundary prediction layer, and a BiLSTM_CRF attribute prediction layer. First, the input text is encoded by RoBERTa to obtain its hidden layer state vector, and then the hidden layer state vector is input into the entity boundary prediction layer and the BiLSTM_CRF attribute prediction layer respectively. In the entity boundary prediction layer, the entity header and the entity boundary are respectively encoded by 0 / 1 encoding. The labeling of the tail of the entity, and then calculate the loss values ​​start_loss and end_loss of the two sequence labels. In the BiLSTM_CRF attribute prediction layer, the output result of the entity boundary prediction layer is used as a feature and concatenated with the text vector, input into BiLSTM_CRF to predict the text attribute label, and then calculate its loss value att_loss. Finally, when optimizing the model, the loss value start_loss, loss value end_loss and loss value att_loss are comprehensively considered, and a weighted sum is performed. The overall optimization of the model is achieved through back propagation, and a list of candidate attributes is obtained to finally form the attributes.

9. The method for recommending tags and tag instances according to claim 1, wherein: The step SS4 specifically includes: Step SS41: extracting user features, including: selecting user-related information from the database, including the user's organization, level, assigned tasks, responsible resources, user usage habits, and most recent usage tag information; first, obtaining a vector value of the user information through pre-trained word vectors; then, using a convolutional neural network (CNN), extracting user-related information features to form a user feature matrix; Step SS42: Extract tag library tag features, including: obtaining tag entities from the tag library, recorded as , Representative The first tag entity in the set of label entities corresponding to the relationship Entities, respectively, map the entities corresponding to different relationships into low-dimensional vector space to obtain the label-entity embedding matrix, and feed the matrix into the KGCNN model to obtain the label-entity feature matrix ; In the convolution layer, the label-entity features are extracted by different window sizes, and each possible position in the embedding matrix is ​​filtered. Finally, the maximum pooling method is used for feature selection to obtain the entity matrix. ; ； in, represents the label-entity embedding matrix, represents the filter kernel for feature extraction, represents the dot product operation, represents a nonlinear activation function, express The bias parameters, Represents the maximum pooling operation; Step SS43: Fusion of user features and label entity features, including: transforming the label-entity feature matrix and the user-label embedding matrix Together as the input of the mixed attention model, calculate the mixed attention weights of different relationship labels-entities ; ； in, represents the attention weight, The function is a matrix multiplication function. and After multiplication, Function normalization obtains attention weight; The output of the KGCNN model and the rest of the label feature matrix are sent to the multi-layer perceptron MLP. After training through multiple fully connected neural network layers in the MLP, the label entity feature matrix is ​​obtained by multiplying it with the hybrid attention weight. ; Step SS44 recommends tags and tag instances, including: fusing user features and tag entity features, passing through a fully connected layer, and finally obtaining tags related to the user; using query technology in the established tag and tag instance knowledge graph, obtaining tags and tag instances related to the user, visually displaying them, and forming recommendations.

Images