Text encoding method and apparatus, model training method and apparatus, and model matching method and apparatus

By generating recombinant sequences and feature trees, the problem of not reflecting the dependence of steps and components in multi-structured procedural text encoding is solved, and more accurate text encoding is achieved.

WO2024230350A1PCT designated stage expired Publication Date: 2024-11-14LANGCHAO ELECTRONIC INFORMATION IND CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/084321
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-10
Filing Date
2024-03-28
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

The prior art is difficult to effectively encode multi-structured procedural text, resulting in inaccurate encoding results and inability to reflect the interdependence between multi-structures in the text.

Method used

By obtaining step text and component text in procedural text data, a recombinant sequence and feature tree are generated, and the feature tree is used to reflect the dependence between steps and components, thereby generating more accurate text encoding.

Benefits of technology

It improves the accuracy of procedural text encoding, can effectively reflect the dependence between steps and components, and improves the quality of encoding results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024084321_14112024_PF_FP_ABST
    Figure CN2024084321_14112024_PF_FP_ABST
Patent Text Reader

Abstract

A text encoding method and apparatus, a model training method and apparatus, and a model matching method and apparatus. The text encoding method comprises: acquiring procedural text data, wherein the procedural text data comprises step texts and component texts; according to the order of the step texts, fusing the step texts to generate recombination sequences; according to component features of the component texts, step text features of the step texts, and the recombination sequence, generating a feature tree corresponding to the procedural text data; and according to the feature tree, generating a text code corresponding to the procedural text data.
Need to check novelty before this filing date? Find Prior Art

Description

A text encoding method, model training method, model matching method and device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on May 10, 2023, with application number 202310520313.9, entitled “A text encoding method, model training method, model matching method and device”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to the field of data encoding technology, and in particular to a text encoding method and device, a model training method and device for image-text mutual inspection, a model matching method and device for image-text mutual inspection, an electronic device, and a non-volatile readable storage medium. Background Art

[0004] Procedural text is commonly used in daily life; for example, it records the processes of scientific experiments, repairing items, and cooking dishes. Procedural text is often multi-structured. For example, procedural text can include not only step-by-step information but also various component information such as materials. For example, a procedural text recording a handmade process can include all the materials used and the production steps.

[0005] Currently, encoding multi-structured procedural text remains an unresolved problem. Conventional methods simply encode the textual content of procedural text, resulting in inaccurate encoding that fails to reflect the interdependencies between the multiple structures within the text. Therefore, developing a procedural text encoding method that deeply explores the interdependencies between multiple structures within the text and improves the accuracy of the encoding results is an urgent issue.

[0006] Summary of the Invention

[0007] The purpose of this application is to provide a text encoding method and device, a model training method and device for image-text mutual inspection, a model matching method and device for image-text mutual inspection, an electronic device and a non-volatile readable storage medium, so as to deeply explore the interdependence between multiple structures in procedural text and improve the accuracy of the encoding results.

[0008] To solve the above technical problems, the present application provides a text encoding method, comprising:

[0009] Obtaining procedural text data; wherein the procedural text data includes step text and component text;

[0010] According to the order of the step texts, the texts of each step are fused to generate a recombinant sequence;

[0011] Generate a feature tree corresponding to the procedural text data based on the component features of each component text, the step text features of each step text, and the recombined sequence; wherein the feature tree includes: a chain connection relationship tree corresponding to the recombined sequence, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features;

[0012] Generate text encoding corresponding to procedural text data based on the feature tree.

[0013] In some embodiments, the texts of each step are fused according to the order of the step texts to generate a recombinant sequence, including:

[0014] Obtaining step text features corresponding to each step text in the procedural text data;

[0015] The step text features are combined in the order of the step texts to generate a recombinant sequence; wherein the number of recombinant sequences and step texts is t, the first recombinant sequence includes the first step text feature, the sth recombinant sequence includes the first step text feature to the sth step text feature, and s is a positive integer greater than 1 and less than or equal to t.

[0016] In some embodiments, a feature tree corresponding to the procedural text data is generated based on the component features of each component text, the step text features of each step text, and the recombined sequence, including:

[0017] Obtaining the recombinant sequence characteristics corresponding to each recombinant sequence;

[0018] Obtain the correlation coefficient between the text features of each step and the features of each recombinant sequence;

[0019] Establishing a spanning tree corresponding to the procedural text data based on the association coefficient; wherein the spanning tree includes a chain-connected relationship tree and step text features mounted on the chain-connected relationship tree, the chain-connected relationship tree being a tree of chain-connected recombinant sequences established according to the length of the recombinant sequences, and the root node of the chain-connected relationship tree being the longest recombinant sequence;

[0020] Obtaining component features corresponding to each component text in the procedural text data;

[0021] According to the correspondence between each component text and each step text, each component feature is mounted on the corresponding step text feature in the generation tree to generate a feature tree corresponding to the procedural text data.

[0022] In some embodiments, obtaining the recombinant sequence characteristics corresponding to each recombinant sequence includes:

[0023] Each recombinant sequence is input into a bidirectional long short-term memory network respectively, and the recombinant sequence features corresponding to each recombinant sequence output by the bidirectional long short-term memory network are obtained.

[0024] In some embodiments, obtaining the correlation coefficient between the text features of each step and each recombinant sequence feature includes:

[0025] The distance between the text features of each step and the features of each recombinant sequence is calculated to obtain the correlation coefficient.

[0026] In some embodiments, generating text encoding corresponding to procedural text data based on the feature tree includes:

[0027] According to the structure of the feature tree, the step text features and component features are aggregated in order from bottom to top to generate text encoding.

[0028] In some embodiments, according to the structure of the feature tree, the step text features and component features are aggregated in order from bottom to top to generate text encoding, including:

[0029] According to the structure of the feature tree, the step text features and component features are aggregated in a dual-feature aggregation manner in a bottom-up order to generate text encoding.

[0030] In some embodiments, according to the structure of the feature tree, step text features and component features are aggregated in a dual-feature aggregation manner in a bottom-up order to generate text encoding, including:

[0031] Get the bottom-level features in the current feature tree; the bottom-level features include step text features, component features, and aggregation features;

[0032] If the number of bottom-level features is 1 and there is no upper-level feature of the previous cascade layer, the text encoding is determined based on the bottom-level features;

[0033] If the number of bottom-level features is 1 and there are upper-level features, the bottom-level features are aggregated with the upper-level features in a dual-feature aggregation manner to generate corresponding aggregated features and update the current feature tree, and the step of obtaining the bottom-level features in the current feature tree is executed;

[0034] If the number of underlying features is greater than 1, any two underlying features are aggregated in a dual-feature aggregation manner to generate corresponding aggregated features and update the current feature tree, and execute the step of obtaining the lowest underlying feature in the current feature tree.

[0035] In some embodiments, the bottom layer features and the upper layer features are aggregated in a dual feature aggregation manner to generate corresponding aggregated features and update the current feature tree, including:

[0036] pass Aggregate the bottom layer features with the upper layer features to generate the corresponding aggregate features; where c j is the aggregation feature, h j =o j tanh(c j ), f jk =σ(U f ·h k +b f ), j = 1, k = 2 or 3, c k are the bottom layer features and the upper layer features, W, U and b are preset matrices, h j c j The corresponding hidden state; h k c k The corresponding hidden state, c k When it is a step text feature or a component feature, h k is a randomly initialized vector.

[0037] The present application also provides a text encoding device, comprising:

[0038] A text acquisition module is used to acquire procedural text data; wherein the procedural text data includes step text and component text;

[0039] The step fusion module is used to fuse the step texts according to the order of the step texts to generate a recombinant sequence;

[0040] A tree generation module is used to generate a feature tree corresponding to the procedural text data based on the component features of each component text, the step text features of each step text, and the recombined sequence; wherein the feature tree includes: a chain connection relationship tree corresponding to the recombined sequence, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features;

[0041] The text encoding module is used to generate text encoding corresponding to program text data based on the feature tree.

[0042] This application also provides a model training method for image-text mutual inspection, including:

[0043] Acquire training data; wherein the training data includes a preset number of training pairs of procedural text data and step diagram data;

[0044] Obtaining a text encoding for each procedural text data; wherein the text encoding is an encoding generated using a feature tree corresponding to the procedural text data, the feature tree including a chain connection relationship tree, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features, the step text features being features of the step text in the procedural text data, the component features being features of the component text in the procedural text data, and the chain connection relationship tree being a tree corresponding to a recombined sequence obtained by fusing each step text in the order of the step texts;

[0045] Obtain image encoding of each step graph data;

[0046] According to the text encoding and image encoding, the preset matching model is trained using the preset loss function to obtain the trained retrieval matching model, so as to use the retrieval matching model to match the procedural text data to be retrieved or the step diagram data to be retrieved.

[0047] In some embodiments, the preset loss function includes a ternary loss function or a ternary loss function and a tree structure loss function, and the ternary loss function is Encode any image among the N images in the current batch, For the current batch Paired text encoding, For the current batch Unpaired text encoding, To preset hyperparameters, For any text encoding among the N text encodings in the current batch, For the current batch Paired image encoding, For the current batch Unpaired image encoding; tree structure loss functions include local structure alignment loss, global structure alignment loss and / or node degree loss.

[0048] In some embodiments, when the preset loss function includes a ternary loss function and a tree structure loss function, the preset matching model is trained using the preset loss function according to the text encoding and the image encoding to obtain a trained retrieval matching model, including:

[0049] Get the preset target tree corresponding to the procedural text data;

[0050] According to the text encoding, image encoding, feature tree and preset target tree, the preset matching model is trained using the preset loss function to obtain a trained retrieval matching model.

[0051] In some embodiments, a preset matching model is trained using a preset loss function based on text encoding, image encoding, feature tree, and preset target tree to obtain a trained retrieval matching model, including: an output process of a tree structure loss function; wherein, when the tree structure loss function includes local structure comparison loss, global structure comparison loss, and node degree loss, the output process of the tree structure loss function includes:

[0052] Constructing a first adjacency matrix corresponding to each node according to the node attributes and neighbor nodes of each node in the current feature tree and the current preset target tree; wherein the current feature tree is a feature tree of the current procedural text data, the current procedural text data is any procedural text data for training, and the current preset target tree is a preset target tree corresponding to the current procedural text data;

[0053] Generate a first embedding vector matrix corresponding to the current feature tree and a second embedding vector matrix corresponding to the current preset target tree according to the first adjacency matrix;

[0054] Calculate the similarity between the first embedding vector matrix and the second embedding vector matrix to obtain a local structure similarity matrix;

[0055] According to the local structure similarity matrix, the local structure comparison loss corresponding to the current procedural text data is obtained;

[0056] Constructing a second adjacency matrix corresponding to the current feature tree and the current preset target tree according to the node attributes and neighbor nodes of each node in the current feature tree and the current preset target tree;

[0057] Generate a first embedding vector corresponding to the current feature tree and a second embedding vector corresponding to the current preset target tree according to the second adjacency matrix;

[0058] Calculate the similarity between the first embedding vector and the second embedding vector to obtain the global structural similarity;

[0059] According to the global structural similarity, the global structural comparison loss corresponding to the current procedural text data is obtained;

[0060] Calculate the degree of each node in the current feature tree and the current preset target tree;

[0061] Calculate the degree difference of all nodes in the current feature tree and the current preset target tree to obtain the degree difference matrix;

[0062] According to the degree difference matrix, obtain the node degree loss corresponding to the current procedural text data;

[0063] The local structure comparison loss, global structure comparison loss and node degree loss corresponding to the current procedural text data are added together to obtain the tree structure loss function corresponding to the current procedural text data.

[0064] In some embodiments, generating, based on the first adjacency matrix, a first embedding vector matrix corresponding to the current feature tree and a second embedding vector matrix corresponding to the current preset target tree includes:

[0065] Calculate the Laplacian matrix corresponding to each first adjacency matrix;

[0066] Calculate the eigenvectors of each Laplacian matrix;

[0067] Obtaining, based on the eigenvectors of the Laplacian matrices, a vector representation corresponding to each of the first adjacency matrices; wherein the vector representation corresponding to the current first adjacency matrix includes a first preset number of eigenvectors of the Laplacian matrix corresponding to the current first adjacency matrix, sorted in ascending order of eigenvalue;

[0068] A first embedding vector matrix and a second embedding vector matrix are generated according to the vector representations corresponding to the first adjacency matrices.

[0069] In some embodiments, obtaining the local structure comparison loss corresponding to the current procedural text data according to the local structure similarity matrix includes:

[0070] By L1=||H|| 2 , the local structure comparison loss is calculated; L1 is the local structure comparison loss, and H is the local structure similarity matrix.

[0071] In some embodiments, when the preset loss function includes a ternary loss function, the preset matching model is trained using the preset loss function according to the text encoding and the image encoding to obtain a trained retrieval matching model, including:

[0072] Initialize the weights of all network layers in the preset matching model to obtain the initialized preset matching model;

[0073] Inputting the current batch of training data into the preset matching model respectively to obtain the training matching results output by the preset matching model; wherein the training data includes N pairs of procedural text data and step diagram data of the current batch;

[0074] Determine whether the preset number of iterations has been reached or the error between the training matching result and the preset matching result is less than the convergence threshold;

[0075] If the preset number of iterations is reached or the error between the training matching result and the preset matching result is less than the convergence threshold, the preset matching model is used as the trained retrieval matching model;

[0076] If the preset number of iterations is not reached and the error between the training matching result and the preset matching result is not less than the convergence threshold, the preset loss function is used to obtain the loss value corresponding to the training matching result;

[0077] According to the loss value, obtain the back propagation error of each network layer in the preset matching model;

[0078] According to the back-propagation error, the weights of each network layer in the preset matching model are updated to obtain an updated preset matching model;

[0079] The next batch of training data is selected and used as the training data of the current batch, and the steps of inputting the training data of the current batch into the preset matching model are executed to obtain the training matching results output by the preset matching model.

[0080] In some embodiments, obtaining the image code of each step diagram data includes:

[0081] Extracting step diagram features of each step diagram in the current step diagram data; wherein the current step diagram data is any step diagram data;

[0082] Generate a heterogeneous graph corresponding to the current step graph data based on the category labels and step graph features corresponding to each step graph; wherein the heterogeneous graph includes component nodes corresponding to the category labels, step graph nodes corresponding to the step graph features, and connection edges between each step graph node and its corresponding component nodes;

[0083] According to the heterogeneous graph, the image encoding corresponding to the current step graph data is generated.

[0084] In some embodiments, extracting step diagram features of each step diagram in the step diagram data includes:

[0085] The current step graph is input into a preset residual network, a feature vector output by a pooling layer of the preset residual network is obtained, and the feature vector is determined as a step graph feature corresponding to the current step graph; wherein the current step graph is any step graph.

[0086] This application also provides a model training device for image-text mutual inspection, including:

[0087] A data acquisition module is used to acquire training data; wherein the training data includes a preset number of training pairs of procedural text data and step diagram data;

[0088] A text encoding module is used to obtain text encodings for each procedural text data; wherein the text encoding is an encoding generated using a feature tree corresponding to the procedural text data, the feature tree including a chain connection relationship tree, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features, the step text features being features of the step texts in the procedural text data, the component features being features of the component texts in the procedural text data, and the chain connection relationship tree being a tree corresponding to a recombined sequence obtained by fusing the step texts in order of the step texts;

[0089] An image coding module, used to obtain image coding of each step graph data;

[0090] The model training module is used to train the preset matching model based on text encoding and image encoding using a preset loss function, obtain the trained retrieval matching model, and use the retrieval matching model to match the procedural text data to be retrieved or the step diagram data to be retrieved.

[0091] This application also provides a model matching method for image-text mutual inspection, including:

[0092] Acquire data to be retrieved; wherein the data to be retrieved is procedural text data to be retrieved or step diagram data to be retrieved;

[0093] A retrieval matching model is used to extract the data encoding of the data to be retrieved, and the data encoding is distance-matched with the sample encoding of each sample data in the sample data set to obtain matching sample data corresponding to the data to be retrieved; wherein the sample data set is a sample step diagram data set or a sample procedural text data set; if the data to be retrieved is procedural text data to be retrieved, the data encoding is a text encoding generated by using a feature tree corresponding to the procedural text data to be retrieved, and the feature tree includes a chain connection relationship tree, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features, the step text features are features of the step text in the procedural text data to be retrieved, the component features are features of the component text in the procedural text data to be retrieved, and the chain connection relationship tree is a tree corresponding to the recombinant sequence obtained by fusing each step text in the order of the step text.

[0094] In some embodiments, performing distance matching on the data code and the sample code of each sample data in the sample data set to obtain matching sample data corresponding to the data to be retrieved includes:

[0095] Calculate the Euclidean distance between the data code and the sample code of each sample data respectively;

[0096] The sample data corresponding to the minimum Euclidean distance is taken as the matching sample data.

[0097] This application also provides a model matching device for image-text mutual inspection, including:

[0098] A retrieval module is used to obtain data to be retrieved; wherein the data to be retrieved is procedural text data to be retrieved or step diagram data to be retrieved;

[0099] The model matching module is used to use the retrieval matching model to extract the data encoding of the data to be retrieved, and perform distance matching on the data encoding with the sample encoding of each sample data in the sample data set to obtain the matching sample data corresponding to the data to be retrieved; wherein the sample data set is a sample step diagram data set or a sample procedural text data set; if the data to be retrieved is procedural text data to be retrieved, the data encoding is a text encoding generated by using a feature tree corresponding to the procedural text data to be retrieved, and the feature tree includes a chain connection relationship tree, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features, the step text features are features of the step text in the procedural text data to be retrieved, the component features are features of the component text in the procedural text data to be retrieved, and the chain connection relationship tree is a tree corresponding to the recombinant sequence obtained by fusing each step text in the order of the step text.

[0100] The present application also provides an electronic device, comprising:

[0101] memory for storing computer programs;

[0102] The processor is used to implement the steps of the above-mentioned text encoding method, the above-mentioned model training method for image-text mutual verification and / or the above-mentioned model matching method for image-text mutual verification when executing the computer program.

[0103] In addition, the present application also provides a non-volatile readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned text encoding method, the above-mentioned image-text mutual inspection model training method and / or the above-mentioned image-text mutual inspection model matching method are implemented.

[0104] A text encoding method provided by the present application includes: obtaining procedural text data; wherein the procedural text data includes step text and component text; according to the order of the step text, fusing each step text to generate a recombined sequence; generating a feature tree corresponding to the procedural text data based on the component features of each component text, the step text features of each step text and the recombined sequence; wherein the feature tree includes: a chain connection relationship tree corresponding to the recombined sequence, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features; based on the feature tree, generating a text encoding corresponding to the procedural text data.

[0105] It can be seen that the present application generates a feature tree corresponding to the procedural text data based on the component features of each component text, the step text features of each step text, and the recombinant sequence, establishes a tree relationship based on the structure of the step information and component information in the procedural text, and uses the tree relationship to reflect the interdependence between the steps and the components; by generating the text encoding corresponding to the procedural text data based on the feature tree, the encoding result can include the dependency between the steps and components in the procedural text, thereby improving the accuracy of the encoding result. In addition, the present application also provides a text encoding device, a model training method and device for text-image mutual inspection, a model matching method and device for text-image mutual inspection, an electronic device, and a non-volatile readable storage medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0106] In order to more clearly illustrate some embodiments of the present application or technical solutions in the prior art, the following briefly introduces the drawings required for use in some embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0107] FIG1 is a flowchart of a text encoding method provided by some embodiments of the present application;

[0108] FIG2 is a schematic diagram of a feature tree of another text encoding method provided by some embodiments of the present application;

[0109] FIG3 is a flowchart of a feature tree generation process of another text encoding method provided by some embodiments of the present application;

[0110] FIG4 is a schematic diagram of components of a probability matrix of another text encoding method provided by some embodiments of the present application;

[0111] FIG5 is a diagram showing a probability matrix of another text encoding method provided by some embodiments of the present application;

[0112] FIG6 is a schematic diagram of a chain connection relationship tree of another text encoding method provided by some embodiments of the present application;

[0113] FIG7 is a schematic diagram of a dual-feature aggregation method provided by some embodiments of the present application;

[0114] FIG8 is a schematic diagram of another dual-feature aggregation method provided by some embodiments of the present application;

[0115] FIG9 is a schematic diagram of a basic unit structure of a tree-shaped long short-term memory network of another text encoding method provided by some embodiments of the present application;

[0116] FIG10 is a flowchart of a model training method for image-text mutual checking provided by some embodiments of the present application;

[0117] FIG11 is a flowchart of an image encoding process of another image-text mutual inspection model training method provided by some embodiments of the present application;

[0118] FIG12 is a schematic diagram of the structure of a ResNet backbone network of another model training method for image-text mutual verification provided by some embodiments of the present application;

[0119] FIG13 is a flow chart of another model training method for image-text mutual verification provided by some embodiments of the present application;

[0120] FIG14 is a schematic diagram of heterogeneous graph construction of another model training method for image-text mutual verification provided by some embodiments of the present application;

[0121] FIG15 is a flowchart of an output process of a local structure comparison loss in another model training method for image-text mutual verification provided by some embodiments of the present application;

[0122] FIG16 is a flowchart of a model matching method for image-text mutual verification provided by some embodiments of the present application;

[0123] FIG17 is a structural block diagram of a text encoding device provided in some embodiments of the present application;

[0124] FIG18 is a structural block diagram of a model training device for image-text mutual checking provided by some embodiments of the present application;

[0125] FIG19 is a structural block diagram of a model matching device for image-text mutual inspection provided by some embodiments of the present application;

[0126] FIG20 is a schematic structural diagram of an electronic device provided by some embodiments of the present application;

[0127] FIG21 is a schematic diagram of a specific structure of an electronic device provided in some embodiments of the present application;

[0128] FIG22 is a schematic structural diagram of a non-volatile readable storage medium provided in some embodiments of the present application. DETAILED DESCRIPTION

[0129] To make the objectives, technical solutions, and advantages of some embodiments of the present application more clear, the technical solutions in some embodiments of the present application will be clearly and completely described below in conjunction with the drawings in some embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0130] Please refer to Figure 1, which is a flowchart of a text encoding method provided by some embodiments of the present application. The method may include:

[0131] Step 101: Obtain procedural text data; wherein the procedural text data includes step text and component text.

[0132] Among them, the procedural text data in some embodiments of the present application can be the data of any procedural text, that is, some embodiments of the present application are demonstrated by taking the encoding of a procedural text data as an example. For the encoding of multiple procedural text data, it can be implemented in the same or similar manner as the method provided in some embodiments of the present application. Some embodiments of the present application do not impose any restrictions on this.

[0133] Specifically, the procedural text data in some embodiments of the present application may include step text (i.e., step information) corresponding to the step information and ingredient text corresponding to the ingredient information. For example, when the procedural text data is a recipe operation step text, the recipe operation step text may include cooking step information (i.e., step text) and material information (i.e., ingredient text). The specific data types of the procedural text data in some embodiments of the present application can be set by the designer according to practical scenarios and user needs. For example, the procedural text data may be a recipe operation step text, that is, the procedural text data includes cooking step information (i.e., step text) and material information (i.e., ingredient text); the procedural text data may also be a handicraft production step text, that is, the procedural text data may include production step information (i.e., step text) and tool information (i.e., ingredient text); the procedural text data may also be a paper abstract text, that is, the procedural text data may include abstract step information (i.e., step text) and keyword information (i.e., ingredient text). Some embodiments of the present application do not impose any restrictions on this.

[0134] For example, if the procedural text data is a step-by-step guide for making a paper cup flower, it could include Step 1: First, use a knife to cut the paper cup approximately 2 cm from the bottom, splitting it into two parts: the cup body and the cup base. Step 2: Take the cut cup base and use scissors to cut evenly spaced notches along the remaining cup wall. Step 3: Continue folding the cut cup wall outward to form the flower petals. Select any two adjacent petals and cut notches approximately two-thirds of the width near the cup base, with the two notches facing each other. Step 4: Next, flatten the cup body and fold the two sides inward to form an elongated triangle, making the cup body a trapezoid. Step 5: Then, push inward along the fold lines on both sides of the cup body, fold the two sides inward, and cut off the shaded area to form the leaves and stem of the flower. Step 6: Assemble the cup body and cup base, and arrange the leaves. The disposable paper cup flower is now complete. The step-by-step guide for making a paper cup flower could include the following tools: a knife for Step 1 and scissors for Step 2.

[0135] Step 102: According to the order of the step texts, the step texts are merged to generate a recombinant sequence.

[0136] It can be understood that the procedural text data in some embodiments of the present application can be a paragraph consisting of multiple sentences. The procedural text data can contain semantic information (i.e., step text and component text), as well as information about the interdependence between steps and steps, and between steps and components, also known as workflow information. For example, in the above-mentioned paper cup flower making step text, steps 2 and 3 and steps 4 and 5 are respectively: the steps for making petals and leaves, which are sub-steps of step 6; at the same time, the paper cup flower making step text also contains tool information (such as scissors and knives); in some embodiments of the present application, the steps of the multi-structured procedural text data shown in the above-mentioned paper cup flower making step text can be sorted into a hierarchical and mutually integrated tree relationship, and the tree relationship reflects the interdependence between steps and steps and steps and tools. That is to say, in some embodiments of the present application, a multi-structured text (i.e., procedural text data) relationship mining scheme based on a spanning tree can be provided to mine the tree-like hierarchical relationship in the multi-structured text.

[0137] In this step, the processor can fuse the step texts according to the order of the step texts to generate a recombination sequence, and use the recombination sequence to establish a spanning tree. The step texts in this step can be the text content of each step in the procedural text data; the recombination sequence in this step can be a sequence generated by fusing the step texts according to the order of the step texts. That is, each recombination sequence can correspond to one or more step texts.

[0138] Specifically, in this step, the processor fuses the step texts according to the order of the step texts to generate the specific method of the recombinant sequence, that is, the specific fusion method of the step texts, can be set by the designer according to the practical scenario and user needs. For example, the processor can directly combine the step texts according to the order of the step texts to generate the recombinant sequence; for example, the number of recombinant sequences and step texts can be the same, such as the number of recombinant sequences and step texts are both t, the 1st recombinant sequence can include the 1st step text, the sth recombinant sequence includes the 1st step text to the sth step text, and s is a positive integer greater than 1 and less than or equal to t. The processor can also combine the step text features corresponding to each step text according to the order of the step texts to generate the recombinant sequence; that is, in this step, the processor can obtain the step text features corresponding to each step text in the procedural text data; combine the step text features according to the order of the step texts to generate the recombinant sequence. The number of recombinant sequences and step texts may be the same, for example, if the number of recombinant sequences and step texts is t, the first recombinant sequence includes the first step text feature, the sth recombinant sequence includes the first step text feature to the sth step text feature, and s is a positive integer greater than 1 and less than or equal to t; the number of recombinant sequences and step texts may also be different, for example, if the number of step texts is t, the number of recombinant sequences is t-1, the qth recombinant sequence includes the first step text feature to the q+1th step text feature, and q is a positive integer greater than or equal to 1 and less than or equal to t-1. Some embodiments of the present application do not impose any restrictions on this.

[0139] For example, the processor can obtain the text of each step in the procedural text data, input all the step texts into the text encoder in sequence, and obtain the features of each step text output by the text encoder (i.e., step text features), which can be recorded as the first step text feature (i.e., step 1 text feature),..., the tth step text feature (i.e., step t text feature). For example, this embodiment can adopt a text encoder based on transformer (a neural network), that is, the processor can use the transformer model to extract the step text features of each step text; the processor can also use other methods to extract the step text features of each step text, and some embodiments of the present application do not impose any restrictions on this.

[0140] Correspondingly, the processor can combine the step text features in the order of the step texts to obtain a recombinant sequence, such as the number of recombinant sequences and step texts is t, the 1st recombinant sequence includes the 1st step text feature, the sth recombinant sequence includes the 1st step text feature to the sth step text feature, s is a positive integer greater than 1 and less than or equal to t; that is, step 1 text feature → recombinant sequence 1 (that is, the 1st recombinant sequence), step 1 text feature and step 2 text feature → recombinant sequence 2, step 1 text feature, step 2 text feature and step 3 text feature → recombinant sequence 3, ..., step 1 text feature - step t text feature → recombinant sequence t (that is, the tth recombinant sequence).

[0141] Step 103: Generate a feature tree corresponding to the procedural text data based on the component features of each component text, the step text features of each step text, and the recombinant sequence; wherein the feature tree includes: a chain connection relationship tree corresponding to the recombinant sequence, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features.

[0142] It can be understood that in this step, the processor can generate a tree relationship (i.e., a feature tree) corresponding to the procedural text data based on the characteristics of each component text in the procedural text data (i.e., component characteristics), the characteristics of each step text in the procedural text data (i.e., step text characteristics) and each recombinant sequence, so as to use the feature tree to reflect the interdependent relationship between steps and steps and between steps and components.

[0143] Among them, the feature tree in some embodiments of the present application may include a chain-connected tree corresponding to the recombinant sequence (i.e., a chain-connected relationship tree), all or part of the nodes on the chain-connected relationship tree (such as the recombinant sequence) can be mounted with corresponding step text features, and all or part of the step text features can be mounted with corresponding component features. As shown in Figure 2, the feature tree may include a chain-connected relationship tree composed of recombinant sequences 2 and 3 (i.e., recombinant sequences), step 1-3 text features (i.e., step text features) mounted on recombinant sequences 2 and 3, and component 1-4 features (i.e., component features) mounted on step 1-3 text features.

[0144] Specifically, the specific manner in which the processor generates a feature tree corresponding to the procedural text data in this step based on the component features of each component text, the step text features of each step text, and the recombined sequence can be set by the designer according to practical scenarios and user needs. As shown in FIG3 , this step may include:

[0145] Step 1031: Obtain the recombinant sequence features corresponding to each recombinant sequence.

[0146] In this step, the processor may obtain the corresponding features of each recombinant sequence (ie, the recombinant sequence features).

[0147] Specifically, some embodiments of the present application do not limit the specific manner in which the processor obtains the recombination sequence features corresponding to each recombinant sequence. For example, when the recombinant sequence includes corresponding step text features, that is, when the recombinant sequence is obtained by combining step text features, the processor can use a bidirectional long short-term memory network (BiLSTM, Bi-directional Long Short-Term Memory) to obtain the recombination sequence features corresponding to each recombinant sequence; for example, the processor can input each recombinant sequence into a bidirectional long short-term memory network to obtain the recombination sequence features corresponding to each recombinant sequence output by the bidirectional long short-term memory network; when the recombinant sequence includes corresponding step texts, that is, when the recombinant sequence is obtained by combining step texts, the processor can also first obtain the step text features corresponding to each recombinant sequence, and then use a bidirectional long short-term memory network based on the step text features corresponding to each recombinant sequence to obtain the recombination sequence features corresponding to each recombinant sequence. As long as the processor can obtain a recombinant sequence feature corresponding to each recombinant sequence, some embodiments of the present application do not impose any restrictions on this.

[0148] Step 1032: Obtain the correlation coefficient between the text features of each step and the features of each recombinant sequence.

[0149] Among them, the correlation coefficient in this step can be the numerical value of the correlation between the step text feature and the corresponding recombinant sequence feature. Some embodiments of the present application do not limit the specific method in which the processor obtains the correlation coefficient between each step text feature and each recombinant sequence feature, such as the processor can calculate the distance between each step text feature and each recombinant sequence feature (such as Euclidean distance, cosine similarity distance, Hamming distance, Chebyshev distance, etc.) to obtain the correlation coefficient; for example, the processor can calculate the Euclidean distance between the current step text feature and the current recombinant sequence feature, and use the Euclidean distance as the correlation coefficient between the current step text feature and the current recombinant sequence feature; the current step text feature is any step text feature, and the current recombinant sequence feature is any recombinant sequence feature.

[0150] Specifically, some embodiments of the present application are demonstrated by taking the processor calculating the correlation coefficient between each step text feature and each recombination sequence feature as an example. The processor can also calculate the correlation coefficient between the target step text feature and the corresponding target recombination sequence feature. As shown in the probability matrix of Figures 4 and 5, the target step text feature and the corresponding target recombination sequence feature can be the step text feature and target recombination sequence feature corresponding to the correlation coefficient in the upper triangular matrix (such as A12, A13, A22, A23 and A31, etc.), that is, the target step text feature can be a step text feature other than the first step text feature; the target recombination sequence feature corresponding to the sth step text feature can be the sth recombination sequence feature to the tth recombination sequence feature, and the number of recombination sequences and step texts is t.

[0151] Step 1033: Based on the correlation coefficient, a spanning tree corresponding to the procedural text data is established; wherein the spanning tree includes a chain connection relationship tree and step text features mounted on the chain connection relationship tree, the chain connection relationship tree is a tree of chain-connected recombinant sequences established according to the length of the recombinant sequences, and the root node of the chain connection relationship tree is the longest recombinant sequence.

[0152] Specifically, in this step, the processor can use the correlation coefficient between each step text feature and each recombinant sequence feature to establish a spanning tree corresponding to the procedural text data. For example, the processor can construct a probability matrix composed of step text features and recombinant sequence features; and use the correlation coefficient within the upper triangular matrix in the probability matrix to establish a spanning tree corresponding to the procedural text data. The area outside the upper triangular matrix in the probability matrix as shown in Figures 4 and 5 can indicate that the corresponding step text and the recombinant sequence are not the association relationship required for the spanning tree, such as in Figure 4, the text feature of step 3 has no association relationship with the recombinant sequence 2.

[0153] Correspondingly, the processor can first establish a tree of chain-connected recombinant sequences (i.e., a chain-connected relationship tree) according to the length of the recombinant sequence (i.e., the sequence extension relationship). That is, the root node of the chain-connected relationship tree is the longest recombinant sequence. For example, in the chain-connected relationship tree shown in FIG6 , the root node can be recombinant sequence 3. Then, according to the correlation coefficient between each step text feature and each recombinant sequence feature, the step text feature is hung on the recombinant sequence corresponding to the largest correlation coefficient. As shown in FIG5 to 7 , the correlation coefficient between the step 1 text feature and the recombinant sequence 2 feature is the largest, that is, the probability that the step 1 text feature belongs to the recombinant sequence 2 is the highest. A connecting line can be established between the step 1 text feature and the recombinant sequence 2, so that the step 1 text feature is hung on the recombinant sequence 2.

[0154] Step 1034: Obtain component features corresponding to each component text in the procedural text data.

[0155] In this step, the processor may obtain the features (ie, component features) corresponding to each component text in the procedural text data.

[0156] Specifically, some embodiments of the present application do not limit the specific method by which the processor obtains the component features corresponding to each component text in the procedural text data. For example, the processor can use a transformer encoder to obtain the component features corresponding to each component text; for example, the processor can input each component text into the transformer encoder respectively to obtain the component features corresponding to each component text output by the transformer encoder.

[0157] It should be noted that some embodiments of the present application do not limit the specific time when the processor obtains the component features corresponding to each component text in the procedural text data. As shown in some embodiments of the present application, the processor can obtain the component features corresponding to each component text after establishing the spanning tree corresponding to the procedural text data, or it can obtain the recombinant sequence features corresponding to each recombinant sequence and the component features corresponding to each component text in step 1031.

[0158] Step 1035: Based on the correspondence between each component text and each step text, each component feature is mounted onto the corresponding step text feature in the generation tree to generate a feature tree corresponding to the procedural text data.

[0159] The correspondence between each component text and each step text may be an inclusion relationship between the step text and the component text, that is, whether the step text contains the corresponding component text; for example, if the step 1 text is to wash and chop the tomatoes for later use, and the component text is tomatoes, then the step 1 text contains the component text (tomatoes). In other words, the processor can determine the correspondence between the component text and the step text by judging whether the step text contains the component text. If the step text contains the component text, the correspondence relationship may be 1; if the step text does not contain the component text, the correspondence relationship may be 0.

[0160] Specifically, in this step, the processor mounts each component feature onto the corresponding step text feature in the generation tree according to the correspondence between each component text and each step text, and generates a specific method of generating a feature tree corresponding to the procedural text data. The specific method can be set by the designer. For example, the processor can traverse all component texts and step texts to establish a correspondence between each component text and each step text; according to the established correspondence, each component feature is mounted onto the corresponding step text feature in the generation tree to generate a feature tree corresponding to the procedural text data.

[0161] It should be noted that the feature tree generation process shown in Figure 3 is demonstrated by taking each recombinant sequence directly as a node on the chain connection relationship tree in the feature tree as an example; some recombinant sequences can also be directly used as nodes on the chain connection relationship tree in the feature tree, such as using other recombinant sequences other than the first recombinant sequence as nodes on the chain connection relationship tree in the feature tree; the identifier corresponding to the recombinant sequence (such as the recombinant sequence feature) can also be used as a node on the chain connection relationship tree in the feature tree, and some embodiments of the present application do not impose any restrictions on this.

[0162] Step 104: Generate text encoding corresponding to the procedural text data based on the feature tree.

[0163] Among them, in some embodiments of the present application, the processor can use the step text features and component features mounted on the feature tree to generate text encoding corresponding to the procedural text data, so that the encoding result (i.e., text encoding) can include the dependency relationship between steps and steps and between steps and components.

[0164] Specifically, for some embodiments of the present application, the specific manner in which the processor generates text encoding corresponding to procedural text data according to the feature tree can be set by the designer according to practical scenarios and user needs. For example, the processor aggregates the step text features and component features in a bottom-up order according to the structure of the feature tree to generate a text encoding; For example, the processor can aggregate the step text features and component features in a dual-feature aggregation manner according to the structure of the feature tree in a bottom-up order to generate a text encoding, that is, the processor can aggregate two features at a time, such as two component features, one component feature and one step text feature, one component feature and one aggregation feature, one step text feature and one aggregation feature, etc.; the processor can also aggregate the step text features and component features in a three-feature aggregation manner according to the structure of the feature tree in a bottom-up order to generate a text encoding. Some embodiments of the present application do not impose any restrictions on this.

[0165] Correspondingly, the above-mentioned processor aggregates the step text features and component features in a dual-feature aggregation manner in accordance with the structure of the feature tree from bottom to top to generate a specific method of text encoding, which can be set by the designer. For example, the processor can obtain the bottom-level features in the current feature tree; wherein the bottom-level features include step text features, component features and aggregation features; the current feature tree can be the current feature tree.

[0166] If the number of underlying features is 1 and there is no upper-layer feature in the cascaded upper layer, the text encoding is determined based on the underlying features; that is, the underlying features at the bottom of the current feature tree have been integrated into the top layer, and the text encoding can be determined using the underlying features.

[0167] If the number of underlying features is 1 and there are upper-level features, the underlying features and the upper-level features are aggregated in a dual-feature aggregation manner to generate corresponding aggregated features and update the current feature tree, and the step of obtaining the lowest underlying feature in the current feature tree is executed to continue the dual-feature aggregation; that is, when there is only one underlying feature in the current feature tree and the underlying feature is cascaded with the upper-level feature of the previous layer, the underlying feature can be dual-feature aggregated with the upper-level feature to obtain the corresponding fusion feature and update the current feature tree. As shown in Figure 7, the two features directly cascaded can be directly dual-feature aggregated.

[0168] If the number of underlying features is greater than 1, any two underlying features are aggregated in a dual-feature aggregation manner to generate the corresponding aggregated features and update the current feature tree. The step of obtaining the lowest underlying feature in the current feature tree is then executed to continue the dual-feature aggregation. That is, when there are multiple underlying features in the current feature tree, any two underlying features can be selected for dual-feature aggregation to obtain the corresponding fused features and update the current feature tree. As shown in Figure 8, the principle of aggregating component features first and then step text features can be followed to select the corresponding two underlying features for dual-feature aggregation to obtain the corresponding fused features and update the current feature tree.

[0169] Specifically, for the specific aggregation process of the step text features and component features in the feature tree in some embodiments of the present application, the designer can set it according to the practical scenario and user needs. For example, the processor can use the tree-shaped long short-term memory network (Tree LSTM) to generate the text encoding corresponding to the procedural text data based on the feature tree. For example, the tree-shaped long short-term memory network in some embodiments of the present application can adopt the structure shown in Figure 9, which includes 3 nodes: node1, node2 and node3. For all nodes, including implicit states (h1, h2 or h3) and node cell states (c1, c2 or c3); the calculation method of the tree-shaped long short-term memory network can be as follows: f jk =σ(U f ·h k +b f ) (1)

[0170] In formula (1), f jk It can be the forgetting coefficient, which can represent f2 and f3 in Figure 9. In this embodiment, j = 1, k = 2 or 3, and the value of the forgetting coefficient depends on the current node input and the implicit state output h of the child node. k .

[0171] In formula (2), the implicit state h of all child nodes can be obtainedk And sum them to get

[0172] In formula (3), i j It can be the input gate weight value, which controls the weight coefficient of the current j node input value.

[0173] In formula (4) It can be an alternative cell state, which represents the input value size of the current j node.

[0174] Formula (5) can be expressed as the state c of the root node j j Updates.

[0175] Finally, the hidden state output of the root node j is h j Calculated by the following formula: h j =o j tanh(c j ) (7)

[0176] Among them, j The W, U, and b matrices in formulas (1)-(7) can all be training parameters and can be random values ​​during initialization; that is, W, U, and b can be preset matrices.

[0177] That is to say, the input of the tree-shaped long short-term memory network shown in Figure 9 can be the features of the node inputs of node2 and node3, such as component features, aggregation features or step text features; the two features are input to the state input terminals C_k (such as c2 or c3) of node2 and node3, and the h_j (such as h2 or h3) of the initial node is input as a randomly initialized vector, and the vector dimension is consistent with C_k; the output of the tree-shaped long short-term memory network can be C_j (such as c1).

[0178] Accordingly, in some embodiments of the present application, the processor can aggregate the step text features and component features in a double-feature aggregation manner of a tree-shaped long short-term memory network in accordance with the structure of the feature tree, from bottom to top, to generate text encoding. That is, the processor can aggregate the above-mentioned bottom-level features and upper-level features or two bottom-level features in a double-feature aggregation manner of a tree-shaped long short-term memory network, that is, the process of aggregating the bottom-level features and upper-level features in a double-feature aggregation manner to generate corresponding aggregated features and update the current feature tree can include: Aggregate the bottom layer features with the upper layer features to generate the corresponding aggregate features; where c j is the aggregation feature, h j =o j tanh(c j ), fj k= σ(U f ·h k +b f ), j = 1, k = 2 or 3, c k are the bottom layer features and the upper layer features, W, U and b are preset matrices, h j c j The corresponding hidden state; h k c k The corresponding hidden state, c k When it is a step text feature or a component feature, h k is a randomly initialized vector.

[0179] That is to say, in some embodiments of the present application, the processor can use the feature encoding output of the tree-shaped long short-term memory network corresponding to the top-level root node as the output of the feature in the entire feature tree, that is, the text encoding; e rec You can encode text, It can be the implicit state of the root node corresponding to the top layer of the feature tree, that is, the text encoding is determined based on the underlying features, and the implicit state corresponding to the underlying features can be determined as the text encoding.

[0180] It should be noted that the encoding of procedural text data in some embodiments of the present application can enable the encoding results (i.e., text encoding) to include the dependencies between steps and steps and between steps and components in the procedural text, thereby improving the encoding accuracy of the procedural text, thereby enabling the subsequent use of text encoding for other processing, such as matching retrieval of step diagram data or training of retrieval matching models.

[0181] In some embodiments of the present application, the embodiments of the present application generate a feature tree corresponding to the procedural text data based on the component features of each component text, the step text features of each step text, and the recombinant sequence, establish a tree relationship with the structure of the step information and component information in the procedural text, and use the tree relationship to reflect the mutual dependence between the steps and the components; by generating the text encoding corresponding to the procedural text data based on the feature tree, the encoding result can include the dependency between the steps and components in the procedural text, thereby improving the accuracy of the encoding result.

[0182] Based on the above embodiments, some embodiments of the present application also provide a model training method for image-text mutual inspection, which uses the text encoding corresponding to the procedural text data obtained through the above embodiments to complete the training of the retrieval matching model used for the image-text mutual inspection of the procedural text and the step diagram. Specifically, please refer to Figure 10, which is a flow chart of a model training method for image-text mutual inspection provided by some embodiments of the present application. The method may include:

[0183] Step 201: Acquire training data; wherein the training data includes a preset number of training pairs of procedural text data and step diagram data.

[0184] It is understood that the training data in some embodiments of the present application may be data required for training a preset matching model. In some embodiments of the present application, the training data may include a preset training number of pairs of procedural text data and step diagram data, that is, the training data may include paired procedural text data and step diagram data, and the number of procedural text data and step diagram data is the preset training number.

[0185] Correspondingly, the step diagram data in the training data in some embodiments of the present application can be the step diagram data corresponding to the procedural text data, such as the image data corresponding to each step text in the procedural text data, such as the photo images taken during the production of steps 1-6 in the above-mentioned paper cup flower production step text.

[0186] Step 202: Obtain the text code of each program text data.

[0187] Among them, the text encoding is an encoding generated by using the feature tree corresponding to the procedural text data. The feature tree includes a chain connection relationship tree, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features. The step text features are the features of the step text in the procedural text data, and the component features are the features of the component text in the procedural text data. The chain connection relationship tree is a tree corresponding to the recombinant sequence obtained by fusing the step texts in the order of the step texts.

[0188] It should be noted that, in some embodiments of the present application, the processor may adopt the same or similar method as the text encoding method provided in the above embodiments to obtain the text encoding of each procedural text data in the training data; for example, in some embodiments of the present application, the processor may obtain the current procedural text data; wherein, the current procedural text data is any procedural text data in the training data, and the current procedural text data includes step text and component text; according to the order of the step text, the step texts in the current procedural text data are fused to generate a recombined sequence; according to the component features of each component text in the current procedural text data, the step text features of each step text in the current procedural text data and the recombined sequence, a corresponding feature tree in the current procedural text data is generated; wherein, the feature tree includes: a chain connection relationship tree corresponding to the recombined sequence, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features; according to the feature tree, the corresponding text encoding in the current procedural text data is generated.

[0189] Step 203: Obtain the image code of each step image data.

[0190] Among them, the step diagram data in some embodiments of the present application can be the step diagram data corresponding to the procedural text data, such as the image data corresponding to each step text in the procedural text data, such as the photo image corresponding to each step text.

[0191] It is understood that in some embodiments of the present application, the procedural text data and step diagram data in the training data can be paired training data for training a preset matching model, that is, the training data for training the preset matching model can include a preset plurality of pairs (i.e., a preset number of training pairs) of procedural text data and step diagram data. In some embodiments of the present application, the image encoding can be the encoding result (i.e., image encoding) obtained by encoding the step diagram data.

[0192] Specifically, the specific manner in which the processor obtains the image encoding of the step diagram data corresponding to the procedural text data in this step can be set by the designer. For example, the processor can obtain the image encoding of the step diagram data based on a heterogeneous graph.

[0193] As shown in FIG11 , this step may include:

[0194] Step 2031: Extract the step diagram features of each step diagram in the current step diagram data.

[0195] Wherein, the current step graph data can be any step graph data in the training data. In this step, the processor can extract the features (i.e., step graph features) corresponding to each step graph in the current step graph data. For some embodiments of the present application, the specific method for the processor to extract the step graph features of each step graph in the current step graph data can be set by the designer. For example, the processor can use the backbone network (backbone) of the deep residual network (ResNet, Residual Network) to extract the step graph features corresponding to each step graph, for example, the features of the ResNet backbone network in the previous layer of the classification layer (such as the pooling layer avg pool layer (average pooling layer)) are used as step graph features, that is, in this step, the processor can input the current step graph into the preset residual network, obtain the feature vector output by the pooling layer of the preset residual network, and determine the feature vector as the step graph feature corresponding to the current step graph; wherein, the current step graph is any step graph in the current step graph data. The processor can also extract the step graph features of each step graph in the step graph data in other ways, and some embodiments of the present application do not impose any restrictions on this.

[0196] Specifically, the structure of the ResNet backbone network (i.e., the preset residual network) can be shown in Figure 12, including seven parts connected in sequence. The first part may not include a residual block, and mainly performs convolution, regularization, activation function and maximum pooling calculations on the input. The second, third, fourth and fifth parts can all include residual blocks, and each residual block includes three layers of convolution. After the convolution calculations of the first five parts, the pooling layer of the sixth part converts the input into a feature vector, and finally the classifier of the seventh part calculates this feature vector and outputs the category probability (output); in some embodiments of the present application, the processor can use the feature vector output by the pooling layer of the sixth part of each step graph as the step graph feature of each step graph.

[0197] Step 2032: Generate a heterogeneous graph corresponding to the current step graph data based on the category labels and step graph features corresponding to each step graph; wherein the heterogeneous graph includes component nodes corresponding to the category labels, step graph nodes corresponding to the step graph features, and connection edges between each step graph node and its corresponding component nodes.

[0198] It can be understood that the category label corresponding to each step diagram can be a category label obtained by classifying the step diagram according to the ingredients in the step diagram. As shown in Figure 13, the step diagram in the step diagram data (images 1-Q taken during the cooking process) can be classified through a classification network (lassification network) to obtain category labels (labels 1-M, i.e. ing1-ingM); for example, the recipe step diagram can obtain a certain number of category labels and ingredients through image classification; for example, the recipe step diagram of scrambled eggs with tomatoes can include category labels such as tomatoes, eggs, and oil.

[0199] Specifically, in this step, the processor can generate a heterogeneous graph corresponding to the step graph data based on the category labels and step graph features corresponding to each step graph in any step graph data. As shown in Figure 14, the heterogeneous graph can include nodes and connections; nodes can include component nodes corresponding to category labels. (ingredient node) and the step graph node corresponding to the step graph feature (instruction node), Step image features that can represent cooking step images φ() represents the ResNet backbone network; Component Node The classification labels for the step diagram can come from the image classification network, such as the category labels (mango, water and sugar) obtained by classifying the step diagram by the ingredients (i.e., ingredients) of the dish in Figure 14; in some embodiments of the present application, the category features corresponding to the category labels can be obtained and the category features can be used as the corresponding ingredient nodes For example, the category label is input into a neural network encoder (such as a Transformer encoder, Transformer layers, such as the BERT network model (Bidirectional Encoder Representations from Transformers, a language representation model)), and the encoding feature corresponding to each category label (i.e., category feature) is obtained. The encoding feature can be used as the component node corresponding to the category label The relationship can be a connecting edge between each step graph node and its corresponding component node, that is, the relationship can be established through a classification network. If the classification label of a step graph includes the classification label corresponding to a component node, then a connecting edge is established between the step graph node corresponding to the step graph and the component node. As shown in Figure 14, mango appears in all step graphs, so all step graph nodes establish connecting edges with the component node of mango; the component node of sugar establishes a connecting edge e43 with the step graph node of the third step graph.

[0200] Step 2033: Generate image code corresponding to the current step graph data according to the heterogeneous graph.

[0201] In this step, the processor may use the heterogeneous graph to generate image codes corresponding to the step graph data.

[0202] Specifically, the specific method by which the processor generates the image encoding corresponding to the step graph data based on the heterogeneous graph in this step can be set by the designer. For example, the processor can use a graph attention network to aggregate the nodes in the heterogeneous graph to generate the image encoding corresponding to the step graph data. For example, in some embodiments of the present application, a graph attention network can be used to implement feature aggregation and updating, and the update method is to traverse each heterogeneous node in turn for updating.

[0203] In formula (8), Wd, We and Wf are dimensional matrix, It can represent matrix multiplication and vector mapping.

[0204] As shown in formula (8), first update the step graph nodes, can represent the mth step graph node, It can represent the nth component node. If there is a connection edge between the mth step graph node and the nth component node of the component node, the mth step graph node can be updated with the nth component node, but a weight needs to be calculated. Formula (8) is the weight calculation, where z mn It can represent the relative weight of the mth step graph node and the nth component node.

[0205] For each step graph node, such as Traverse all component nodes with connected edges, assuming there are N N Each of them can be obtained through formula (8) with its corresponding correlation weight z mn .

[0206] The weights of all component nodes connected to the step graph nodes are normalized using the following formula (9):

[0207] The normalized correlation weight α is obtained mn , exp represents the exponential function. It represents the sum of the relative weights of all component nodes connected to the step graph node.

[0208] Finally, the node features of the step nodes are updated through the normalized related weights. The formula is as follows:

[0209] Where σ represents a hyperparameter, which is in the range [0,1]. v yes dimensional matrix. It is a new step graph node after being updated by the component node connected to it.

[0210] Furthermore, some implementations of this application may introduce the idea of ​​residual network to update Compared with the initial features before updating Add. The formula is as follows:

[0211] Similarly, we perform the same calculation and update on the component nodes. The formula is as follows:

[0212] Among them, N M It can represent that a total of M step graph nodes are connected to the component node.

[0213] After traversing all component nodes and step graph nodes, the network update of the first layer of the graph attention network is completed.

[0214] In some embodiments of the present application, T layers of graph attention networks can be superimposed, and the update method of the node features of each layer is as above, with t representing the graph attention network of the tth layer. In some embodiments of the present application, an integrated fully connected layer can be added after each layer of graph attention network to realize the re-encoding of node features (including component nodes and step graph nodes), such as through and Re-encode the node features; FFN can represent the fully connected layer, and It can represent the initialization node features of the graph attention network at layer t+1.

[0215] As described above, the updating of nodes in the heterogeneous graph is completed. In some embodiments of the present application, since the step graph nodes integrate the information of the component nodes, the component nodes are updated through the graph neural network, and the characteristics of the relevant step graph nodes are emphasized in the form of keywords, thereby facilitating the subsequent retrieval between the procedural text data and the step graph data.

[0216] Next, in some embodiments of the present application, the step graph nodes in the heterogeneous graph can be input into a long short-term memory network (LSTM network) to obtain the overall characteristics of the entire step graph nodes. The formula is as follows:

[0217] Among them, LSTM() can represent each unit of the LSTM network. It can represent the output of the mth LSTM unit. Represents the step graph node, which comes from the heterogeneous graph node features of the last layer, and m represents the mth step graph.

[0218] Finally, in some embodiments of the present application, the feature encoding output of the last LSTM unit can be used as the feature output of the entire step graph data; that is, Among them, e csi The image corresponding to the step diagram data can be encoded for subsequent retrieval.

[0219] Step 204: Based on the text encoding and the image encoding, the preset matching model is trained using a preset loss function to obtain a trained retrieval matching model, so as to use the retrieval matching model to match the procedural text data to be retrieved or the step diagram data to be retrieved.

[0220] It can be understood that the preset matching model in some embodiments of the present application can be a pre-set network model for retrieving and matching sample step diagram data corresponding to the procedural text data to be retrieved and / or retrieving and matching sample procedural text data corresponding to the step diagram data, that is, the preset matching model can include a text feature encoding network (such as the network on the left side of Figure 13) and / or an image feature encoding network (such as the network on the right side of Figure 13) to implement the above-mentioned procedural text data encoding and / or step diagram data encoding process, that is, the processor can use the preset matching model to implement the process of step 202 and step 203. In this embodiment, the processor can use training data including multiple pairs of procedural text data and step diagram data, and train the preset matching model using a preset loss function according to the text encoding corresponding to the multiple pairs of procedural text data and the image encoding corresponding to the step diagram data, to obtain a trained preset matching model (i.e., a retrieval matching model).

[0221] Specifically, the specific setting of the preset loss function in this step can be set by the designer according to the usage scenario and user needs. For example, the preset loss function includes a ternary loss function (such as Triplet Loss in Figure 13) or a ternary loss function and a tree structure loss function. For example, the preset loss function can be the sum of the ternary loss function and the tree structure loss function; wherein the ternary loss function can be For any image code among the N image codes in the current batch (i.e. image code), For the current batch Paired text encoding, For the current batch Unpaired text encoding, To preset hyperparameters, For any text encoding in the current batch of N image encodings, For the current batch Paired image encoding, For the current batch Unpaired image encoding.

[0222] For example, when the preset loss function includes a ternary loss function, for training data consisting of paired procedural text data and step diagram data, each image encoding and text encoding can be traversed to calculate the average value of the loss function using the following formula:

[0223] That is to say, in some embodiments of the present application, N pairs of training data can be selected for each batch; the image codes are traversed (a total of N), and each image code selected by the traversal can be a can represent anchor (anchor sample). The text encoding paired with the anchor sample is recorded as That is, match the correct text encoding. Similarly, in this batch Unpaired text features are recorded as That is, matching the wrong text encoding. is a pre-set hyperparameter, which can be fixed at 0.3. Similarly, similar traversal operations are performed for text encoding. Represents the text code selected in the traversal, and its corresponding image code is recorded as The non-corresponding

[0224] Correspondingly, when the preset loss function includes a ternary loss function, step 202 may include: initializing the weights of all network layers in the preset matching model to obtain the initialized preset matching model; inputting the training data of the current batch into the preset matching model respectively to obtain the training matching results output by the preset matching model; wherein the training data includes N pairs of procedural text data and step diagram data of the current batch; judging whether the preset number of iterations is reached or the error between the training matching result and the preset matching result is less than the convergence threshold; if the preset number of iterations is reached or the error between the training matching result and the preset matching result is less than the convergence threshold, the preset matching model is used as the retrieval matching result of the training. model; if the preset number of iterations is not reached and the error between the training matching result and the preset matching result is not less than the convergence threshold, the preset loss function is used to obtain the loss value corresponding to the training matching result; according to the loss value, the back propagation error of each network layer in the preset matching model (such as the neural network layer, the fully connected layer and the convolution layer, etc.) is obtained; according to the back propagation error, the weights of each network layer in the preset matching model are updated to obtain the updated preset matching model; the next batch of training data is selected, and the next batch of training data is used as the training data of the current batch, and the training data of the current batch is input into the preset matching model respectively to obtain the training matching result output by the preset matching model.

[0225] That is to say, when the preset loss function includes a ternary loss function, the training process of the preset matching model (such as a convolutional neural network) is divided into two stages. The first stage is the stage in which data is propagated from a low level to a high level, that is, the forward propagation stage. The other stage is when the result obtained by the forward propagation does not meet expectations, the error is propagated from a high level to a low level for training, that is, the back propagation stage. Specifically, the training process can be: 1. Initialize the weights of all network layers of the preset matching model, generally using random initialization; 2. Input image and text data are forward propagated through each layer such as the graph neural network, convolution layer, downsampling layer, and fully connected layer to obtain the output value; 3. Calculate the output value of the network, and calculate the ternary loss function value of the network output value according to the ternary loss function formula. 4. The error Back propagates back to the network, and obtains the back propagation error of each layer of the network in turn: graph neural network layer, fully connected layer, convolutional layer, etc. 5. Each layer of the network adjusts all weight coefficients in the network according to the back propagation error of each layer, that is, updates the weights. 6. Randomly select a new batch of image text data again, and then enter step 2 to obtain the network forward propagation to get the output value. 7. Infinite reciprocating iterations, when the error between the output value of the network and the target value (that is, the preset matching result) is less than a certain threshold (that is, the convergence threshold), or the number of iterations exceeds a certain threshold, the training ends. 8. After the training is completed, save the trained network parameters of all layers to obtain the retrieval matching model.

[0226] It should be noted that when the preset loss function includes a tree structure loss function, some embodiments of the present application can utilize the tree structure loss function to ensure structural similarity between the feature tree generated by the procedural text data and the preset target tree. Accordingly, in step 202, the processor can obtain the preset target tree corresponding to the procedural text data; based on the text encoding, image encoding, feature tree, and preset target tree, the preset matching model is trained using the preset loss function to obtain a trained retrieval matching model.

[0227] Among them, the above-mentioned training of the preset matching model using the preset loss function according to the text encoding, image encoding, feature tree and preset target tree, and obtaining the trained retrieval matching model may include: the output process of the tree structure loss function; when the tree structure loss function may include local structure comparison loss, global structure comparison loss and node degree loss: the output process of the tree structure loss function may include the output process of local structure comparison loss, global structure comparison loss and node degree loss.

[0228] Specifically, the output process of the local structure comparison loss can be shown in Figure 15, including:

[0229] Step 301: Construct a first adjacency matrix corresponding to each node based on the node attributes and neighbor nodes of each node in the current feature tree and the current preset target tree; wherein the current feature tree is the feature tree of the current procedural text data, the current procedural text data is any procedural text data used for training, and the current preset target tree is the preset target tree corresponding to the current procedural text data.

[0230] It can be understood that for each node in the current feature tree and the current preset target tree, some embodiments of the present application can use the neighboring nodes and their own attributes to construct a subgraph and represent it as an adjacency matrix, namely the first adjacency matrix A; thereby being able to calculate the local structural similarity between the current feature tree and the current preset target tree by identifying whether the subgraphs are isomorphic.

[0231] Specifically, in some embodiments of the present application, the subgraph structure is represented by a first adjacency matrix. Assuming there are r nodes in total, the first adjacency matrix can be an r×r matrix, where r is the number of nodes in the subgraph. uv Indicates whether there is an edge connection between node u and node v. If there is an edge connection, then A uv =1, otherwise A uv =0.

[0232] Step 302: Generate a first embedding vector matrix corresponding to the current feature tree and a second embedding vector matrix corresponding to the current preset target tree according to the first adjacency matrix.

[0233] In this step, the processor can use the first adjacency matrix to represent each subgraph as a vector, namely the subgraph embedding vector, and form a matrix of all subgraph embedding vectors in the current feature tree and the current preset target tree, namely the first embedding vector matrix and the second embedding vector matrix.

[0234] Accordingly, in this step, the processor can calculate the Laplace matrix corresponding to each first adjacency matrix; calculate the eigenvector of each Laplace matrix; obtain the vector representation corresponding to each first adjacency matrix based on the eigenvector of each Laplace matrix; wherein the vector representation corresponding to the current first adjacency matrix includes the first preset number of eigenvectors of the eigenvector of the Laplace matrix corresponding to the current first adjacency matrix, sorted from small to large according to the eigenvalue, and the current first adjacency matrix is ​​any first adjacency matrix; based on the vector representation of each first adjacency matrix, generate a first embedded vector matrix and a second embedded vector matrix.

[0235] It can be understood that there are three types of Laplacian matrices corresponding to the first adjacency matrix A. For example, the Laplacian matrix can use the degree matrix D corresponding to the adjacency matrix A, or directly use the adjacency matrix A, or use the difference L = DA between the degree matrix and the adjacency matrix; where the degree matrix D is a diagonal matrix, and its element D uu Represents the degree of node u.

[0236] Correspondingly, the process of calculating the eigenvectors of each Laplacian matrix can be performed by performing eigenvalue decomposition on the Laplacian matrix to obtain the eigenvectors of the Laplacian matrix. Eigenvectors can be used to describe the structural information of the subgraph. Each eigenvector corresponds to an eigenvalue. The smaller the eigenvalue, the more important the structural information described by the corresponding eigenvector.

[0237] Accordingly, the processor can select the first k eigenvectors sorted from small to large in the Laplacian matrix as the vector representation of the subgraph (i.e., the first adjacency matrix) according to the size of the eigenvalue. The matrix composed of these eigenvectors can be the vector representation matrix of the subgraph, which can represent the structural information of the subgraph, i.e., the subgraph embedding vector; all subgraph embedding vectors corresponding to the current feature tree and the current preset target tree can be respectively represented by matrix X1 (i.e., the first embedding vector matrix) and matrix X2 (i.e., the second embedding vector matrix). The first embedding vector matrix and the second embedding vector matrix.

[0238] Step 303: Calculate the similarity between the first embedding vector matrix and the second embedding vector matrix to obtain a local structure similarity matrix.

[0239] It can be understood that in this step, the processor can calculate the similarity between the first embedding vector matrix and the second embedding vector matrix (such as cosine similarity or Euclidean distance) to obtain a matrix, namely the local structure similarity matrix; for example, in this step, the processor can calculate the cosine similarity between the first embedding vector matrix and the second embedding vector matrix to obtain the local structure similarity matrix.

[0240] Step 304: Obtain the local structure comparison loss corresponding to the current procedural text data according to the local structure similarity matrix.

[0241] It is understandable that in this step, the processor can use the local structure similarity matrix to calculate the local structure comparison loss corresponding to the current program text data. The specific method for the processor to obtain the local structure comparison loss corresponding to the current program text data based on the local structure similarity matrix can be set by the designer. For example, the processor can use L1=||H|| 2 , the local structure comparison loss is calculated; L1 is the local structure comparison loss, H is the local structure similarity matrix, ||·|| 2It can represent the matrix two norm.

[0242] It should be noted that the output process of the above-mentioned global structure comparison loss may include: constructing a second adjacency matrix corresponding to the current feature tree and the current preset target tree according to the node attributes and neighbor nodes of each node in the current feature tree and the current preset target tree; generating a first embedding vector corresponding to the current feature tree and a second embedding vector corresponding to the current preset target tree according to the second adjacency matrix; calculating the similarity between the first embedding vector and the second embedding vector to obtain the global structure similarity; and obtaining the global structure comparison loss corresponding to the current procedural text data according to the global structure similarity.

[0243] That is to say, the processor can use a method similar to the output process of the above-mentioned local structure comparison loss to map the current feature tree and the current preset target tree to the same vector space, and obtain the first embedding vector corresponding to the current feature tree and the second embedding vector corresponding to the current preset target tree; calculate the similarity between the first embedding vector and the second embedding vector to obtain the global structure similarity; and obtain the global structure comparison loss corresponding to the current procedural text data based on the global structure similarity.

[0244] For example, the processor can use the node attributes and neighbor nodes of each node in the current feature tree and the current preset target tree to construct an adjacency matrix (i.e., a second adjacency matrix) corresponding to each of the current feature tree and the current preset target tree; calculate the Laplace matrix corresponding to each of the two second adjacency matrices; calculate the eigenvectors of each Laplace matrix; obtain the vector representation corresponding to each of the two second adjacency matrices based on the eigenvectors of each of the two Laplace matrices; wherein the vector representation corresponding to the current second adjacency matrix includes the first preset number of eigenvectors in the eigenvectors of the Laplace matrix corresponding to the current second adjacency matrix, sorted from small to large according to the eigenvalue, and the current second adjacency matrix is ​​any second adjacency matrix; based on the vector representation of each second adjacency matrix, generate a first embedding vector and a second embedding vector, such as using the matrix composed of the eigenvectors in the vector representation of the second adjacency matrix corresponding to the current feature tree as the embedding vector corresponding to the current feature tree (i.e., the first embedding vector). After the processor uses the above-mentioned graph embedding algorithm to map the current feature tree and the current preset target tree to the same vector space and obtains two embedding vectors (i.e., the first embedding vector and the second embedding vector), it can calculate the similarity between the first embedding vector and the second embedding vector (such as cosine similarity or Euclidean distance) to obtain a global structural similarity; then the global structural similarity and the global weight coefficient can be calculated to obtain the global structural comparison loss corresponding to the current procedural text data; for example, the global structural comparison loss L2 is calculated by L2=w1*sim(X3,X4), w1 is the global weight coefficient, X3 and X4 are the first embedding vector and the second embedding vector respectively, and sim() can represent the cosine similarity.

[0245] Accordingly, the process of outputting the node degree loss can include: calculating the degree of each node in the current feature tree and the current preset target tree; calculating the degree difference between all nodes in the current feature tree and the current preset target tree to obtain a degree difference matrix; and obtaining the node degree loss corresponding to the current procedural text data based on the degree difference matrix. Node degree reflects the number of connections between a node and other nodes. In this embodiment, node degree is incorporated into the loss function to strengthen the constraints on node connectivity.

[0246] For example, the processor may first calculate the degree of each node in the current feature tree and the current preset target tree; then calculate the difference between the degrees of all nodes in the current feature tree and the current preset target tree to obtain a degree difference matrix; finally, the degree difference matrix may be obtained by L3=||D|| 2 , calculate the second norm of the degree difference matrix D and obtain the node degree loss L3.

[0247] Correspondingly, after obtaining the local structure comparison loss, global structure comparison loss, and node degree loss corresponding to the current procedural text data, the local structure comparison loss, global structure comparison loss, and node degree loss corresponding to the current procedural text data can be added together to obtain the tree structure loss function corresponding to the current procedural text data, that is, the tree structure loss function L_structure = L1 + L2 + L3. In some embodiments of the present application, during the training process, the tree structure loss function can be optimized to make the preset target tree and the generated feature tree as similar as possible; for example, an optimization algorithm such as gradient descent can be used to solve the optimal parameters, thereby obtaining a model that minimizes the loss function.

[0248] Furthermore, after step 204, the processor can also obtain data to be retrieved; wherein, the data to be retrieved is procedural text data to be retrieved or step diagram data to be retrieved; using the retrieval matching model, the data encoding of the data to be retrieved is extracted, and the data encoding is distance matched with the sample encoding of each sample data in the sample data set to obtain matching sample data corresponding to the data to be retrieved; wherein, the sample data set is a sample step diagram data set or a sample procedural text data set.

[0249] For example, when the data to be retrieved is procedural text data to be retrieved, the processor can retrieve the matching model, extract the text encoding of the procedural text data to be retrieved, and perform distance matching on the text encoding with the step diagram encoding of each sample step diagram data in the sample step diagram data set to obtain matching sample data corresponding to the procedural text data to be retrieved, such as determining the sample step diagram data corresponding to the step diagram encoding that is closest to the text encoding as the matching sample data.

[0250] In some embodiments of the present application, a preset matching model is trained using a preset loss function based on text encoding and image encoding to obtain a trained retrieval matching model. The text encoding can be obtained by using procedural text data encoding and the image encoding can be obtained by using step diagram data encoding. The preset loss function is used to train the preset matching model, thereby realizing the training of the retrieval matching model and improving the retrieval accuracy of the retrieval matching model.

[0251] Based on some of the above embodiments, this application also provides a model matching method for image-text mutual verification, which uses the retrieval matching model trained through the above embodiments to complete the retrieval matching of the image-text mutual verification of the procedural text and the step diagram. Specifically, please refer to Figure 16, which is a flow chart of a model matching method for image-text mutual verification provided by some embodiments of this application. The method may include:

[0252] Step 401: Acquire data to be retrieved; wherein the data to be retrieved is procedural text data to be retrieved or step diagram data to be retrieved.

[0253] It is understandable that the data to be retrieved in some embodiments of the present application may be data that requires text-graphic mutual verification, that is, data that requires retrieval and matching to corresponding step diagram data or procedural text data. Some embodiments of the present application do not limit the specific data type of the data to be retrieved, such as the data to be retrieved may be procedural text data (i.e., procedural text data to be retrieved), so as to utilize the retrieval matching model to retrieve the step diagram data (i.e., sample step diagram data) corresponding to the procedural text data to be retrieved in the sample step diagram data set (i.e., sample data set), that is; the data to be retrieved may also be step diagram data (i.e., step diagram data to be retrieved), so as to utilize the retrieval matching model to retrieve the procedural text data (i.e., sample procedural text data) corresponding to the step diagram data to be retrieved in the sample procedural text data set (i.e., sample data set).

[0254] Step 402: Using the retrieval matching model, extract the data code of the data to be retrieved, and perform distance matching on the data code with the sample code of each sample data in the sample data set to obtain matching sample data corresponding to the data to be retrieved; wherein the sample data set is a sample step diagram data set or a sample procedural text data set.

[0255] It can be understood that if the data to be retrieved is the procedural text data to be retrieved, the data encoding is a text encoding generated using the feature tree corresponding to the procedural text data to be retrieved. The feature tree includes a chain connection relationship tree, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features. The step text features are the features of the step text in the procedural text data to be retrieved, and the component features are the features of the component text in the procedural text data to be retrieved. The chain connection relationship tree is a tree corresponding to the recombinant sequence obtained by fusing the step texts in the order of the step texts.

[0256] That is to say, when the data to be retrieved is procedural text data (i.e., procedural text data to be retrieved), in some embodiments of the present application, the processor can use the retrieval matching model to extract the text encoding (i.e., data encoding) of the procedural text data to be retrieved in the same or similar manner as the text encoding method provided in some of the above embodiments; for example, in some embodiments of the present application, the processor can input the procedural text data to be retrieved into the retrieval matching model; wherein, the procedural text data to be retrieved includes step text and component text; using the retrieval matching model, according to the order of the step text, the step texts in the procedural text data to be retrieved are fused to generate a recombined sequence; according to the component features of each component text in the procedural text data to be retrieved, the step text features of each step text in the procedural text data to be retrieved and the recombined sequence, a corresponding feature tree in the procedural text data to be retrieved is generated; wherein, the feature tree includes: a chain connection relationship tree corresponding to the recombined sequence, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features; according to the feature tree, the corresponding text encoding in the procedural text data to be retrieved is generated.

[0257] It should be noted that if the data to be retrieved is procedural text data to be retrieved, the data encoding of the data to be retrieved can be a text encoding, such as the text encoding corresponding to the procedural text data to be retrieved obtained by using the text encoding method provided in some of the above embodiments; the sample data set can be a sample step diagram data set, and the sample data in the sample data set can be step diagram data (i.e., sample step diagram data), and the sample encoding can be an image encoding, such as the image encoding corresponding to each sample procedural text data obtained by using the encoding method of the step diagram data provided in some of the above embodiments.

[0258] If the data to be retrieved is step diagram data to be retrieved, the data encoding of the data to be retrieved can be an image encoding, such as the image encoding corresponding to the step diagram data to be retrieved obtained by adopting the encoding method of the step diagram data provided in the above embodiment; the sample data set can be a sample procedural text data set, and the sample data in the sample data set can be procedural text data (i.e., sample procedural text data), and the sample encoding can be a text encoding, such as the text encoding corresponding to each sample procedural text data obtained by adopting the text encoding method provided in the above embodiment.

[0259] Correspondingly, if the data to be retrieved is the step diagram data to be retrieved, the current sample code is a text code generated using the feature tree corresponding to the current sample procedural text data. The feature tree includes a chain connection relationship tree, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features. The step text features are the features of the step text in the current procedural text data, and the component features are the features of the component text in the current procedural text data. The chain connection relationship tree is a tree corresponding to the recombinant sequence obtained by fusing each step text in the order of the step text; the current sample procedural text data is any sample procedural text data.

[0260] It should be noted that the specific method for the processor to use the retrieval matching model to perform distance matching between the data code and the sample code of each sample data in the sample data set to obtain the matching sample data corresponding to the data to be retrieved can be set by the designer. For example, the processor can use the retrieval matching model to calculate the Euclidean distance between the data code and the sample code of each sample data respectively; and use the sample data corresponding to the smallest Euclidean distance as the matching sample data; if the data to be retrieved is the procedural text data to be retrieved, the processor can calculate the Euclidean distance between the text code of the procedural text data to be retrieved and the image code of each sample step diagram data respectively; and use the sample step diagram data corresponding to the smallest Euclidean distance as the matching sample data corresponding to the procedural text data to be retrieved.

[0261] In some embodiments of the present application, by utilizing a retrieval matching model, the data encoding of the data to be retrieved is extracted, and the data encoding is distance-matched with the sample encoding of each sample data in the sample data set to obtain the matching sample data corresponding to the data to be retrieved. The procedural text data encoding can be used to obtain the text encoding and the step diagram data encoding can be used to obtain the image encoding, thereby ensuring the matching accuracy of the retrieval matching model and improving the accuracy of the image-text mutual inspection.

[0262] Corresponding to the above method embodiments, some embodiments of the present application further provide a text encoding device. The text encoding device described below and the text encoding method described above can refer to each other.

[0263] Please refer to Figure 17, which is a block diagram of a text encoding device provided in some embodiments of the present application. The device may include:

[0264] The text acquisition module 110 is used to acquire procedural text data; wherein the procedural text data includes step text and component text;

[0265] A step fusion module 120 is used to fuse the step texts according to the order of the step texts to generate a recombined sequence;

[0266] The tree generation module 130 is configured to generate a feature tree corresponding to the procedural text data based on the component features of each component text, the step text features of each step text, and the recombined sequence; wherein the feature tree includes: a chain connection relationship tree corresponding to the recombined sequence, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features;

[0267] The text encoding module 140 is used to generate text encoding corresponding to the procedural text data according to the feature tree.

[0268] In some embodiments, the step fusion module 120 may include:

[0269] The text extraction submodule is used to obtain the step text features corresponding to each step text in the procedural text data;

[0270] The feature recombination submodule is used to combine the step text features in the order of the step text to generate a recombined sequence; wherein the number of recombined sequences and step texts is n, the first recombined sequence includes the first step text feature, the sth recombined sequence includes the first step text feature to the sth step text feature, and s is a positive integer greater than 1 and less than or equal to t.

[0271] In some embodiments, the tree generation module 130 may include:

[0272] The sequence extraction submodule is used to obtain the recombinant sequence features corresponding to each recombinant sequence;

[0273] The correlation calculation submodule is used to obtain the correlation coefficient between the text features of each step and the features of each recombinant sequence;

[0274] A spanning tree submodule is used to establish a spanning tree corresponding to the procedural text data based on the association coefficient; wherein the spanning tree includes a chain connection relationship tree and step text features mounted on the chain connection relationship tree, the chain connection relationship tree is a tree of chain-connected recombinant sequences established according to the length of the recombinant sequences, and the root node of the chain connection relationship tree is the longest recombinant sequence;

[0275] The text extraction submodule is used to obtain the component features corresponding to each component text in the procedural text data;

[0276] The mounting submodule is used to mount each component feature to the corresponding step text feature in the generation tree according to the correspondence between each component text and each step text, so as to generate a feature tree corresponding to the procedural text data.

[0277] In some embodiments, the sequence extraction submodule can be specifically used to input each recombinant sequence into a bidirectional long short-term memory network, and obtain the recombinant sequence features corresponding to each recombinant sequence output by the bidirectional long short-term memory network.

[0278] In some embodiments, the association calculation submodule may be specifically configured to calculate the distance between each step text feature and each recombinant sequence feature to obtain an association coefficient.

[0279] In some embodiments, the text encoding module 140 may be specifically configured to aggregate step text features and component features in a bottom-up order according to the structure of the feature tree to generate text encoding.

[0280] In some embodiments, the text encoding module 140 may include:

[0281] The aggregation submodule is used to aggregate the step text features and component features in a dual-feature aggregation manner in a bottom-up order according to the structure of the feature tree to generate text encoding.

[0282] In some embodiments, the aggregation submodule may include:

[0283] The bottom layer acquisition unit is used to obtain the bottom layer features in the bottom layer of the current feature tree; wherein the bottom layer features include step text features, component features and aggregation features;

[0284] a coding determination unit, configured to determine a text coding based on the bottom-layer features if the number of bottom-layer features is 1 and there is no upper-layer feature of the upper cascade layer;

[0285] The first aggregation unit is configured to aggregate the bottom-level features with the upper-level features in a dual-feature aggregation manner if the number of bottom-level features is 1 and the upper-level features exist, generate corresponding aggregated features, update the current feature tree, and execute sending a start signal to the bottom-level acquisition unit;

[0286] The second aggregation unit is used to aggregate any two underlying features in a dual-feature aggregation manner if the number of underlying features is greater than 1, generate corresponding aggregated features, update the current feature tree, and execute sending a start signal to the underlying acquisition unit.

[0287] In some embodiments, the first polymeric unit can be specifically used to Aggregate the bottom layer features with the upper layer features to generate the corresponding aggregate features; where c j is the aggregation feature, h j =o j tanh(c j ), f jk =σ(U f ·h k +b f ), j = 1, k = 2 or 3, c k are the bottom layer features and the upper layer features, W, U and b are preset matrices, h j c j The corresponding hidden state; h k c k The corresponding hidden state, c k When it is a step text feature or a component feature, h k is a randomly initialized vector.

[0288] In some embodiments of the present application, a feature tree corresponding to the procedural text data is generated by the tree generation module 130 based on the component features of each component text, the step text features of each step text, and the recombinant sequence, and a tree relationship is established with the structure of the step information and component information in the procedural text, and the tree relationship is used to reflect the mutual dependence between the steps and the components; the text encoding module 140 generates text encoding corresponding to the procedural text data based on the feature tree, so that the encoding result can include the dependency between the steps and components in the procedural text, thereby improving the accuracy of the encoding result.

[0289] Corresponding to the above method embodiments, some embodiments of the present application also provide a model training device for image-text mutual checking. The model training device for image-text mutual checking described below and the model training method for image-text mutual checking described above can be referred to each other.

[0290] Please refer to Figure 18, which is a structural block diagram of a model training device for image-text mutual verification provided by some embodiments of the present application. The device may include:

[0291] The data acquisition module 210 is used to acquire training data; wherein the training data includes a preset number of training pairs of procedural text data and step diagram data;

[0292] A text encoding module 220 is configured to obtain a text encoding for each procedural text data; wherein the text encoding is an encoding generated using a feature tree corresponding to the procedural text data, wherein the feature tree includes a chain connection relationship tree, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features. The step text features are features of the step texts in the procedural text data, and the component features are features of the component texts in the procedural text data. The chain connection relationship tree is a tree corresponding to a recombined sequence obtained by fusing the step texts in order of the step texts.

[0293] An image coding module 230 is used to obtain image coding of each step diagram data;

[0294] The model training module 240 is used to train the preset matching model based on the text encoding and image encoding using the preset loss function, obtain the trained retrieval matching model, and use the retrieval matching model to match the procedural text data to be retrieved or the step diagram data to be retrieved.

[0295] In some embodiments, the preset loss function includes a ternary loss function or a ternary loss function and a tree structure loss function, and the ternary loss function is Encode any image among the N images in the current batch, For the current batch Paired text encoding, For the current batch Unpaired text encoding, To preset hyperparameters, For any text encoding among the N text encodings in the current batch, For the current batch Paired image encoding, For the current batch Unpaired image encoding; tree structure loss functions include local structure alignment loss, global structure alignment loss and / or node degree loss.

[0296] In some embodiments, when the preset loss function includes a ternary loss function and a tree structure loss function, the model training module 240 may include:

[0297] The target tree acquisition submodule is used to obtain the preset target tree corresponding to the procedural text data;

[0298] The training submodule is used to train the preset matching model using the preset loss function according to the text encoding, image encoding, feature tree and preset target tree to obtain the trained retrieval matching model.

[0299] In some embodiments, the training submodule may include:

[0300] Structural loss unit, used for the output process of tree structure loss function;

[0301] Among them, when the tree structure loss function includes local structure comparison loss, global structure comparison loss and node degree loss, the structure loss unit may include:

[0302] A first adjacency matrix subunit is configured to construct a first adjacency matrix corresponding to each node based on the node attributes and neighbor nodes of each node in the current feature tree and the current preset target tree; wherein the current feature tree is a feature tree of the current procedural text data, the current procedural text data is any procedural text data for training, and the current preset target tree is a preset target tree corresponding to the current procedural text data;

[0303] A vector matrix subunit, configured to generate a first embedding vector matrix corresponding to the current feature tree and a second embedding vector matrix corresponding to the current preset target tree according to the first adjacency matrix;

[0304] A similarity matrix subunit is used to calculate the similarity between the first embedding vector matrix and the second embedding vector matrix to obtain a local structure similarity matrix;

[0305] The local structure subunit is used to obtain the local structure comparison loss corresponding to the current procedural text data according to the local structure similarity matrix;

[0306] A second adjacency matrix subunit is used to construct a second adjacency matrix corresponding to the current feature tree and the current preset target tree according to the node attributes and neighbor nodes of each node in the current feature tree and the current preset target tree;

[0307] A vector subunit, configured to generate a first embedding vector corresponding to the current feature tree and a second embedding vector corresponding to the current preset target tree according to the second adjacency matrix;

[0308] A global similarity subunit, configured to calculate the similarity between the first embedding vector and the second embedding vector to obtain a global structural similarity;

[0309] The global result subunit is used to obtain the global structure comparison loss corresponding to the current procedural text data based on the global structure similarity;

[0310] The degree calculation subunit is used to calculate the degree of each node in the current feature tree and the current preset target tree;

[0311] The difference calculation subunit is used to calculate the degree difference of all nodes in the current feature tree and the current preset target tree to obtain a degree difference matrix;

[0312] The degree loss subunit is used to obtain the node degree loss corresponding to the current procedural text data according to the degree difference matrix;

[0313] The tree structure subunit is used to add the local structure comparison loss, global structure comparison loss and node degree loss corresponding to the current procedural text data to obtain the tree structure loss function corresponding to the current procedural text data.

[0314] In some embodiments, the vector matrix sub-unit can be specifically used to calculate the Laplacian matrix corresponding to each first adjacency matrix; calculate the eigenvector of each Laplacian matrix; obtain the vector representation corresponding to each first adjacency matrix based on the eigenvector of each Laplacian matrix; wherein the vector representation corresponding to the current first adjacency matrix includes the first preset number of eigenvectors of the Laplacian matrix corresponding to the current first adjacency matrix sorted from small to large according to the eigenvalue; generate a first embedded vector matrix and a second embedded vector matrix based on the vector representation corresponding to each first adjacency matrix.

[0315] In some embodiments, the local structure subunit can be specifically used to pass L1=||H|| 2 , the local structure comparison loss is calculated; L1 is the local structure comparison loss, and H is the local structure similarity matrix.

[0316] In some embodiments, when the preset loss function includes a ternary loss function, the model training module 240 may include:

[0317] The initialization submodule is used to initialize the weights of all network layers in the preset matching model and obtain the initialized preset matching model;

[0318] A matching submodule is used to input the current batch of training data into a preset matching model and obtain the training matching results output by the preset matching model; wherein the training data includes N pairs of procedural text data and step diagram data of the current batch;

[0319] A judgment submodule is used to judge whether a preset number of iterations has been reached or whether the error between the training matching result and the preset matching result is less than a convergence threshold;

[0320] A model determination submodule is used to use the preset matching model as the trained retrieval matching model if a preset number of iterations is reached or the error between the training matching result and the preset matching result is less than a convergence threshold;

[0321] A loss acquisition submodule is used to obtain the loss value corresponding to the training matching result using a preset loss function if the preset number of iterations has not been reached and the error between the training matching result and the preset matching result is not less than the convergence threshold;

[0322] The error acquisition submodule is used to obtain the back propagation error of each network layer in the preset matching model based on the loss value;

[0323] An update submodule is used to update the weights of each network layer in the preset matching model according to the back-propagation error to obtain an updated preset matching model;

[0324] The continue training submodule is used to select the next batch of training data and use the next batch of training data as the training data of the current batch, execute the steps of inputting the current batch of training data into the preset matching model respectively, and obtain the training matching results output by the preset matching model.

[0325] In some embodiments, the image encoding module 230 may include:

[0326] The step diagram extraction submodule is used to extract the step diagram features of each step diagram in the current step diagram data; wherein the current step diagram data is any step diagram data;

[0327] A heterogeneous graph generation submodule is used to generate a heterogeneous graph corresponding to the current step graph data based on the category labels and step graph features corresponding to each step graph; wherein the heterogeneous graph includes component nodes corresponding to the category labels, step graph nodes corresponding to the step graph features, and connection edges between each step graph node and its corresponding component nodes;

[0328] The image code generation submodule is used to generate the image code corresponding to the current step graph data based on the heterogeneous graph.

[0329] In some embodiments, the step graph extraction submodule can be specifically used to input the current step graph into a preset residual network, obtain the feature vector output by the pooling layer of the preset residual network, and determine the feature vector as the step graph feature corresponding to the current step graph; wherein the current step graph is any step graph.

[0330] In some embodiments of the present application, the model training module 240 trains the preset matching model according to the text encoding and the image encoding using the preset loss function to obtain a trained retrieval matching model. The text encoding can be obtained by using the procedural text data encoding and the image encoding can be obtained by using the step diagram data encoding. The preset loss function is used to train the preset matching model, thereby realizing the training of the retrieval matching model and improving the retrieval accuracy of the retrieval matching model.

[0331] Corresponding to the above method embodiments, some embodiments of the present application further provide a text encoding device. The model matching device for image-text mutual inspection described below and the model matching method for image-text mutual inspection described above can refer to each other.

[0332] Please refer to Figure 19, which is a structural block diagram of a model matching device for image-text mutual verification provided by some embodiments of the present application. The device may include:

[0333] The retrieval module 310 is used to obtain data to be retrieved; wherein the data to be retrieved is procedural text data to be retrieved or step diagram data to be retrieved;

[0334] The model matching module 320 is used to use the retrieval matching model to extract the data encoding of the data to be retrieved, and perform distance matching on the data encoding with the sample encoding of each sample data in the sample data set to obtain the matching sample data corresponding to the data to be retrieved; wherein, the sample data set is a sample step diagram data set or a sample procedural text data set; if the data to be retrieved is procedural text data to be retrieved, then the data encoding is a text encoding generated using a feature tree corresponding to the procedural text data to be retrieved, and the feature tree includes a chain connection relationship tree, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features, the step text features are features of the step text in the procedural text data to be retrieved, and the component features are features of the component text in the procedural text data to be retrieved, and the chain connection relationship tree is a tree corresponding to the recombinant sequence obtained by fusing each step text in the order of the step text.

[0335] In some embodiments, the model matching module 320 may include:

[0336] The Euclidean distance calculation submodule is used to calculate the Euclidean distance between the data code and the sample code of each sample data;

[0337] The distance matching submodule is used to take the sample data corresponding to the minimum Euclidean distance as the matching sample data.

[0338] In some embodiments of the present application, the retrieval matching model is utilized by the model matching module 320 to extract the data encoding of the data to be retrieved, and the data encoding is distance-matched with the sample encoding of each sample data in the sample data set to obtain the matching sample data corresponding to the data to be retrieved. The text encoding can be obtained by utilizing the procedural text data encoding and the image encoding can be obtained by utilizing the step diagram data encoding, thereby ensuring the matching accuracy of the retrieval matching model and improving the accuracy of the mutual inspection between the image and text.

[0339] Corresponding to the above method embodiments, some embodiments of the present application also provide an electronic device. The electronic device described below and the text encoding method, image-text mutual checking model training method and image-text mutual checking model matching method described above can be referenced to each other.

[0340] Please refer to Figure 20, which is a schematic diagram of the structure of an electronic device provided in some embodiments of the present application. The electronic device may include:

[0341] Memory D1, for storing computer programs;

[0342] Processor D2 is used to implement the steps of the text encoding method, the image-text mutual verification model training method and / or the image-text mutual verification model matching method provided in the above method embodiments when executing the computer program.

[0343] Specifically, please refer to Figure 21, which is a schematic diagram of the specific structure of an electronic device provided by some embodiments of the present application. The electronic device 410 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 422 (for example, one or more processors) and memory 432, one or more storage media 430 (for example, one or more mass storage devices) storing application programs 442 or data 444. Among them, the memory 432 and the storage medium 430 can be temporary storage or permanent storage. The program stored in the storage medium 430 may include one or more units (not shown in the figure), and each unit may include a series of instruction operations on the host. Furthermore, the central processing unit 422 can be configured to communicate with the storage medium 430 to execute a series of instruction operations in the storage medium 430 on the electronic device 410.

[0344] The electronic device 410 may further include one or more power supplies 426, one or more wired or wireless network interfaces 450, one or more input / output interfaces 458, and / or one or more operating systems 441. For example, Windows Server™ (a server operating system), Mac OS X™ (an operating system), Unix™ (an operating system), Linux™ (an operating system), FreeBSD™ (an operating system), etc.

[0345] The steps in the above-described text encoding method, image-text mutual verification model training method, and / or image-text mutual verification model matching method can be implemented by the structure of an electronic device.

[0346] Corresponding to the above method embodiments, some embodiments of the present application also provide a non-volatile readable storage medium. The non-volatile readable storage medium described below and the text encoding method, image-text mutual inspection model training method and image-text mutual inspection model matching method described above can be referenced to each other.

[0347] Please refer to Figure 22, which is a schematic diagram of the structure of a non-volatile readable storage medium provided in some embodiments of the present application. The non-volatile readable storage medium 50 stores a computer program 51, which, when executed by a processor, implements the steps of the text encoding method, the image-text mutual verification model training method, and / or the image-text mutual verification model matching method provided in the above-mentioned method embodiments.

[0348] The non-volatile readable storage medium 50 can specifically be any non-volatile readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0349] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. The devices, electronic devices, and computer-readable storage media disclosed in the embodiments are described briefly because they correspond to the methods disclosed in the embodiments. For relevant details, refer to the description of the methods.

[0350] The above is a detailed introduction to a text encoding method and device, a model training method and device for image-text mutual inspection, a model matching method and device for image-text mutual inspection, an electronic device, and a computer-readable storage medium provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A text encoding method, characterized in that: include: Acquire procedural text data; wherein the procedural text data includes step text and component text; According to the order of the step texts, the step texts are merged to generate a recombinant sequence; According to the component features of each of the component texts, the step text features of each of the step texts and the reorganized sequence, a feature tree corresponding to the procedural text data is generated; wherein the feature tree includes: a chain connection relationship tree corresponding to the reorganized sequence, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features; According to the feature tree, a text code corresponding to the procedural text data is generated.

2. The text encoding method according to claim 1, characterized in that: According to the order of the step texts, the step texts are merged to generate a recombinant sequence, including: Obtaining step text features corresponding to each step text in the procedural text data; The step text features are combined in the order of the step texts to generate the recombinant sequence; wherein the number of the recombinant sequences and the step texts are both t, the first recombinant sequence includes the first step text feature, the sth recombinant sequence includes the first step text feature to the sth step text feature, and s is a positive integer greater than 1 and less than or equal to t.

3. The text encoding method according to claim 1, characterized in that: The step of generating a feature tree corresponding to the procedural text data according to the component features of each of the component texts, the step text features of each of the step texts and the reorganized sequence comprises: Obtaining the recombinant sequence characteristics corresponding to each of the recombinant sequences; Obtaining the correlation coefficient between the text features of each step and the features of each recombinant sequence; According to the association coefficient, a spanning tree corresponding to the procedural text data is established; wherein the spanning tree includes the chain connection relationship tree and the step text features mounted on the chain connection relationship tree, the chain connection relationship tree is a tree of the chain-connected recombinant sequences established according to the length of the recombinant sequences, and the root node of the chain connection relationship tree is the longest recombinant sequence; Obtaining component features corresponding to each component text in the program text data; According to the correspondence between each component text and each step text, each component feature is mounted on the corresponding step text feature in the generation tree to generate a feature tree corresponding to the procedural text data.

4. The text encoding method according to claim 3, characterized in that: The obtaining of the recombinant sequence features corresponding to each of the recombinant sequences comprises: The recombinant sequences are respectively input into a bidirectional long short-term memory network to obtain the recombinant sequence features corresponding to the recombinant sequences output by the bidirectional long short-term memory network.

5. The text encoding method according to claim 3, characterized in that: The obtaining of the correlation coefficient between the text features of each step and the recombinant sequence features comprises: The distance between each of the step text features and each of the recombinant sequence features is calculated to obtain the association coefficient.

6. The text encoding method according to any one of claims 1 to 5, characterized in that: Generating the text encoding corresponding to the procedural text data according to the feature tree includes: According to the structure of the feature tree, the step text features and the result text features are sequentially analyzed from bottom to top. The features are aggregated to generate the text encoding.

7. The text encoding method according to claim 6, characterized in that: The step text features and the component features are aggregated in order from bottom to top according to the structure of the feature tree to generate the text code, including: According to the structure of the feature tree, the step text features and the component features are aggregated in a dual-feature aggregation manner in a bottom-up order to generate the text code.

8. The text encoding method according to claim 7, characterized in that: The step of aggregating the step text features and the component features in a dual-feature aggregation manner in a bottom-up order according to the structure of the feature tree to generate the text code includes: Obtaining the bottom-level features at the bottom of the current feature tree; wherein the bottom-level features include the step text features, the component features, and the aggregation features; If the number of the bottom-level features is 1 and there is no upper-level feature of the upper cascade layer, determining the text encoding according to the bottom-level features; If the number of the bottom-level features is 1 and the upper-level features exist, the bottom-level features are aggregated with the upper-level features in a dual-feature aggregation manner to generate corresponding aggregated features and update the current feature tree, and the step of obtaining the bottom-level features in the current feature tree is performed; If the number of the underlying features is greater than 1, any two of the underlying features are aggregated in a dual-feature aggregation manner to generate corresponding aggregated features and update the current feature tree, and the step of obtaining the lowest underlying feature in the current feature tree is performed.

9. The text encoding method according to claim 8, characterized in that: The step of aggregating the bottom layer features with the upper layer features in a dual feature aggregation manner, generating corresponding aggregated features and updating the current feature tree includes: pass Aggregate the bottom layer features with the upper layer features to generate corresponding aggregate features; where c j is the aggregation feature, h j =o j ·tanh(c j ), f jk =σ(U f ·h k +b f ), j = 1, k = 2 or 3, c k are the bottom layer features and the upper layer features, W, U and b are preset matrices, h j c j The corresponding hidden state; h k c k The corresponding hidden state, c k When h is the step text feature or the component feature, k is a randomly initialized vector.

10. A text encoding device, characterized in that: include: A text acquisition module, used to acquire procedural text data; wherein the procedural text data includes step text and component text; A step fusion module, used to fuse the step texts according to the order of the step texts to generate a recombinant sequence; A tree generation module is used to generate a feature tree corresponding to the procedural text data according to the component features of each of the component texts, the step text features of each of the step texts and the reorganization sequence; wherein the feature tree includes: a chain connection relationship tree corresponding to the reorganization sequence, step text features mounted on the chain connection relationship tree and component features mounted on the step text features; The text encoding module is used to generate the text encoding corresponding to the program text data according to the feature tree.

11. A model training method for image-text mutual inspection, characterized in that: include: Acquire training data; wherein the training data includes a preset number of pairs of procedural text data and step diagram data; Obtaining a text encoding of each of the procedural text data; wherein the text encoding is an encoding generated using a feature tree corresponding to the procedural text data, the feature tree comprising a chain connection relationship tree, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features, the step text features are features of the step text in the procedural text data, the component features are features of the component text in the procedural text data, and the chain connection relationship tree is a tree corresponding to a recombinant sequence obtained by fusing each of the step texts in the order of the step texts; Obtaining image codes of each of the step diagram data; According to the text encoding and the image encoding, a preset matching model is trained using a preset loss function to obtain a trained retrieval matching model, so as to use the retrieval matching model to match the procedural text data to be retrieved or the step diagram data to be retrieved.

12. The model training method for image-text mutual checking according to claim 11, characterized in that: The preset loss function includes a ternary loss function or the ternary loss function and a tree structure loss function, and the ternary loss function is , Encode any image among the N images in the current batch, For the current batch Paired text encoding, For the current batch Unpaired text encoding, To preset hyperparameters, For any text encoding among the N text encodings in the current batch, For the current batch Paired image encoding, For the current batch Unpaired image coding; the tree structure loss function includes local structure alignment loss, global structure alignment loss and / or node degree loss.

13. The model training method for image-text mutual checking according to claim 12 is characterized in that: When the preset loss function includes the ternary loss function and the tree structure loss function, the preset matching model is trained using the preset loss function according to the text encoding and the image encoding to obtain a trained retrieval matching model, including: Obtaining a preset target tree corresponding to the procedural text data; According to the text encoding, the image encoding, the feature tree and the preset target tree, a preset matching model is trained using a preset loss function to obtain a trained retrieval matching model.

14. The model training method for image-text mutual checking according to claim 13, characterized in that: The method of training a preset matching model using a preset loss function according to the text encoding, the image encoding, the feature tree and the preset target tree to obtain a trained retrieval matching model includes: an output process of the tree structure loss function; wherein, when the tree structure loss function includes the local structure comparison loss, the global structure comparison loss and the node degree loss, the output process of the tree structure loss function includes: According to the node attributes and neighbor nodes of each node in the current feature tree and the current preset target tree, a first adjacency matrix corresponding to each node is constructed; wherein the current feature tree is a feature tree of the current procedural text data, the current procedural text data is any procedural text data for training, and the current preset target tree is the current procedural text data. The preset target tree corresponding to the sequential text data; Generate a first embedding vector matrix corresponding to the current feature tree and a second embedding vector matrix corresponding to the current preset target tree according to the first adjacency matrix; Calculating the similarity between the first embedding vector matrix and the second embedding vector matrix to obtain a local structure similarity matrix; According to the local structure similarity matrix, obtaining the local structure comparison loss corresponding to the current procedural text data; According to the node attributes and neighbor nodes of each node in the current feature tree and the current preset target tree, construct a second adjacency matrix corresponding to the current feature tree and the current preset target tree respectively; Generate a first embedding vector corresponding to the current feature tree and a second embedding vector corresponding to the current preset target tree according to the second adjacency matrix; Calculating the similarity between the first embedding vector and the second embedding vector to obtain a global structural similarity; According to the global structural similarity, obtaining the global structural comparison loss corresponding to the current procedural text data; Calculate the degree of each node in the current feature tree and the current preset target tree; Calculate the degree difference of all nodes in the current feature tree and the current preset target tree to obtain the degree difference matrix; According to the degree difference matrix, obtaining the node degree loss corresponding to the current procedural text data; The local structure comparison loss, global structure comparison loss and node degree loss corresponding to the current procedural text data are added together to obtain the tree structure loss function corresponding to the current procedural text data.

15. The model training method for image-text mutual checking according to claim 14 is characterized in that: The step of generating, according to the first adjacency matrix, a first embedding vector matrix corresponding to the current feature tree and a second embedding vector matrix corresponding to the current preset target tree includes: Calculate the Laplacian matrices corresponding to each of the first adjacency matrices; Calculating the eigenvectors of each of the Laplacian matrices; According to the eigenvectors of the Laplacian matrices, obtaining the vector representations corresponding to the first adjacency matrices; wherein the vector representation corresponding to the current first adjacency matrix includes the first preset number of eigenvectors of the Laplacian matrix corresponding to the current first adjacency matrix sorted in ascending order according to the eigenvalues; The first embedding vector matrix and the second embedding vector matrix are generated according to the vector representations corresponding to the first adjacency matrices.

16. The model training method for image-text mutual checking according to claim 14, characterized in that: The step of obtaining the local structure comparison loss corresponding to the current procedural text data according to the local structure similarity matrix includes: Through L1=||H|| 2 , the local structure comparison loss is calculated; L1 is the local structure comparison loss, and H is the local structure similarity matrix.

17. The model training method for image-text mutual checking according to claim 12, characterized in that: When the preset loss function includes a ternary loss function, training a preset matching model using the preset loss function according to the text encoding and the image encoding to obtain a trained retrieval matching model includes: Initializing the weights of all network layers in the preset matching model to obtain the initialized preset matching model; Input the current batch of training data into the preset matching model respectively, and obtain the training matching results output by the preset matching model; wherein the training data includes N pairs of procedural text data and steps of the current batch Graph data; Determine whether a preset number of iterations is reached or the error between the training matching result and the preset matching result is less than a convergence threshold; If the preset number of iterations is reached or the error between the training matching result and the preset matching result is less than the convergence threshold, the preset matching model is used as the trained retrieval matching model; If the preset number of iterations is not reached and the error between the training matching result and the preset matching result is not less than the convergence threshold, the preset loss function is used to obtain the loss value corresponding to the training matching result; According to the loss value, obtaining the back propagation error of each network layer in the preset matching model; According to the back-propagation error, the weights of each network layer in the preset matching model are updated to obtain the updated preset matching model; Select the next batch of training data and use the next batch of training data as the current batch of training data, and execute the step of inputting the current batch of training data into the preset matching model respectively to obtain the training matching result output by the preset matching model.

18. The model training method for image-text mutual checking according to claim 11, characterized in that: The step of obtaining the image encoding of each step diagram data comprises: Extracting step diagram features of each step diagram in the current step diagram data; wherein the current step diagram data is any of the step diagram data; Generate a heterogeneous graph corresponding to the current step graph data according to the category labels corresponding to each of the step graphs and the step graph features; wherein the heterogeneous graph includes component nodes corresponding to the category labels, step graph nodes corresponding to the step graph features, and connection edges between each of the step graph nodes and their corresponding component nodes; According to the heterogeneous graph, an image code corresponding to the current step graph data is generated.

19. The model training method for image-text mutual checking according to claim 18, characterized in that: The step of extracting the step diagram features of each step diagram in the step diagram data comprises: The current step graph is input into a preset residual network, a feature vector output by a pooling layer of the preset residual network is obtained, and the feature vector is determined as a step graph feature corresponding to the current step graph; wherein the current step graph is any of the step graphs.

20. A model training device for image-text mutual checking, characterized in that: include: A data acquisition module, used to acquire training data; wherein the training data includes a preset number of training pairs of procedural text data and step diagram data; A text encoding module, used for obtaining the text encoding of each of the procedural text data; wherein the text encoding is an encoding generated by using a feature tree corresponding to the procedural text data, the feature tree includes a chain connection relationship tree, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features, the step text features are features of the step text in the procedural text data, the component features are features of the component text in the procedural text data, and the chain connection relationship tree is a tree corresponding to a recombinant sequence obtained by fusing each of the step texts in the order of the step texts; An image coding module, used for obtaining the image coding of each step diagram data; The model training module is used to train the preset matching model according to the text encoding and the image encoding using a preset loss function, obtain the trained retrieval matching model, and use the retrieval matching model to match the procedural text data to be retrieved or the step diagram data to be retrieved.

21. A model matching method for image-text mutual inspection, characterized in that: include: Acquire the data to be retrieved; wherein the data to be retrieved is procedural text data to be retrieved or step diagram data to be retrieved; The data encoding of the data to be retrieved is extracted by using a retrieval matching model, and the data encoding is distance matched with the sample encoding of each sample data in a sample data set to obtain matching sample data corresponding to the data to be retrieved; wherein, the sample data set is a sample step diagram data set or a sample procedural text data set; if the data to be retrieved is the procedural text data to be retrieved, then the data encoding is a text encoding generated by using a feature tree corresponding to the procedural text data to be retrieved, the feature tree includes a chain connection relationship tree, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features, the step text features are features of the step text in the procedural text data to be retrieved, the component features are features of the component text in the procedural text data to be retrieved, and the chain connection relationship tree is a tree corresponding to a recombinant sequence obtained by fusing each step text in the order of the step texts.

22. The model matching method for image-text mutual checking according to claim 21, characterized in that: The step of performing distance matching between the data code and the sample code of each sample data in the sample data set to obtain matching sample data corresponding to the data to be retrieved includes: respectively calculating the Euclidean distance between the data code and the sample code of each sample data; The sample data corresponding to the smallest Euclidean distance is used as the matching sample data.

23. A model matching device for image-text mutual inspection, characterized in that: include: A search and acquisition module, used to acquire data to be searched; wherein the data to be searched is procedural text data to be searched or step diagram data to be searched; A model matching module is used to use a retrieval matching model to extract the data encoding of the data to be retrieved, and to perform distance matching between the data encoding and the sample encoding of each sample data in a sample data set, so as to obtain matching sample data corresponding to the data to be retrieved; wherein, the sample data set is a sample step diagram data set or a sample procedural text data set; if the data to be retrieved is the procedural text data to be retrieved, then the data encoding is a text encoding generated by using a feature tree corresponding to the procedural text data to be retrieved, the feature tree includes a chain connection relationship tree, step text features mounted on the chain connection relationship tree, and component features mounted on the step text features, the step text features are features of the step text in the procedural text data to be retrieved, the component features are features of the component text in the procedural text data to be retrieved, and the chain connection relationship tree is a tree corresponding to a recombinant sequence obtained by fusing each step text in the order of the step texts.

24. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the text encoding method as described in any one of claims 1 to 9, the model training method for image-text mutual checking as described in any one of claims 11 to 19, and / or the model matching method for image-text mutual checking as described in claim 21 or 22 when executing the computer program.

25. A non-volatile readable storage medium, characterized in that: The non-volatile readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the text encoding method according to any one of claims 1 to 9, the model training method for image-text mutual checking according to any one of claims 11 to 19, and / or the model matching method for image-text mutual checking according to claim 21 or 22.

Citation Information

Patent Citations

  • Source code coding sequence representation method based on static program analysis

    CN110737466A

  • Menu file creation method and device, storage medium and electronic device

    CN111104630A

  • Question structure reasoning method and system

    CN113988300A

  • Chinese abstract generation method based on component syntactic analysis

    CN114580376A

  • Text operation graph mutual inspection method, model training method, device, equipment and medium

    CN115618043A