Task matching method and device for supervision instruction, equipment, storage medium and product
By converting regulatory instructions into instruction images and extracting features, the correlation with task features is calculated, and the problem of low efficiency of task matching in traditional regulatory instructions is solved, and efficient and accurate task matching is achieved.
Patent Information
- Application Number
- CN202411971510.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-18
AI Technical Summary
The task matching efficiency of traditional regulatory instructions is inefficient, and the limitations of text understanding lead to mismatch, which reduces efficiency.
By converting regulatory instructions into instruction images, extracting style enhances visual features and text semantic style features, and splicing them into instruction feature matrix, and computing the correlation with task feature vectors to achieve accurate matching.
It improves the task matching efficiency of regulatory instructions, reduces the probability of mismatch and the number of reworks, deepens the understanding of supervision, and achieves automatic and accurate matching.
Smart Images
Figure CN119851101A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment, storage medium and product for task matching of supervisory instructions. Background Art
[0002] The regulatory guarantee system refers to a series of systems, mechanisms and measures established to ensure the effective conduct of regulatory activities. As an important part of the regulatory guarantee system, regulatory disposal has the following main system responsibilities: receiving regulatory instructions, matching disposal tasks corresponding to regulatory instructions, and completing disposal tasks. Among them, how to match disposal tasks corresponding to regulatory instructions is the most critical.
[0003] In traditional solutions, operators generally determine the disposal tasks corresponding to the supervision instructions based on the content of the supervision instructions. However, as the number of supervisions increases, traditional solutions are prone to inefficiency. In addition, existing supervision instructions are generally only expressed in text form, which has certain limitations in understanding the text, resulting in insufficient understanding of the supervision instructions. The matched disposal tasks do not match the supervision instructions, and sometimes need to be re-matched, which reduces the efficiency of task matching. Therefore, the traditional solution has the problem of low task matching efficiency.
[0004] How to improve the efficiency of task matching for regulatory instructions remains an urgent issue. Summary of the invention
[0005] Based on this, it is necessary to provide a task matching method, device, equipment, storage medium and product for regulatory instructions that can improve the efficiency of task matching for regulatory instructions in response to the above technical problems.
[0006] In a first aspect, the present application provides a task matching method for a supervisory instruction, the method comprising:
[0007] Obtaining a supervision instruction, and converting the supervision instruction into an instruction image; wherein the content displayed by the instruction image includes text information of the supervision instruction;
[0008] Inputting the instruction image into a feature extraction model to obtain output style-enhanced visual features;
[0009] Identifying a text area in the instruction image that displays text information, and extracting text semantic style features of the text area;
[0010] The features in the style enhancement visual features are spliced with the features in the text semantic style features to form a plurality of spliced features, and an instruction feature matrix is constructed based on the plurality of spliced features; wherein the instruction feature matrix includes M groups of spliced features, each group of spliced features includes N spliced features; M and N are both natural numbers greater than zero;
[0011] Obtaining the correlation between each group of splicing features and each task feature vector, and determining the correlation of each task feature vector with respect to the instruction feature matrix;
[0012] According to the correlation of each task feature vector to the instruction feature matrix, the target task feature vector corresponding to the maximum correlation is determined, and the task corresponding to the target task feature vector is used as the best matching task for the supervision instruction.
[0013] In one embodiment, the feature extraction model includes a visual feature extraction model and a style feature extraction model; the visual feature extraction model includes a convolutional layer, a maximum pooling layer, a plurality of residual structure layers, an average pooling layer, a flattening layer and a fully connected layer connected in sequence; the style feature extraction model includes a deep residual structure layer, an average pooling layer, a flattening layer and a fully connected layer connected in sequence;
[0014] The step of inputting the instruction image into a feature extraction model to obtain output style-enhanced visual features includes:
[0015] Input the command image into the convolution layer of the visual feature extraction model to obtain the visual features output by the fully connected layer;
[0016] In the process of the visual feature extraction model processing the command image, extracting the intermediate feature map output by each of the convolutional layer, the maximum pooling layer and the residual structure layer in the visual feature extraction model, and obtaining the style feature map of the command image according to the intermediate feature map output by each layer; inputting the style feature map into the deep residual structure layer of the style feature extraction model to obtain the image style features output by the fully connected layer;
[0017] The visual features and the image style features are concatenated to obtain the style-enhanced visual features of the instruction image.
[0018] In one embodiment, obtaining the style feature map of the instruction image according to the intermediate feature map output by each layer includes:
[0019] For the intermediate feature map output by each layer, extract the edge style feature matrix of the intermediate feature map;
[0020] The edge style feature matrices corresponding to the intermediate feature maps output by each layer are concatenated to obtain the style feature map of the instruction image.
[0021] In one embodiment, the step of extracting an edge style feature matrix of an intermediate feature map includes:
[0022] Perform Hessian matrix operation on the intermediate feature map to extract edge features of the intermediate feature map;
[0023] A Gram matrix operation is performed on the edge features to extract the style feature matrix of the edge features.
[0024] In one embodiment, extracting the text semantic style features of the text region includes:
[0025] dividing the text region into a plurality of text sub-regions;
[0026] For each text sub-region, convert the text sub-region into a first feature map; wherein the width of the first feature map is w, the height is h, and the number of channels is c;
[0027] Perform average pooling on the height h of the first feature map to obtain a second feature map;
[0028] Representing the second feature map as a time series, wherein the length of the time series is equal to the width w;
[0029] Inputting the time series into a long short-term memory network, and outputting the features retained by the memory unit in the long short-term memory network at the final moment as the text semantic style features of the text sub-region;
[0030] The text semantic style features of the text region are obtained according to the text semantic style features of the plurality of text sub-regions.
[0031] In one embodiment, determining the target task feature vector corresponding to the maximum correlation according to the correlation of each task feature vector to the instruction feature matrix includes:
[0032] For each task feature vector and each splicing feature in each group of splicing features, calculate the correlation between each splicing feature and the task feature vector, and obtain N first correlations for a group of splicing features;
[0033] For each group of splicing features, N first correlations are summed and averaged to obtain a second correlation between a group of splicing features and the task feature vector, and M second correlations are obtained for the task feature vector; the M second correlations are summed and averaged to obtain a third correlation of the task feature vector with respect to the instruction feature matrix;
[0034] A target task feature vector corresponding to the maximum correlation is determined from the third correlations corresponding to the plurality of task feature vectors.
[0035] In a second aspect, the present application provides a task matching device for a supervisory instruction, comprising:
[0036] An acquisition module, used to obtain regulatory instructions;
[0037] A processing module, used for converting the supervision instruction into an instruction image; wherein the content displayed by the instruction image includes the text information of the supervision instruction;
[0038] The processing module is further used to input the instruction image into a feature extraction model to obtain output style-enhanced visual features;
[0039] The processing module is also used to identify a text area in the instruction image that displays text information, and extract text semantic style features of the text area;
[0040] The processing module is further used to splice the features in the style enhancement visual features with the features in the text semantic style features to form a plurality of spliced features, and construct an instruction feature matrix based on the plurality of spliced features; wherein the instruction feature matrix includes M groups of spliced features, each group of spliced features includes N spliced features; M and N are both natural numbers greater than zero;
[0041] The processing module is further used to obtain the correlation between each group of splicing features and each task feature vector, and determine the correlation of each task feature vector to the instruction feature matrix;
[0042] The matching module is used to determine the target task feature vector corresponding to the maximum correlation according to the correlation of each task feature vector to the instruction feature matrix, and take the task corresponding to the target task feature vector as the best matching task for the supervision instruction.
[0043] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first aspect are implemented.
[0044] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.
[0045] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect.
[0046] The task matching method, device, equipment, storage medium and product of the supervision instruction provided by the embodiment of the present application visualize the content of the supervision instruction, rely on the extraction and splicing of the image visual style enhancement feature and the text semantic style feature, and finally match and calculate with the task feature. In this way, the supervision instruction is upgraded from a single text content to an instruction image and the instruction image contains the text information of the supervision instruction, and the style enhancement visual features and text semantic style features of the instruction image are extracted respectively. Compared with the traditional scheme, the method provided by this embodiment deepens the understanding of supervision, can accurately extract the features of the supervision instruction, and thus accurately determine the disposal task that matches the supervision instruction. And the method provided by this embodiment, by calculating the correlation between the instruction feature matrix and multiple task feature vectors, can further realize the automatic and accurate matching of the monitoring instruction and the task. Because it can accurately match, the probability of mismatching is reduced and the number of reworks is reduced, so the task matching efficiency for the supervision instruction can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0048] Figure 1 An application environment diagram of a task matching method for supervisory instructions in an embodiment;
[0049] Figure 2 A flowchart of a task matching method for supervisory instructions in one embodiment;
[0050] Figure 3 is a schematic diagram of a feature extraction model in one embodiment;
[0051] Figure 4 A partial flow chart of a task matching method for supervisory instructions in one embodiment;
[0052] Figure 5 A partial flow chart of a task matching method for supervisory instructions in another embodiment;
[0053] Figure 6 A partial flow chart of a task matching method for supervisory instructions in yet another embodiment;
[0054] Figure 7 A schematic diagram of extracting text semantic style features in a task matching method for supervisory instructions in an embodiment;
[0055] Figure 8A partial flow chart of a task matching method for supervisory instructions in another embodiment;
[0056] Fig. 9 Another flowchart of a task matching method for supervisory instructions in one embodiment;
[0057] Fig.10 A structural block diagram of a task matching device for supervisory instructions in an embodiment;
[0058] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0060] The task matching method for supervisory instructions provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.
[0061] In an exemplary embodiment, Figure 2 As shown, a task matching method for supervisory instructions is provided, which is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 201 to 205. Among them:
[0062] Step 201: Obtain a supervisory instruction and convert the supervisory instruction into an instruction image.
[0063] Among them, the supervision instruction refers to the instruction issued by the superior system or sent by the terminal during the supervision process to request supervision of a certain system. A certain system is for example a user information management system. The supervision instruction may be a piece of text, a code snippet, a flowchart or other forms of information.
[0064] The regulatory instruction can be converted into an instruction image by using a screenshot tool to capture the instruction image on the interface containing the regulatory instruction. The content displayed by the instruction image includes the text information of the regulatory instruction. Other tools can also be used to convert the regulatory instruction into an instruction image, as long as the instruction image includes the text information of the regulatory instruction. The text information of the regulatory instruction refers to the text information displayed on the interface of the regulatory instruction. If the regulatory instruction is a flowchart, the text information of the regulatory instruction includes the text information in the flowchart.
[0065] Step 202: input the instruction image into a feature extraction model to obtain output style-enhanced visual features.
[0066] Among them, the style enhanced visual features can be understood as visual features that have been processed by style enhancement.
[0067] The feature extraction model is used to extract the visual features and style features of the instruction image, and then concatenate or fuse the visual features with the style features to obtain style-enhanced visual features.
[0068] Exemplarily, the instruction image is input into the visual extraction network of the feature extraction model to obtain the output result of the visual extraction network, and the output result is the visual feature of the instruction image.
[0069] Optionally, the visual extraction network includes multiple series-connected visual extraction layers (i.e., convolutional layers), each of which divides the input command image into multiple overlapping sub-visual features (i.e., rectangular sensory areas), and then obtains the visual features of the command image by performing weighted combination processing on the pixel values in each sub-visual feature.
[0070] Exemplarily, the command image is input into the style extraction network of the feature extraction model to obtain the style features of the command image extracted by the style extraction network. Then, the visual features are enhanced according to the style features of the command image to obtain the style-enhanced visual features. Optionally, when performing the style enhancement process on the visual features, the visual features of the command image and the style features of the command image may be feature spliced to obtain the style-enhanced visual features of the command image.
[0071] Step 203: Identify a text area in the instruction image that displays text information, and extract text semantic style features of the text area.
[0072] By detecting the text information in the instruction image, the text area in the instruction image where the text information is displayed can be identified. Exemplarily, the text area detection is performed on the instruction image, and the potential text area can be identified by finding the connected area (i.e., pixel block) in the instruction image. Exemplarily, a trained model (such as a convolutional neural network) can be used to detect the text area in the instruction image.
[0073] After the text region is identified, the text region can be input into a semantic style feature extraction model to obtain output text semantic style features.
[0074] Step 204 , concatenating the features in the style-enhanced visual features with the features in the text semantic style features to form a plurality of concatenated features, and constructing an instruction feature matrix based on the plurality of concatenated features.
[0075] It should be noted that the style-enhanced visual feature and the text semantic style feature can be understood as feature matrices, and concatenating the features in the style-enhanced visual feature with the features in the text semantic style feature can be understood as concatenating any two features in the two feature matrices. For example, a concatenated feature can be expressed as Represents the concatenation value of the i-th style-enhanced visual feature and the j-th text semantic style feature. Assuming that the style-enhanced visual feature is a 2×2 matrix and the text semantic style feature is a 3×3 matrix, 36 concatenation features can be obtained. It should be noted that the distance between the style-enhanced visual feature and the text semantic style feature is only used to illustrate how to form multiple concatenation features, and does not limit the style-enhanced visual feature and the text semantic style feature.
[0076] An instruction feature matrix is constructed based on multiple splicing features, wherein the instruction feature matrix includes M groups of splicing features, each group of splicing features includes N splicing features, and both M and N are natural numbers greater than zero.
[0077] The instruction feature matrix can be expressed as .
[0078] Step 205, obtaining the correlation between each group of splicing features and each task feature vector, and determining the correlation of each task feature vector with respect to the instruction feature matrix.
[0079] Among them, each set of splicing features can be expressed as [ , , …, ],in Represents the concatenation value of the i-th style enhancement visual feature and the j-th text semantic style feature, j=1, 2,3,…,N, i=1,2,3,…,M. There are M sets of concatenated features in the instruction feature matrix.
[0080] Among them, multiple task feature vectors can be expressed as [ , , …, ]. Each task feature vector can be expressed as , where j=1,2,3,…,N. One task feature vector corresponds to one task. Optionally, before executing step 205, each task to be matched is expressed as a task feature vector, and multiple task feature vectors are formed based on multiple tasks to be matched.
[0081] The correlation between each set of concatenated features and each task feature vector can be understood as [ , , …, ]and The correlation between them. Where j=1,2,3,…,N, i=1,2,3,…,M. For example, the correlation between the first set of concatenated features and the first task feature vector is [ , , …, ]and The correlation between them. This correlation can also be understood as a correlation coefficient.
[0082] Optionally, for each splicing feature in a group of splicing features, the correlation between the splicing feature and the task vector feature can be obtained, and then the correlation between a group of splicing features and the task feature vector can be determined based on the correlation corresponding to each splicing feature. For example, the correlation corresponding to each splicing feature in a group of splicing features is summed, and the summed value is used as the correlation between a group of splicing features and the task feature vector. By traversing M groups of splicing features, the correlation between each group of splicing features in the M groups of splicing features and the task feature vector can be obtained.
[0083] After obtaining the correlation between each group of spliced features and the task feature vector, the correlation between each group of spliced features and the task feature vector can be added, and the added value is used as the correlation of the task feature vector to the instruction feature matrix. For example, M correlations are added, and the value after the addition of the M correlations is used as the correlation of the task feature vector to the instruction feature matrix. Assuming that there are N task feature vectors in total, after traversing each task feature vector, the correlation of the N task feature vectors to the instruction feature matrix can be obtained, that is, N correlations are obtained.
[0084] Step 206, based on the correlation of each task feature vector to the instruction feature matrix, determine the target task feature vector corresponding to the maximum correlation, and use the task corresponding to the target task feature vector as the best matching task for the supervision instruction.
[0085] The maximum correlation refers to the correlation with the largest absolute value.
[0086] As described above, after obtaining N correlations, the maximum correlation can be screened out from the N correlations, and the task feature vector corresponding to the maximum correlation is determined as the target task feature vector. The correlation degree of the instruction feature matrix is the largest, then is the target task feature vector.
[0087] The task corresponding to the target task feature vector is the best matching task for the supervisory instruction.
[0088] The method provided in this embodiment visualizes the content of the supervision instruction, relies on the extraction and splicing of the image visual style enhancement features and the text semantic style features, and finally performs matching calculations with the task features. In this way, the supervision instruction is upgraded from a single text content to an instruction image, and the instruction image contains the text information of the supervision instruction, and the style enhancement visual features and text semantic style features of the instruction image are extracted respectively. Compared with the traditional scheme, the method provided in this embodiment deepens the understanding of supervision, and can accurately extract the features of the supervision instruction, so as to accurately determine the disposal tasks that match the supervision instruction. And the method provided in this embodiment can further realize the automatic and accurate matching of monitoring instructions and tasks by calculating the correlation between the instruction feature matrix and multiple task feature vectors. Because it can accurately match, the probability of mismatching is reduced and the number of rework is reduced, so the task matching efficiency for supervision instructions can be improved.
[0089] For an exemplary embodiment, see Figure 3 The schematic diagram of the feature extraction model shown in FIG. 1 includes a visual feature extraction model and a style feature extraction model. The visual feature extraction model includes a sequentially connected convolutional layer, a maximum pooling layer, multiple residual structure layers, an average pooling layer, a flattening layer, and a fully connected layer; the style feature extraction model includes a sequentially connected deep residual structure layer, an average pooling layer, a flattening layer, and a fully connected layer.
[0090] See also Figure 3 The schematic diagram of the feature extraction model shown in Figure 4 In the flowchart shown, step 202 includes:
[0091] Step 401, input the instruction image into the convolution layer of the visual feature extraction model to obtain the visual features output by the fully connected layer.
[0092] Combination Figure 3 It can be seen that the instruction image is input into the convolution layer of the visual feature extraction model, and is processed in sequence by the convolution layer, the maximum pooling layer, the residual structure 1, the residual structure 2, the residual structure 3, the residual structure 4, the average pooling layer, and the flattening layer, and finally the visual features are output through the fully connected layer. It can be understood that the feature extraction model can be trained according to actual needs, and this embodiment does not limit it.
[0093] Step 402, during the process of the visual feature extraction model processing the instruction image, extract the intermediate feature map output by each layer of the convolution layer, the maximum pooling layer and each residual structure layer in the visual feature extraction model, and obtain the style feature map of the instruction image according to the intermediate feature map output by each layer.
[0094] Combination Figure 3 It can be seen that each layer in the convolution layer, the maximum pooling layer and the residual structure layer in the visual feature extraction model will output an intermediate feature map. After processing the intermediate feature map, the style feature map of the instruction image can be obtained.
[0095] Exemplarily, for each intermediate feature map, edge features of the intermediate feature map are extracted, and style element features of the edge features are extracted. The edge style features for the intermediate feature map can be obtained by combining the style elements and the edge features. After traversing each intermediate feature map, the edge style features of each intermediate feature map can be obtained. After splicing the edge style features of each intermediate feature map, the style feature map of the instruction image can be obtained. It should be noted that the style feature map can be a Gram feature map.
[0096] Optionally, an edge detection algorithm may be used to extract edge features of the intermediate feature map. A style extraction network may be used to extract style element features of the edge features.
[0097] Step 403: input the style feature map into the deep residual structure layer of the style feature extraction model to obtain the image style features output by the fully connected layer.
[0098] The style feature map of the instruction image is input into the deep residual structure layer of the style feature extraction model. After being processed in sequence by the deep residual structure layer, the average pooling layer, the flattening layer and the fully connected layer, the image style features of the instruction image are output by the fully connected layer.
[0099] Step 404 , concatenating the visual features and the image style features to obtain the style-enhanced visual features of the instruction image.
[0100] The visual features and image style features can be understood as feature matrices, and the concatenation of the two feature matrices can be understood as the random combination of each feature vector in the feature matrix, and finally the style-enhanced visual features of the instruction image are obtained. The style-enhanced visual features are feature matrices.
[0101] The method provided in this embodiment extracts the visual features of the instruction image through a visual feature extraction model, and supports the extraction of image style features of the instruction image by adding a style feature extraction model. The visual features and style features of the instruction image can be accurately extracted.
[0102] For an exemplary embodiment, see Figure 5 Step 402 obtains the style feature map of the instruction image according to the intermediate feature map output by each layer, including:
[0103] Step 501: for the intermediate feature map output by each layer, extract the edge style feature matrix of the intermediate feature map.
[0104] First, the edge features of the intermediate feature map are extracted, and then the style features of the edge features are extracted to obtain the edge style feature matrix of the intermediate feature map.
[0105] Exemplarily, an edge detection algorithm may be used to extract edge features of the intermediate feature map.
[0106] Exemplarily, for each intermediate feature map, a Hessian matrix operation is performed on the intermediate feature map to extract the edge features of the intermediate feature map. The Hessian Matrix, also translated as the Hessian Matrix, the Hessian Matrix, the Hessian Matrix, etc., is mainly used to extract the key points of the image in image processing, and helps to identify the image features by analyzing the curvature and shape information of the image at a certain point. It can be understood that the edge is where the brightness or color in the image changes significantly. At the edge, the gradient (first-order derivative) of the image is usually large, while the change in the gradient (second-order derivative) may have extreme values. Therefore, the Hessian matrix can be used to detect edge features in the image. Specifically, the eigenvalues of the Hessian matrix can be used to determine whether the pixel belongs to the edge. If the absolute values of the eigenvalues of the Hessian matrix are all greater than a certain threshold, the pixel is considered to belong to the edge. Usually, the largest eigenvalue is selected as the threshold.
[0107] Exemplarily, when extracting style features of edge features, the edge features are input into a style feature extraction network to obtain style features output by the style feature extraction network.
[0108] For example, when extracting style features of edge features, each edge feature map may be flattened into a vector, and then the product of the vector and its transpose may be calculated.
[0109] For example, a Gram matrix operation is performed on the edge features to extract the edge style feature matrix of the edge features. Specifically, first, the set of feature vectors in the edge features is organized into a matrix, in which each row represents a feature vector. The product of the transposed matrix of this matrix and the original matrix is calculated to obtain the Gram matrix. The elements of the Gram matrix represent the inner products between different feature vectors, that is, the correlation or similarity between them.
[0110] Specifically, assuming that after a certain intermediate feature map is processed by the Hessian matrix, the feature map size of the output edge feature is in Represents width, Represents high, is the number of channels. Flatten the features of the ith channel in the edge feature into a first feature vector with width and height. Flatten the features of the jth channel in the edge feature into a second feature vector with width and height. Calculate the inner product of the first feature vector and the second feature vector and use it as the style feature matrix of the edge feature. The feature map of the channels is flattened into The first eigenvector of dimension is denoted as ,have Flatten the feature map of the jth channel in the edge feature to The second eigenvector of dimension is After the Gram matrix operation, the obtained Gram matrix for: .in, Represents the feature vector and The inner product of . The Gram matrix represents the marginal style feature matrix.
[0111] Step 502, concatenate the edge style feature matrices corresponding to the intermediate feature maps output by each layer to obtain the style feature map of the instruction image.
[0112] The Gram matrices of the intermediate feature maps at different levels (including convolutional layers, maximum pooling layers, and multiple residual structure layers) are aggregated and concatenated to form a Gram feature map. Specifically, the elements of the edge style feature matrices (Gram matrices) at different levels are used as the pixel values of the image to generate a two-dimensional image, which is the Gram feature map.
[0113] The Gram feature map is input into the deep residual structure layer of the style feature extraction model to obtain the image style features output by the fully connected layer. Among them, the Gram matrix describes the general style of the image.
[0114] It can be understood that the Gram matrix feature map is composed of the Gram matrix of convolution results at different levels, which measures the image style from a holistic perspective and provides style information to supplement the image visual features. The use of the Gram matrix map can deepen the understanding of the instruction image.
[0115] For an exemplary embodiment, see Figure 6 The process diagram and Figure 7 Schematic diagram of extracting text semantic style features.
[0116] Step 203 extracts the text semantic style features of the text area, including:
[0117] Step 601: divide the text area into multiple text sub-areas.
[0118] It can be understood that the image of the text area is divided into multiple sub-images, and one sub-image corresponds to one text sub-area.
[0119] Step 602: For each text sub-region, convert the text sub-region into a first feature map.
[0120] Among them, the width of the first feature map is w, the height is h, and the number of channels is c.
[0121] Convert text sub-regions into sizes using convolutional neural networks The feature map is used as the first feature map. Represents the width, Represents height, Represents the number of channels.
[0122] Step 603: Perform average pooling on the height h of the first feature map to obtain a second feature map.
[0123] Please refer to Figure 7 , after inputting the sub-text region into the convolutional neural network, the feature map of the sub-text region is obtained. Dimensions are averaged and pooled to obtain a size of and used as the second feature map.
[0124] Step 604: represent the second feature map as a time series, where the length of the time series is equal to the width w.
[0125] The second feature map is represented as a length of (i.e., a time series of width w), expressed as ,in Indicates the first indivual -dimensional feature vector.
[0126] Step 605: input the time series into the long short-term memory network, and output the features retained by the memory units in the long short-term memory network at the final moment as the text semantic style features of the text sub-region.
[0127] See also Figure 7 , the feature map time series As Xinput, input the Long Short-Term Memory (LSTM) network, and convert the features retained by the memory unit in the LSTM network at the final moment into The output is the text semantic style feature of the text sub-region. LSTM also outputs the time series output feature Y output .
[0128] Step 606: Obtain text semantic style features of the text region based on the text semantic style features of the multiple text sub-regions.
[0129] After averaging the text semantic style features of multiple text sub-regions, the text semantic style features of the text region are obtained. The averaging process can be understood as obtaining the value of the element in the text semantic style features of each text sub-region for each element in the same position in the text semantic style features of multiple text sub-regions, and averaging the value of the element in the text semantic style features of each text sub-region, and finally obtaining the text semantic style features of the text region after averaging after traversing each element and the text semantic style features of each text sub-region.
[0130] The method provided in this embodiment extracts the text semantic style features of the text region by dividing the text region into multiple text sub-regions and based on the multiple text sub-regions and the long short-term memory network. And by extracting the style enhancement visual features of the instruction image, the supervision instruction can be upgraded from a single text content to an instruction image, and the instruction image contains the text information of the supervision instruction. Compared with the traditional solution, the understanding of supervision is deepened, and the features of the supervision instruction can be accurately extracted, so as to accurately determine the disposal task matching the supervision instruction.
[0131] For an exemplary embodiment, see Figure 8 Step 206 determines the target task feature vector corresponding to the maximum correlation according to the correlation of each task feature vector to the instruction feature matrix, including:
[0132] Step 801 , for each task feature vector and each splicing feature in each group of splicing features, calculate the correlation between each splicing feature and the task feature vector, and obtain N first correlations for a group of splicing features.
[0133] For example, select a set of feature vector concatenation values and compare them with the task feature vector Calculate the correlation.
[0134] Using formula Calculate splicing features With the task feature vector The first correlation between . The maximum value of j is equal to N, and the maximum value of i is equal to M. is a set of average eigenvectors calculated by concatenating eigenvectors. For example, the concatenated value of a set of eigenvectors is [ , , …, ], that is, i is equal to 1, where .in is the average feature vector calculated based on N task feature vectors, .
[0135] The above formula is used to calculate each splicing feature in a set of splicing features With each task feature vector After the first correlation between them is obtained, N first correlations can be obtained for a set of splicing features.
[0136] Step 802 , for each group of splicing features, N first correlations are summed and averaged to obtain a second correlation between a group of splicing features and the task feature vector, and M second correlations are obtained for one task feature vector.
[0137] For a set of splicing features, the N first correlations are summed and averaged to obtain a set of splicing features and task feature vectors For a task feature vector, after traversing M groups of spliced features in the instruction feature matrix, M second correlations for a task feature vector can be obtained.
[0138] Step 803 , performing sum and average processing on the M second correlations to obtain a third correlation of the task feature vector with respect to the instruction feature matrix.
[0139] For each task feature vector, step 803 is performed to determine the third correlation of each task feature vector to the instruction feature matrix. Assuming there are N task feature vectors, N third correlations can be obtained.
[0140] Step 804: determine the target task feature vector corresponding to the maximum correlation from the third correlations corresponding to the multiple task feature vectors.
[0141] Assume that there are 3 third correlations corresponding to 3 task feature vectors, that is, there are 3 third correlations, among which the task feature vector The corresponding third correlation is greater than The corresponding third correlation and The corresponding third correlation, then the task feature vector is the target task feature vector.
[0142] Exemplarily, after obtaining the third relevances corresponding to the multiple task feature vectors, the third relevances corresponding to the multiple task feature vectors are sorted, and the task feature vector corresponding to the maximum third similarity is used as the target task feature vector.
[0143] Exemplarily, a correlation matrix between the instruction feature matrix and a plurality of task feature vectors may be constructed, and then the target task feature vector may be determined based on the correlation matrix.
[0144] The incidence matrix can be expressed as: .
[0145] The method provided in this embodiment matches the instruction feature matrix with the task feature vector to achieve the best match between the instruction and the task.
[0146] For ease of understanding, the following is combined with Fig. 9 The method provided in this embodiment is described. The task matching method of the supervision instruction includes:
[0147] The first step is to receive regulatory instructions.
[0148] The second step is to convert the supervision instruction into an instruction image, and input the instruction image into the visual feature extraction model, style feature extraction model and text semantic style feature extraction network. The visual feature extraction model is used to output the visual features of the instruction image, and the style feature extraction model is used to extract the image style features of the instruction image. The text semantic style feature extraction network is used to output the text semantic style features of the text area in the instruction image.
[0149] The third step is to concatenate the visual features of the instruction image with the image style features to obtain the style-enhanced visual features of the instruction image. The style-enhanced visual features of the instruction image and the text semantic style features of the text region in the instruction image are concatenated to form a plurality of concatenated features.
[0150] The fourth step is to construct an instruction feature matrix based on multiple splicing features, and perform task matching based on the instruction feature matrix to finally obtain the best matching task for the supervisory instruction.
[0151] It should be understood that, although the steps in the flowcharts of the embodiments described above are shown in sequence according to the instructions of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.
[0152] Based on the same inventive concept, the embodiment of the present application also provides a task matching device for supervisory instructions for implementing the task matching method for supervisory instructions involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiment of the task matching device for one or more supervisory instructions provided below can refer to the limitations of the task matching method for supervisory instructions above, and will not be repeated here.
[0153] In an exemplary embodiment, Fig.10 As shown, a task matching device 10 for supervisory instructions is provided, comprising: an acquisition module 11, a processing module 12 and a matching module 13, wherein:
[0154] The acquisition module 11 is used to acquire the supervision instruction.
[0155] The processing module 12 is used to convert the supervision instruction into an instruction image, wherein the content displayed by the instruction image includes the text information of the supervision instruction.
[0156] The processing module 12 is further used to input the instruction image into a feature extraction model to obtain output style-enhanced visual features.
[0157] The processing module 12 is further used to identify a text area in the instruction image that displays text information, and to extract text semantic style features of the text area.
[0158] The processing module 12 is also used to splice the features in the style-enhanced visual features with the features in the text semantic style features to form a plurality of spliced features, and to construct an instruction feature matrix based on the plurality of spliced features; wherein the instruction feature matrix includes M groups of spliced features, each group of spliced features includes N spliced features; and M and N are both natural numbers greater than zero.
[0159] The processing module 12 is further used to obtain the correlation between each group of splicing features and each task feature vector, and determine the correlation of each task feature vector with respect to the instruction feature matrix.
[0160] The matching module 13 is used to determine the target task feature vector corresponding to the maximum correlation according to the correlation of each task feature vector to the instruction feature matrix, and take the task corresponding to the target task feature vector as the best matching task for the supervision instruction.
[0161] In one embodiment, the feature extraction model includes a visual feature extraction model and a style feature extraction model; the visual feature extraction model includes a sequentially connected convolutional layer, a maximum pooling layer, a plurality of residual structure layers, an average pooling layer, a flattening layer and a fully connected layer; the style feature extraction model includes a sequentially connected deep residual structure layer, an average pooling layer, a flattening layer and a fully connected layer;
[0162] The processing module 12 is used to: input the instruction image into the convolution layer of the visual feature extraction model to obtain the visual features output by the fully connected layer; in the process of the visual feature extraction model processing the instruction image, extract the intermediate feature map output by each layer of the convolution layer, the maximum pooling layer and the residual structure layer in the visual feature extraction model, and obtain the style feature map of the instruction image according to the intermediate feature map output by each layer; input the style feature map into the deep residual structure layer of the style feature extraction model to obtain the image style features output by the fully connected layer; splice the visual features and the image style features to obtain the style-enhanced visual features of the instruction image.
[0163] In one embodiment, the processing module 12 is used to: extract the edge style feature matrix of the intermediate feature map output by each layer; and splice the edge style feature matrices corresponding to the intermediate feature map output by each layer to obtain the style feature map of the instruction image.
[0164] In one embodiment, the processing module 12 is used to: perform Hessian matrix operation on the intermediate feature map to extract edge features of the intermediate feature map; perform Gram matrix operation on the edge features to extract a style feature matrix of the edge features.
[0165] In one embodiment, the processing module 12 is used to: divide the text region into multiple text sub-regions; for each text sub-region, convert the text sub-region into a first feature map; wherein the width of the first feature map is w, the height is h, and the number of channels is c; average pooling is performed on the height h of the first feature map to obtain a second feature map; represent the second feature map as a time series, the length of the time series is equal to the width w; input the time series into a long short-term memory network, and output the features retained by the memory units in the long short-term memory network at the final moment as the text semantic style features of the text sub-region; obtain the text semantic style features of the text region based on the text semantic style features of multiple text sub-regions.
[0166] In one embodiment, the matching module 13 is used to: for each task feature vector and each splicing feature in each group of splicing features, calculate the correlation between each splicing feature and the task feature vector, and obtain N first correlations for a group of splicing features; for each group of splicing features, sum and average the N first correlations to obtain a second correlation between a group of splicing features and the task feature vector, and obtain M second correlations for the task feature vector; sum and average the M second correlations to obtain a third correlation of the task feature vector with respect to the instruction feature matrix; and determine the target task feature vector corresponding to the maximum correlation from the third correlations corresponding to multiple task feature vectors.
[0167] Each module in the task matching device 10 of the above-mentioned supervisory instruction can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0168] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.11As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store style-enhanced visual features, text semantic style features, instruction feature matrices, multiple task feature vectors, etc. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the task matching method of the supervisory instruction provided in any of the above embodiments is implemented.
[0169] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0170] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the task matching method of the supervisory instruction provided in any of the above embodiments is implemented.
[0171] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the task matching method of the supervisory instruction provided in any of the above embodiments.
[0172] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0173] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0174] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A task matching method for supervisory instructions, characterized in that: The method comprises: Obtaining a supervision instruction, and converting the supervision instruction into an instruction image; wherein the content displayed by the instruction image includes text information of the supervision instruction; Inputting the instruction image into a feature extraction model to obtain output style-enhanced visual features; Identifying a text area in the instruction image that displays text information, and extracting text semantic style features of the text area; The features in the style enhancement visual features are spliced with the features in the text semantic style features to form a plurality of spliced features, and an instruction feature matrix is constructed based on the plurality of spliced features; wherein the instruction feature matrix includes M groups of spliced features, each group of spliced features includes N spliced features; M and N are both natural numbers greater than zero; Obtaining the correlation between each group of splicing features and each task feature vector, and determining the correlation of each task feature vector with respect to the instruction feature matrix; According to the correlation of each task feature vector to the instruction feature matrix, the target task feature vector corresponding to the maximum correlation is determined, and the task corresponding to the target task feature vector is used as the best matching task for the supervision instruction.
2. The method according to claim 1, characterized in that The feature extraction model includes a visual feature extraction model and a style feature extraction model; the visual feature extraction model includes a convolutional layer, a maximum pooling layer, a plurality of residual structure layers, an average pooling layer, a flattening layer and a fully connected layer connected in sequence; the style feature extraction model includes a deep residual structure layer, an average pooling layer, a flattening layer and a fully connected layer connected in sequence; The step of inputting the instruction image into a feature extraction model to obtain output style-enhanced visual features includes: Input the command image into the convolution layer of the visual feature extraction model to obtain the visual features output by the fully connected layer; In the process of the visual feature extraction model processing the command image, extracting the intermediate feature map output by each of the convolutional layer, the maximum pooling layer and the residual structure layer in the visual feature extraction model, and obtaining the style feature map of the command image according to the intermediate feature map output by each layer; inputting the style feature map into the deep residual structure layer of the style feature extraction model to obtain the image style features output by the fully connected layer; The visual features and the image style features are concatenated to obtain the style-enhanced visual features of the instruction image.
3. The method according to claim 2, characterized in that The step of obtaining the style feature map of the instruction image according to the intermediate feature map output by each layer includes: For the intermediate feature map output by each layer, extract the edge style feature matrix of the intermediate feature map; The edge style feature matrices corresponding to the intermediate feature maps output by each layer are concatenated to obtain the style feature map of the instruction image.
4. The method according to claim 3, characterized in that The step of extracting the edge style feature matrix of the intermediate feature map includes: Perform Hessian matrix operation on the intermediate feature map to extract edge features of the intermediate feature map; A Gram matrix operation is performed on the edge features to extract the style feature matrix of the edge features.
5. The method according to claim 1, characterized in that The extracting of text semantic style features of the text area includes: dividing the text region into a plurality of text sub-regions; For each text sub-region, convert the text sub-region into a first feature map; wherein the width of the first feature map is w, the height is h, and the number of channels is c; Perform average pooling on the height h of the first feature map to obtain a second feature map; Representing the second feature map as a time series, wherein the length of the time series is equal to the width w; Inputting the time series into a long short-term memory network, and outputting the features retained by the memory unit in the long short-term memory network at the final moment as the text semantic style features of the text sub-region; The text semantic style features of the text region are obtained according to the text semantic style features of the plurality of text sub-regions.
6. The method according to claim 1, characterized in that The step of determining the target task feature vector corresponding to the maximum correlation according to the correlation of each task feature vector to the instruction feature matrix includes: For each task feature vector and each splicing feature in each group of splicing features, calculate the correlation between each splicing feature and the task feature vector, and obtain N first correlations for a group of splicing features; For each group of splicing features, N first correlations are summed and averaged to obtain a second correlation between a group of splicing features and the task feature vector, and M second correlations are obtained for the task feature vector; the M second correlations are summed and averaged to obtain a third correlation of the task feature vector with respect to the instruction feature matrix; A target task feature vector corresponding to the maximum correlation is determined from the third correlations corresponding to the plurality of task feature vectors.
7. A task matching device for supervisory instructions, characterized in that: include: An acquisition module, used to obtain regulatory instructions; A processing module, used for converting the supervision instruction into an instruction image; wherein the content displayed by the instruction image includes the text information of the supervision instruction; The processing module is further used to input the instruction image into a feature extraction model to obtain output style-enhanced visual features; The processing module is also used to identify a text area in the instruction image that displays text information, and extract text semantic style features of the text area; The processing module is further used to splice the features in the style enhancement visual features with the features in the text semantic style features to form a plurality of spliced features, and construct an instruction feature matrix based on the plurality of spliced features; wherein the instruction feature matrix includes M groups of spliced features, each group of spliced features includes N spliced features; M and N are both natural numbers greater than zero; The processing module is further used to obtain the correlation between each group of splicing features and each task feature vector, and determine the correlation of each task feature vector to the instruction feature matrix; The matching module is used to determine the target task feature vector corresponding to the maximum correlation according to the correlation of each task feature vector to the instruction feature matrix, and take the task corresponding to the target task feature vector as the best matching task for the supervision instruction.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.