Automatic content editor
Through trained neural network editing tools, the updated version of content items is automatically predicted and calculated, which solves the complexity and time-consuming problems of end users when editing content items, and improves editing efficiency and user experience.
Patent Information
- Application Number
- CN201980064197.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-09-27
- Filing Date
- 2019-06-28
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2039-06-28
AI Technical Summary
In the prior art, when end users use computers to edit content items (such as audio files, images, text documents, etc.), they need to manually enter commands, which makes operations complex and time-consuming, especially on resource-constrained devices difficult to achieve efficient editing.
Using trained neural network editing tools, we predict and calculate updated versions of content items by receiving the first version and change representation of content items, reducing user input burden, and providing automated editing functions.
It greatly reduces the input burden of users when editing content items, improves editing efficiency, and realizes simplified human-computer interaction, especially on resource-constrained devices.
Smart Images

Figure CN112789591B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure generally relate to automatic content editors. Background Art
[0002] Editing content items such as audio files, images, text documents, emails, blog posts, code snippets, and other content items is time consuming and difficult for end users to accomplish using computer technology. End users are often unfamiliar with how to enter commands into a computer to edit content items, or find it difficult to perform this operation due to keyboard input errors, voice command input errors, or gesture input errors. The limitations of space and processing power on resource-constrained devices also result in difficulties when attempting to edit content using such resource-constrained devices.
[0003] The embodiments described below are not limited to implementations that solve any or all disadvantages of known content editing methods and apparatus. Summary of the invention
[0004] A simplified summary of the present disclosure is presented below in order to provide the reader with a basic understanding. This summary is neither intended to identify key features or essential features of the claimed subject matter nor to be used to limit the scope of the claimed subject matter. Its sole purpose is to present a selection of concepts disclosed herein in a simplified form as a prelude to a more detailed description that is presented later.
[0005] In various examples, there is an editing tool that includes a memory storing a neural network that has been trained to compute a variation representation from a plurality of pairs, each pair including a representation of a first version of a content item and a second version of the content item, and for each variation representation, the neural network has been trained to predict an updated content item based on the variation representation and the first version of the content item. The editing tool has a processor configured to receive an input content item and compute an updated version of the input content item based on the variation representation using the neural network.
[0006] Many of the attendant features will become better understood as the same can be better appreciated by reference to the following detailed description considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] The present specification will be better understood from the following detailed description read in conjunction with the accompanying drawings, in which:
[0008] Figure 1 is a schematic diagram of a content editor communicating with various end-user devices via a communication network;
[0009] Figure 2A is a schematic diagram of a content editor for editing a content item;
[0010] Figure 2B is a schematic diagram of a content editor for computing variation representations from a corpus of pairs, showing clustering of variation representations;
[0011] Figure 2C is generating a schematic diagram of a tree representing the content item based on the variation representation and the graph representing the first version of the content item;
[0012] Figure 3 is a schematic diagram showing more details of the content editor;
[0013] Figure 4 It is a schematic diagram of two changing representations;
[0014] Figure 5 is a flow chart of a method for training a neural network of a content editor;
[0015] Figure 6 is a flow chart of a method performed by a content editor having a neural network that has been trained;
[0016] Figure 7 is a schematic diagram of a tree representing a first version of a content item and a second tree representing a predicted second version of the content item;
[0017] Figure 8 is a flow chart of a method performed by a generator; and
[0018] Fig. 9 An exemplary computing-based device is shown in which embodiments of a content editor are implemented.
[0019] In the drawings, like reference numerals are used to indicate like parts. DETAILED DESCRIPTION
[0020] The detailed description provided below in conjunction with the accompanying drawings is intended as a description of this example, and is not intended to represent the only form of constructing or utilizing this example. This description sets forth the functions of the example and the sequence of operations for constructing and operating the example. However, the same or equivalent functions and sequences can be implemented by different examples.
[0021] One of the great advantages of electronic storage of documents is the ability to edit them. With the widespread deployment of version control systems, these edits are a data source that can be exploited in machine learning, for example, as described in various embodiments of this document. In various examples, rules for fixing grammar, agility, and style are automatically extracted from regular edits to documents. Similarly, program source code is often changed to implement new features, follow best practices, and fix bugs. In various examples, rules are automatically extracted from common edits to source code. In a similar manner, images are often edited using image editing tools, such as cropping, contrast enhancement, and other edits. In various examples, ways to automatically edit images to achieve common style changes are described.
[0022] The technology described herein improves the functionality of a computing device because edits to content items are predicted and a user is able to input the predicted edits to the computing device, which saves the user the burden of input and provides improved human-computer interaction. The technology uses a trained neural network to predict edits to content items, and the result is a computing device that operates in an unconventional manner.
[0023] In general, the examples described herein use data related to minor edits to a content item, where a first version of the content item and a second version of the content item are substantially identical.
[0024] Figure 1 1 is a schematic diagram of a content editor 100 that communicates with one or more end-user computing devices 106, 108, 110 via a communication network 104, such as the Internet, an intranet, or any communication network. The content editor 100 is computer-implemented, and examples of computing-based devices for deploying the content editor 100 are described later in connection with Fig. 9 Given. The content editor 100 is optionally connected to or has access to a memory 102 of variation representations. The variation representation is a number, such as a vector, and it specifies a point in a multidimensional space of possible variations to a content item. Points that are close to each other in the multidimensional space represent variations of semantically similar content items. Variation representations will be described in detail later in this document.
[0025] The content editor 100 receives a content item 112 from an end-user device or other entity in a communication network 104. A content item is data, and a non-exhaustive list of examples of content items for use with the present technology is: images, videos, text, emails, blog posts, social media posts, audio files, code snippets, encyclopedia entries. The content editor uses a trained neural network to compute a predicted second version of the content item it receives. The predicted second version is stored and / or sent to the originating end-user device. It is assumed that there is a content item 112 to be edited. The trained neural network takes as input a representation of the content item 112, which includes the data of the content item and metadata about the content item. The neural network also takes as input a variation representation, which indicates at a high level how the input content item 112 is desired to be edited. The variation representation input to the neural network is computed from one or more of the following: user input, rules, clusters of variation representations, memory 102 of variation representations. More details on how to train the neural network will be given later in this document. At this point, it should be noted that by including metadata in the representation of content items, a neural network can be trained so that close points in a multi-dimensional space indicate semantically similar changes to the content items.
[0026] Because the content editor is fully automated, it greatly reduces the burden of user input at the end-user computing devices 106, 108, 110. Assume that a user of a smart phone is trying to edit a text document to put the text document in a prose style. He or she is able to achieve the desired edit without manually typing commands into the smart phone or having to issue complex voice commands or gesture commands. The user of the smart phone indicates that an edit is desired, or the smart phone predicts that an edit is desired, and the client software at the smart phone accesses the content editor 100 service via the communication network 104. A change representation indicating the desired change can be sent from the end-user device to the content editor or obtained from the change representation storage 102. The content editor 100 calculates an updated version of the text document by inputting a first version of the text document and the change representation into a neural network. The representation of the first version of the text document includes data and metadata in the text document, such as paragraph length, adjective to noun ratio, number of semicolons, or other metadata. The updated version of the text document is stored, suggested to the user, or used to replace the first version of the text document. In one example, an updated version of the text document is presented to the user at the user interface, and the user can make a choice to input the updated version of the text document to an end-user computing device or other computing device. As a result, the end-user has less complexity and can edit the text document without manual input.
[0027] Users of laptop computers trying to write software can significantly reduce the burden of manually typing code on the keyboard because the editing tool can automatically edit the code according to the change representation. When the programmer writes a code snippet, the content editor calculates the representation of the code snippet according to the metadata of the execution flow of the relevant code snippet, such as a syntax tree or a graph formed by the syntax tree. The representation of the code snippet has data from the code snippet and metadata (also known as semantic information) about the code snippet. The editing tool can find a cluster of trained similar code snippets and find one or more change representations associated with the cluster. The programmer is advised to suggest the found change representation, and / or select one of the found change representations and input it into the trained neural network together with the representation of the code snippet. Then, after applying the change of the change representation, the neural network calculates the predicted code snippet, which is the predicted second version of the code snippet. Then, the predicted code snippet is presented to the user at a graphical user interface or other user interface, so that the programmer can input the predicted code snippet into a computing device. In this way, the programmer can edit the code snippet without having to perform complex typing or other commands at the user interface.
[0028] A user of an augmented reality computing device is able to edit an image that he or she is viewing in an augmented reality environment, and is able to do so with a greatly reduced burden on user input. Without a content editor, the user of the augmented reality computing device must give gesture commands and / or voice commands, and it is difficult to obtain the fine instruction granularity required to edit an image using gestures or voice commands. Using a content editor, an image is automatically edited by a neural network based on a change representation. The neural network receives an image as input and calculates or selects a change representation of the change to be applied to the image. The change representation is calculated from one or more of the following: user input, a cluster of change representations, a database of change representations. The neural network calculates a prediction, which is a second version of the image in which the change representation has been applied to the input image. The second version of the image is stored or displayed to the user. In this way, the user can edit the image using simple or minimal user interface commands.
[0029] exist Figure 1 In an example of the invention, the content editor has a processor configured to perform one or more of: saving an updated version of a content item, replacing an input content item with the updated version, correcting the input content item using the updated version, and completing the input content item using the updated version.
[0030] exist Figure 1 In some examples, the content editor functionality is deployed as a cloud service. In other examples, the content editor functionality is distributed between the end-user device and the remote entity, or deployed at the end-user device.
[0031] exist Figure 2A In the example of , content item 112 is input to content editor 100. The content editor computes a representation of the content item and selects a variation representation. In one example, the variation representation is computed by selecting the variation representation from memory 102. A neural network at content editor 100 receives the representation of the content item and the variation representation as input and computes a prediction. The prediction is used to compute an edited version of content item 114.
[0032] exist Figure 2B In an example of , a database 200 of paired content item versions is available. Each pair of content item versions has a first version of the content item and a "ground truth" second version of the content item that has been observed after editing of the first version of the content item. For example, in the case where the content item is an email, the first version is the email automatically generated by a computer, and the second version is the same email after editing by a specific person. For example, in the case where the content item is a code snippet, the first version is the code snippet written by an individual, and the second version is the same code snippet after being edited to comply with a specific set of code design requirements. For example, in the case where the content item is an image, the first version is an image captured by a photojournalist, and the second version is the same image after editing by a newspaper graphic artist.
[0033] The neural network in the content editor 100 takes as input a pair of individual content items from the database 200 and computes a prediction as a variation representation. The neural network computes a variation representation for each pair in the database 200, and this gives a plurality of variation representations 202. The variation representations are clustered by an unsupervised clustering process 204, such as k-means or any other clustering process. Thus, a plurality of clusters 206 are computed, each cluster comprising variation representations that are close to each other in the multidimensional variation representation space.
[0034] Clusters 206 are used in various ways by content editor 100. In one example, such as Figure 2A In the case of a content item 112, the content editor 100 uses the clusters to calculate a change representation to be applied to the content item 112. The cluster is selected based on the properties of the content item 112 and the clusters 206 (e.g., the content editor can be designed to select the most common change cluster). The content editor 100 then suggests to the user that the change representation of the selected cluster be applied. In another example, features of each pair in the cluster are calculated, and rules are automatically generated, for example, if the content item exhibits features of a specified cluster, the change representation of the specified cluster is automatically applied. In another example, a change type corresponding to the change representation of the cluster is identified, and the change type is provided to the user in a menu of editing options provided by the content editor or used to automatically extract rules.
[0035] In another example, an author can view changes made regularly by content editors (identified as common change clusters) to avoid such problems in the future. In another example, a software development team can view changes commonly made to fix errors illustrated by the clusters and thus improve their engineering practices or develop techniques to automatically avoid such problems.
[0036] Figure 2C As shown, a token T1 . . . T μ A graphical representation 210 of a first version b of a content item such as a code snippet of a sequence. Figure 2C Also shown in FIG. 2 is a schematic variation representation 212, which is a vector shown as a rectangle, wherein the vector is represented as R T The graphical representation 210 and the vector 212 are input to a neural network, which generates a prediction 214 (denoted as A), which in this case is a tree. The tree is a syntax tree for the code snippet, which is a predicted second version of the first version of the code snippet. The syntax tree is converted into a code snippet, which is provided to the programmer as a suggestion or used to replace or correct the original code snippet.
[0037] The graphical representation is computed by the end-user device or by the content editor or any other entity. The graphical representation of the code snippet is computed by computing a syntax tree for the code snippet and adding edges to the syntax tree according to metadata about the code snippet. The code snippet includes a sequence of tokens, which are variable, operator, method names. The sequence of tokens is parsed to generate a syntax tree. Additional edges are added to the tree to produce a graph. The additional edges indicate variable flows or other metadata about the code snippet.
[0038] In one example, two different types of extra edges are added to a syntax tree. The first type of extra edge is a syntactic edge, and the second type of extra edge is a semantic edge. Syntactic edges include next token edges and sub-edges, which are used to represent an abstract syntax tree of a code snippet. Syntactic edges also include a last lexical use edge that connects a token to the last time the token appeared in the code snippet. Semantic edges include a last use edge that connects a variable to the last time the variable may have been used in program execution, a last write edge that connects a variable to the last time the variable was written in program execution, and a computed from edge that connects a variable to the value of the variable calculated from it in program execution. Other types of semantic edges are also possible, which can capture more information about program execution, such as aliasing between different program variables, data dependencies, etc.
[0039] Figure 33 is a more detailed schematic diagram of a content editor 100. The content editor 100 includes a neural network encoder 300 and a neural network decoder 306 with an associated generator 304. The neural network decoder computes a predicted second version 308 of the content item. The content editor 100 is trained using paired “ground truth” content item versions. The paired content item versions are as described with respect to Figure 2B The training algorithm 310 is used to train the content editor 100 such as a neural network back propagation or a variation of a neural network back propagation. The objective function 312 used by the training algorithm 310 is expressed as:
[0040] Given an observed variation consisting of a pair of content item versions (b, a), the generator function is trained such that:
[0041] g(b,f(b,a))=a
[0042] In words, the generator function computes the second version of the content item with the first version of the content item and a vector representing the change from the first version of the content item to the second version of the content item as arguments. The symbol g is Figure 3 The symbol f is a function implemented by the neural network decoder 306. Figure 3 The function implemented by the neural network encoder 300.
[0043] Where the content item is text, in some examples the neural network encoder is a recurrent neural network and the neural network decoder is a recurrent neural network.
[0044] Where the content item is an image, in some examples the neural network encoder is a convolutional neural network and the neural network decoder is a deconvolutional neural network.
[0045] Where the content item is software including code snippets and the representation of the content item input to the encoder is a syntax tree, in some examples the neural network encoder is a recursive neural network of a type suitable for taking a tree as input and predicting a tree as output.
[0046] Where the content item is software including code snippets and the representation of the content item input to the encoder is a graph, in one example the neural network encoder is a graph neural network of a type suitable for taking a graph as input and predicting a tree as output.
[0047] Figure 4 is a schematic diagram of a change representation space 400 having two dimensions (denoted as K=2). The change from B to A is represented by point 402, and the change from B to A is represented by point 403. 1 To A 1The change in is represented by point 404. The distance between points 402 and 404 is the change in (B, A) and (B 1 , A 1 ). In practice, the variation representation space has many dimensions, although Figure 4 Two are shown for ease of explanation.
[0048] Reference now Figure 3 and 5 A method of training a content editor is described. The neural network encoder 300 takes as input 500 a representation of a content item pair and computes 502 a variation representation, which is a vector in a multidimensional space. The first version of the content item from the content item pair and the variation representation are input 504 to the neural network decoder 306. The neural network decoder 306, together with the generator 304, computes a second version 308 of a prediction 506 of the content item. The predicted second version 308 of the content item is compared 508 to the actual second version of the content item from the database 200 by the training algorithm 310. The training algorithm updates (using back propagation 510 or another update process) the neural network encoder 300 and the neural network decoder 306 based on the comparison and the objective function 312. If convergence 512 is reached, the training process ends and the neural network parameters are stored 514. If convergence is not reached, the process repeats from operation 500.
[0049] Reference now Figure 6 A method performed by a content editor after training is described. A content item is received 600. The content item is text or a code snippet or an image or another type of content item. Optionally, a representation of the content item is calculated 602. In an example where the content item is text or a code snippet, a syntax tree or a graph is calculated as the representation.
[0050] The content editor selects 604 a variation representation. It does this using one or more of: a previous content item version pair, user input, a cluster 616 of variation representations of rules. The content editor inputs 606 the selected variation representation and the representation of the content item (or the content item itself) into a trained neural network ( Figure 3 The neural network encoder predicts 608 the edited content item and saves the prediction 610, or suggests the predicted content item to the user 612, or replaces 614 the received content item with the predicted content item.
[0051] Figure 7A schematic diagram of a tree 700 representing a first version of a content item and a second tree 704 representing a predicted second version of the content item is shown. The content item is a code snippet "y=x+z", and the tree 700 has a leaf for each tag y, =, x, +, z. Tags y and = are directly connected to the root node of the tree 700, while tags x, +, z are connected to the root node via intermediate nodes to reflect the addition operation expressed in the code snippet. Associated with each leaf node is a numerical value, such as a vector, represented by v, which is indexed by an index of a node number. Information from the leaf node numerical values is propagated back to the tree and stored at a high-level node of the tree. In this way, the parent node has information about the child branches of the tree extending from it. A similar tree 702 representing the second version of the code snippet is not yet known because it needs to be predicted by a content editor.
[0052] Figure 6 The process is used when the representation at operation 602 is a tree such as tree 700. The result at operation 608 is a prediction tree 704 output by the neural network decoder. In this example, the prediction tree 704 has eight nodes labeled U1 to U8. As now referred to Figure 8 As described, a generator, such as generator 304 , generates code snippets from prediction tree 704 . Figure 8 The example enables the generator to copy, where appropriate, sub-branches of the tree 700 representing the first version of the content item into the prediction tree 704. As a result, the quality of the predicted second version of the content item is improved.
[0053] The generator may access a tree representation 800 of the first content item, wherein information about individual child branches of the tree is captured at parent nodes in the tree.The content editor computes 802 a tree representation of a predicted second version of the content item.
[0054] Starting from the root node of the tree representation of the predicted second version of the content item, the generator chooses 804: to produce a known tag or node; or to copy an item of the tree representing the first content. More details on how this selection is made will be provided later.
[0055] If the generator chooses to copy, it copies 808 the child branches of the corresponding node in the tree of the first content version. If the generator chooses to generate, it generates 806 the known tags by selecting tags from a specified plurality of possible tags randomly or in any other manner.
[0056] The generator checks 810 whether there is a node in the tree of the predicted second version of the content item that has not yet been processed. If so, it moves 812 to the next node in depth-first order and repeats from operation 804. Figure 8If there are no more nodes, the process ends 814 and the generated token sequence is stored and output as a predicted second version of the content item.
[0057] Detailed examples using mathematical notation are now given.
[0058] Formally, suppose that the original version of some input χ - and an edited version of the input χ + , where χ - and χ + Mostly the same. Given an edited dataset The goal is to learn the representation function f Δ , the function f Δ From χ - To + The edit operations are mapped to real-valued change representations f Δ The desired quality is that the computed change representation has the following property: that is, semantically similar edits are It has a close-up representation.
[0059] A good variation representation function f Δ Compute the change representation, which is then used to convert the same semantic edit f Δ (·,·) is applied to the new input. This is achieved by learning a neural network for the content editor, called the neural editor α, which takes the change representation and the new input and applies the edits to the new input. For example, given the change representation f Δ (x-, x+) and new input χ′ - , semantically similar edits can be applied as χ′ + =α(χ′ - , f Δ (x - , x + )).
[0060] The technique described in this paper casts the variational representation problem as a task similar to autoencoding, aiming to obtain Δ (x - , x + ) and the original version x - In the case of + The reconstruction error of α is minimized. By limiting f Δ ' output capacity, this technique introduces a "bottleneck", forcing the entire framework to not simply Δ (·,x +) is considered as the encoder in the traditional autoencoder setting and α is considered as the decoder in the traditional autoencoder setting; instead, optimizing for the objective requires α to be - Get information about x + as much information as possible.
[0061] Formally, given the probability editor function P α (such as neural networks) and datasets The technique aims to minimize the maximum possible loss:
[0062]
[0063] Expressed in words, the maximum likelihood loss is equal to the negative of the inverse of the number of training pairs multiplied by the sum over the training pairs of the logarithm of predictions computed by the neural network given the training pairs and the variation representation.
[0064] As will now be described by way of non-limiting example, a neural network is used to implement P α and f Δ .
[0065] Editor Function Neural Network P α With the initial version x - and changes in f Δ (x - , x + ) as input, and learn to convert x - Transform to x + . Therefore, an encoder-decoder architecture with the ability to copy from the input is suitable. In one example, a sequence-to-sequence neural network model is used for text and a graph-to-tree model is used for source code, whose known semantics are exploited on both the encoder and decoder sides. As discussed earlier in this document, convolutional models may better meet the needs of other categories of editing (e.g., image processing).
[0066] Sequence to Sequence Neural Editor
[0067] In one example, a sequence-to-sequence neural network model with attention (on x - The architecture uses a bidirectional long short-term memory (LSTM) in the encoder and a token-level copy mechanism to copy the tokens directly into the decoded sequence. Compared to previous sequence-to-sequence models, the decoder represents the change f Δ (·,·) as additional input. Specifically, the decoder LSTM receives as input the concatenation of the representation of the previous token and the variation representation. This allows the LSTM decoder to take the variation representation into account when generating the edited sequence.
[0068] Graph to Tree Neural Editor
[0069] In one example, a graph-to-tree neural network is used that is able to utilize x - and x + To this end, a graph-based encoder is combined with a tree-based decoder. First, let T(x) denote a tree representation of an element, e.g., a syntax tree for a fragment of source code. The tree is expanded to a graph by encoding additional relations (e.g., "next token" relations between terminal nodes, occurrences of the same token in several locations, etc.). This yields x - The graph-based representation of is denoted as G(x). In order to - ) is encoded into a vector representation using a gated graph neural network (GGNN). Similar to a recurrent neural network for sequences, a GGNN computes representations for each node in the graph. These are used in a set of attention mechanisms within the decoder, which are explained below. In addition, the representations of each node in the graph are used to obtain the full input x by computing their weighted average in the following way - Representation: Calculate the score of each node, normalize the score using softmax, and use the resulting value as the weight.
[0070] Specifically, the generation of the tree T is represented as a sequence of actions There are two possible types of actions: ApplyRule action, which uses the grammar rules to expand the current non-terminal; and terminal action, which generates a terminal token from the vocabulary. The probability of taking an action is modeled as p(α t |α <t , x - ), where x - is the original version of the input, α <t is the partial tree that has been generated so far in the generation process. In practice, for the partial tree α <t The dependencies of are modeled using LSTM units for maintaining state during action sequence generation, as well as a “parent feeding” technique where the state of the LSTM is used as an additional input when generating parent nodes. In addition, the model uses an attention mechanism that allows the decoder to attend to the input sequence. The decoder considers G(x - ) and the graph level representation of the computation is used as the condition, and the node representation in this graph is used as the memory of the attention mechanism. The variation representation f Δ (·) is used as additional input.
[0071] Based on the observation that edits to source code often manipulate syntax trees by moving expressions around, some but not all embodiments of the technique add the convenience of copying entire subtrees from the input. To this end, a decoder action SubTreeCopy is added. This operation copies the entire subtree pointed to.
[0072] Adding the SubTreeCopy action means that the target tree now has many correct generation sequences. This problem cannot be easily solved by marginalizing out all correct choices at each generation step, by normalizing the probability distribution of allowed actions to summarize actions with the same effect. The reason is that the lengths of action sequences representing different choices may also be different. This problem can be solved by choosing a correct generation sequence (greedily choosing SubTreeCopy when possible) and by changing the training objective so that correct decoder action choices are not penalized. To achieve this, a "many-hot" encoding of the correct choices is used. At test time, beam search is used, and merging beams leads to the same result.
[0073] Change indication
[0074] In order to compute a useful representation of the variation, the model needs to focus on x - With x + The risk is that f Δ Degenerates to x + To avoid this, we generate the variation representation into a low-dimensional space that acts as a bottleneck. To limit f Δ The ability of the result, and only for the - Refactoring x + Encode the required information.
[0075] Edited Sequence Code
[0076] Given x - (resp.x+) as a marker sequence The alignment of the markers in the two sequences is calculated using a difference algorithm. Additional symbols For filling, + for adding, - for deleting, for substitution, = for invariant marking, to generate representation x - and x + A single sequence of , such as the following sequence.
[0077]
[0078] Here, mark Deleted during editing, use To replace, then add Etc. Embedding each element in the sequence with three entries separately and concatenating their representations can be fed into a sequence encoder whose final state is the desired variation representation.
[0079] Edited graphics code
[0080] For the graph-to-tree neural editor, change x - and x + Represented as a syntax tree T(x - ) and T(x + ). These trees are combined into a graph representation G(x) by merging the two trees into a single graph using the “Child” and “NextToken” edges. - →x + ). To connect two trees, the same alignment as in the sequence case is computed, identical leaves are connected together, and each replaced leaf is connected to its replacement. This information is propagated in the tree, i.e., two internal nodes are connected by a "=" edge if all their descendants are connected by a "=" edge.
[0081] Use the same "+" / "-" / / "=" labels are used for the initial node representation, which is calculated as the concatenation of the string label (i.e., token or non-terminal name) and the embedding of the label. To obtain the variational representation, a graph neural network (GNN) is used that is unrolled for a fixed number of time steps, and the weighted averaging strategy described above is used again.
[0082] Fig. 9 Various components of an exemplary computing-based device 918 are shown, which are implemented as any form of computing and / or electronic device and in which embodiments of the content editor are implemented in some examples.
[0083] The computing-based device 918 includes one or more processors 900, which are microprocessors, controllers, or any other suitable type of processors, for processing computer executable instructions to control the operation of the device to edit content and / or predict a second version of a content item based on a first version of the content item. In some examples, such as when using a system-on-chip architecture, the processor 900 includes one or more fixed function blocks (also referred to as accelerators) that implement a portion of the method of training a content editor and / or implement it in hardware (rather than software or firmware) using a trained content editor. Platform software including an operating system 922 or any other suitable platform software is provided at the computing-based device to enable application software 924 to execute on the device. A neural network 926 and optional training logic 921 for training the neural network are stored at the computing-based device 918. A data memory 928 stores content items, training parameters, predicted content items, and other data.
[0084] Computer executable instructions are provided using any computer readable medium accessible by computing-based device 918. Computer readable media include, for example, computer storage media such as memory 912, and communication media. Computer storage media such as memory 912 include volatile and non-volatile removable and non-removable media implemented by any method or technology for storing information such as computer readable instructions, data structures, program modules, etc. Computer storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electronic erasable programmable read-only memory (EEPROM), flash memory or other storage technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic tape, cassette, magnetic tape, disk storage or other magnetic storage device, or any other non-transmission medium for storing information accessed by a computing device. In contrast, communication media embody computer readable instructions, data structures, program modules, etc. in modulated data signals such as carrier waves or other transmission mechanisms. As defined herein, computer storage media do not include communication media. Therefore, computer storage media should not be interpreted as propagating signals themselves. Although computer storage media (memory 912) is shown as being within computing-based device 918, it will be appreciated that in some examples the storage is remotely distributed or located and may be accessed via a network or other communications link (eg, using communications interface 913).
[0085] The device 918 based on calculation also includes an output interface 910, which is arranged to output display information to a display device 920, which can be separated from or integrated into the device 918 based on calculation. Display information can provide a graphical user interface. The input interface 902 is arranged to receive and process input from one or more devices (such as a game controller 904, a keyboard 906 or a mouse 908 or other user input devices). In some examples, the user input device detects voice input, user gestures or other user actions, and provides a natural user interface (NUI). The user input can be used to select a change representation, view clusters, and select the content item version proposed. In one embodiment, if the display device 920 is a touch-sensitive display device, it also serves as a user input device. In some examples, the output interface 910 outputs data to other devices except the display device.
[0086] Any of the input interface 902, the output interface 910, the display device 920, and the user input device may include NUI technology that enables a user to interact with a computing-based device in a natural way without being subject to artificial constraints imposed by input devices such as a mouse, keyboard, remote control, etc. Examples of NUI technology provided in some examples include, but are not limited to, technologies that rely on sound and / or voice recognition, touch and / or stylus recognition (touch-sensitive displays), gesture recognition on and adjacent to the screen, air gestures, head and eye tracking, sound and voice, vision, touch, gestures, and machine intelligence. Other examples of NUI technology used in some examples include intent and target understanding systems, motion gesture detection systems using depth cameras (such as stereo camera systems, infrared camera systems, red, green, blue (RGB) camera systems, and combinations thereof), motion gesture detection using accelerometers / gyroscopes, facial recognition, three-dimensional (3D) displays, head, eye, and gaze tracking, immersive augmented reality and virtual reality systems, and technologies for sensing brain activity using electric field sensing electrodes (electroencephalograms (EEGs) and related methods).
[0087] As an alternative or in addition to other examples described herein, examples include any combination of the following:
[0088] Clause 1. An editing tool comprising:
[0089] a memory storing a neural network that has been trained to compute a variation representation from pairs, each pair comprising a representation of a first version of a content item and a second version of the content item, and for each of the variation representations, the neural network has been trained to predict an updated content item based on the variation representation and the first version of the content item;
[0090] A processor is configured to receive an input content item and to compute an updated version of the input content item based on the variation representation using the neural network.
[0091] 2. An editing tool according to claim 1, wherein the processor is configured to output suggestions to a user, the suggestions comprising the updated version of the input content item, and wherein the processor is configured to perform one or more of the following: saving the updated version, replacing the input content item with the updated version, correcting the input content item using the updated version, and completing the input content item using the updated version.
[0092] 3. An editing tool according to any preceding clause, wherein the variation representation is a vector representing a point in a multi-dimensional space in which semantically similar variations are close to each other.
[0093] 4. An editing tool according to any preceding clause, wherein the processor is configured to calculate a change representation to be used in calculating the updated version of the input content item.
[0094] 5. An editing tool according to clause 4, wherein the processor is configured to calculate the variation representation to be used based on one or more of: a rule, a user input, each pair of content items comprising a first version of the content item and a second version of the content item.
[0095] 6. An editing tool according to any preceding clause, wherein the content item is text or software, and the neural network is configured to receive as input a representation of the content item as a tree or a graph.
[0096] 7. An editing tool according to any of the preceding clauses, wherein the processor is configured to use the neural network to calculate a variation representation for each of a plurality of the pairs, cluster the variation representations, and use the clusters for one or more of: making suggestions to a user, creating rules, identifying variation representations to be applied to content items.
[0097] 8. An editing tool according to any preceding clause, wherein the content item is software and the neural network is configured to receive as input a representation of the content item, the representation of the content item being a graph formed by parsing a sequence of tokens of the software to form a tree and adding edges to the tree based on a program analysis of the software.
[0098] 9. An editing tool according to any preceding clause, wherein the content item is software, and wherein the processor is configured to calculate a graphical representation of the content item by parsing tags of the software to form a tree and adding edges to the tree based on program analysis results of the software.
[0099] 10. An editing tool according to claim 9, wherein the processor is configured to add edges to the tree based on one or more of: connecting a variable to the last time the variable was used in program execution of the software, connecting a variable to the last time the variable was written in program execution of the software, connecting a variable to a value from which the variable was calculated in program execution of the software, aliasing in program execution of the software, pointing analysis in program execution of the software, and path conditions in program analysis of the software.
[0100] 11. An editing tool according to any preceding clause, wherein the neural network comprises a recursive neural network and the content item is text or software.
[0101] 12. An editing tool according to any preceding clause, wherein the neural network comprises a convolutional neural network and a deconvolutional neural network, and the content item is an image.
[0102] 13. An editing tool according to any preceding clause, wherein the content item is text or software, and the neural network comprises a tree-to-tree neural network or a graph-to-tree neural network.
[0103] 14. An editing tool according to any of the preceding clauses, wherein the editing tool includes a generator configured to generate the second version of the content item from the first version of the content item by selecting whether to copy a sub-branch of a tree representing the first version of the content item to the generated second version of the content item.
[0104] 15. A computer-implemented method of editing a content item, comprising:
[0105] storing in a memory a neural network that has been trained to compute a variation representation from a plurality of pairs, each pair comprising a representation of a first version of a content item and a second version of the content item, and for each of the variation representations, the neural network has been trained to predict an updated content item based on the variation representation and the first version of the content item;
[0106] receiving an input content item at a processor;
[0107] The processor is used to compute an updated version of the input content item based on the change representation using the neural network.
[0108] Clause 16. The method of Clause 15, comprising computing a change representation to be used in computing the updated version of the input content item.
[0109] Clause 17. The method according to clause 14 comprises calculating the variation representation to be used based on one or more of: a rule, a user input, a similarity of the input content item to one of a plurality of clusters of pairs of content items, each pair of content items comprising a first version of the content item and a second version of the content item.
[0110] Clause 18. The method of clause 15, comprising receiving as input a representation of the content item, the representation of the content item being a graph formed by parsing a sequence of tokens of the software to form a tree and adding edges to the tree based on program analysis of the software.
[0111] Clause 19. The method of Clause 15, comprising generating a second version of the content item from the first version of the content item by selecting whether to copy a child branch of a tree representing the first version of the content item to the generated second version of the content item.
[0112] Clause 20. An editing tool comprising:
[0113] a memory storing a neural network that has been trained to compute a variation representation from a plurality of pairs, each pair comprising a representation of a first version of a code snippet and a second version of the code snippet, and for each of the variation representations, the neural network has been trained to predict an updated code snippet based on the variation representation and the first version of the code snippet;
[0114] a processor configured to receive an input code snippet and compute an updated version of the input code snippet based on the variation representation using the neural network;
[0115] wherein the processor computes a graphical representation of the input code snippet for input to the neural network.
[0116] The term "computer" or "computing-based device" is used herein to refer to any device having processing capabilities that enable it to execute instructions. Those skilled in the art will appreciate that such processing capabilities are incorporated into many different devices, so the terms "computer" and "computing-based device" include personal computers (PCs), servers, mobile phones (including smartphones), tablet computers, set-top boxes, media players, game consoles, personal digital assistants, wearable computers, and many other devices, respectively.
[0117] In some examples, the methods described herein are performed by software in a machine-readable form on a tangible storage medium, such as in the form of a computer program including computer program code means, which, when run on a computer, is adapted to perform all operations of one or more methods described herein, and wherein the computer program may be embodied on a computer-readable medium. The software is suitable for execution on a parallel processor or a serial processor, such that the method operations may be performed in any suitable order or simultaneously.
[0118] This recognizes that software is a valuable, separately tradable commodity. It is intended to cover software that runs on or controls "virtual" or standard hardware to perform a desired function. It is also intended to cover software that "describes" or defines the configuration of hardware, such as HDL (Hardware Description Language) software, which is used to design silicon chips or to configure general-purpose programmable chips to perform a desired function.
[0119] Those skilled in the art will recognize that the storage devices for storing program instructions are optionally distributed over the entire network. For example, a remote computer can store examples of processes described as software. A local or terminal computer can access the remote computer and download part or all of the software to run the program. Alternatively, a local computer can download software fragments as needed, or execute some software instructions at a local terminal and execute some software instructions at a remote computer (or computer network). Those skilled in the art will also recognize that, by utilizing conventional techniques known to those skilled in the art, all or part of the software instructions can be executed by special purpose circuits such as digital signal processors (DSPs), programmable logic arrays, etc.
[0120] As will be apparent to one skilled in the art, any range or device value given herein may be extended or altered without losing the effect sought.
[0121] Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
[0122] It should be understood that the above benefits and advantages may relate to one embodiment, or may relate to several embodiments. The embodiments are not limited to embodiments that solve any or all of the problems described, nor are they limited to embodiments that have any or all of the advantages and benefits described. It should also be understood that reference to "an" item refers to one or more of these items.
[0123] The operations of the methods described herein may be performed in any suitable order, or simultaneously where appropriate. In addition, individual blocks may be deleted from any method without departing from the scope of the subject matter described herein. Aspects of any of the above examples may be combined with aspects of any other example described to form other examples without losing the effect sought.
[0124] The term "comprising" is used herein to mean including the identified method blocks or elements, but such blocks or elements do not comprise an exclusive list and the method or apparatus may contain additional blocks or elements.
[0125] It should be understood that the above description is given by way of example only, and various modifications may be made by those skilled in the art. The above description, examples and data provide a complete description of the structure and use of exemplary embodiments. Although various embodiments have been described above with a certain degree of particularity or with reference to one or more individual embodiments, those skilled in the art may make various changes to the disclosed embodiments without departing from the scope of this specification.
Claims
1. An editing tool comprising: a memory storing a neural network that has been trained to compute a plurality of variation representations from a plurality of pairs of content items; as well as The processor is configured as: Receive input code snippet; generating a representation of the code snippet based on metadata of an execution flow for the code snippet; identifying clusters of similar code snippets that have been trained on the neural network; identifying one or more variation representations associated with the cluster; selecting a variation representation from the one or more variation representations; Using the neural network, an updated version of the input code snippet is calculated based on the selected comparison of the variation representations.
2. The editing tool of claim 1, wherein the processor is further configured to: outputting a suggestion to a user, the suggestion comprising the updated version of the input code snippet; and One or more of: saving the updated version, replacing the input code snippet with the updated version, correcting the input code snippet using the updated version, and completing the input code snippet using the updated version. 3 . The editing tool of claim 1 , wherein the processor is further configured to automatically replace the input code snippet with the updated version of the code snippet. 4 . The editing tool of claim 1 , wherein the processor is further configured to compute a change representation to be used in computing the updated version of the input code snippet.
5. An editing tool according to claim 4, wherein the processor is further configured to calculate the variation representation to be used based on one or more of a rule and a user input, wherein each pair of content items includes a first version of a code snippet and a second version of the code snippet.
6. The editing tool of claim 1, wherein the input code snippet is an earlier version of the updated version of the code snippet.
7. An editing tool according to claim 1, wherein the processor is further configured to: use the neural network to calculate a change representation for each pair of content items in the multiple pairs of content items, cluster the change representations, and use the clusters for one or more of the following: making suggestions to the user, creating rules, and identifying change representations to be applied to the code snippet.
8. The editing tool of claim 1 , wherein the code snippet is software, and the neural network is configured to receive as input a representation of the code snippet, the representation of the code snippet being a graph formed by parsing a token sequence of the software to form a tree and adding edges to the tree based on program analysis of the software.
9. The editing tool of claim 1 , wherein the code snippet is software, and wherein the processor is further configured to calculate a graphical representation of the code snippet by parsing tokens of the software to form a tree and adding edges to the tree according to program analysis results of the software.
10. An editing tool according to claim 9, wherein the processor is further configured to add edges to the tree based on one or more of: connecting a variable to the last time the variable was used in program execution of the software, connecting a variable to the last time the variable was written in program execution of the software, connecting a variable to a value from which the variable was calculated in program execution of the software, aliasing in program execution of the software, pointing analysis in program execution of the software, and path conditions in program analysis of the software.
11. The editing tool of claim 1 , wherein the neural network comprises a recursive neural network, and the code snippet is text or software.
12. The editing tool of claim 1, wherein the neural network comprises a convolutional neural network and a deconvolutional neural network, and the code snippet is an image.
13. The editing tool of claim 1, wherein the code snippet is text or software, and the neural network comprises a tree-to-tree neural network or a graph-to-tree neural network.
14. An editing tool according to claim 1, wherein the editing tool includes a generator, which is configured to generate the second version of the code snippet from the first version of the code snippet by selecting whether to copy a sub-branch of a tree representing the first version of the code snippet to the generated second version of the code snippet.
15. A computer-implemented method of editing a content item, comprising: storing in a memory a neural network that has been trained to compute a plurality of variation representations from a plurality of pairs of content items; receiving an input code snippet at a processor; generating a representation of the code snippet based on metadata of an execution flow for the code snippet; identifying clusters of similar code snippets that have been trained on the neural network; identifying one or more variation representations associated with the cluster; selecting a variation representation from the one or more variation representations based on the input code snippet; using the neural network, computing an updated version of the input code snippet based on the selected comparison of the variation representations; as well as The processor is used to compute an updated version of the input code snippet using the neural network and based on a comparison of the selected variation representations.
16. The method according to claim 15, further comprising: A change representation is computed to be used in computing the updated version of the input code snippet.
17. The method according to claim 16, further comprising: The variation representation to be used is calculated based on one or more of: a rule, a user input, and a similarity of the input code snippet to one of a plurality of clusters of a plurality of pairs of content items, wherein each pair of content items includes a first version of the code snippet and a second version of the code snippet.
18. The method according to claim 15, further comprising: A representation of the code snippet is received as input, the representation of the code snippet being a graph formed by parsing a token sequence of software to form a tree and adding edges to the tree based on program analysis of the software.
19. The method according to claim 15, further comprising: The second version of the code snippet is generated from the first version of the code snippet by selecting whether to copy a child branch of a tree representing the first version of the code snippet into the generated second version of the code snippet.
20. An editing tool comprising: a memory storing a neural network that has been trained to compute a plurality of variation representations from a plurality of pairs of content items; as well as The processor is configured as: Receive input code snippet; generating a representation of the code snippet based on metadata of an execution flow for the code snippet; identifying clusters of similar code snippets that have been trained on the neural network; identifying one or more variation representations associated with the cluster; selecting a variation representation from the one or more variation representations based on the input code snippet; using the neural network, computing an updated version of the input code snippet based on the selected comparison of the variation representations; wherein the processor computes a graphical representation of the input code snippet for input to the neural network.
