Auxiliary guide to generate candidate paths in image for user tracking input

Abstract

Embodiments of the present disclosure relate to an auxiliary guide for generating candidate paths in an image for user tracking input. The present disclosure relates to systems, methods, and non-transitory computer-readable media for providing auxiliary guidance for path tracking of raster images. Specifically, in one or more implementations, the disclosed system determines a set of outer frames corresponding to boundaries of a set of segments within a raster image. The disclosed system selects, in response to a client device input indicating a point (s) located within a threshold distance of the outer frame, an outer frame corresponding to a segment from a set of outer frames. The disclosed system provides a highlighted indication of an outer frame corresponding to a segment for a display within a graphical user interface of a client device. The disclosed system generates a vector path within a vector image based on an outer frame corresponding to a segment in response to a selection of the outer frame via a graphical user interface.

Description

BACKGROUND

[0001] Advances in computing devices and computer design applications have led to the innovative development of computer image design and editing software. For example, certain computer design applications allow for the use of vector paths, such as Bezier curves, to edit and manipulate digital images to generate various graphical representations with lossless scaling. However, the conversion of content from a raster image to a vector image, including the conversion of image data to mathematical outlines, is a complex process that is often dependent on the resolution quality of the raster image. Unfortunately, current vector-based applications are limited in their ability to interpret the complexity of pixel-based information when converting the details of a raster image to vector content, and often involve the manual conversion of raster images to vector content. Existing image editing systems have multiple shortcomings in terms of flexibility and operational efficiency when tracing, editing, and generating vector paths from raster images. SUMMARY

[0002] One or more embodiments provide benefits and / or address one or more of the above-identified or other problems in the field of systems, methods, and non-transitory computer-readable storage media that provide a graphical user interface and process tailored to simplify object tracing within a vector-based design application via an assistive guide. Specifically, the disclosed system utilizes deep learning to perform semantic analysis of a raster image and performs segmentation to generate object masks for segments within the raster image. The disclosed system utilizes the image masks of the segments to generate outlines for the segments. Further, the disclosed system employs an outline matching model with hit detection to match client device input to generated outlines corresponding to user intent. Further, the disclosed system provides suggestions of possible matching outlines within the graphical user interface as an assistive guide while adapting the assistive guide in real-time to changes in client device input. The disclosed system also generates a vector path in response to selecting an outline from the suggested outlines. BRIEF DESCRIPTION OF DRAWINGS

[0003] The disclosure will describe one or more example implementations of systems and methods with additional specificity and detail through reference of the drawings. The following paragraphs briefly describe a few of these drawings in which:

[0004] Figure 1 A schematic diagram illustrating an example environment of an assistive vector tracing system in accordance with one or more embodiments is shown;

[0005] Figure 2 An example overview of generating a vector path based on a raster image in accordance with one or more embodiments is shown;

[0006] Figure 3A An example of utilizing an object detection model to generate an object mask in accordance with one or more embodiments is shown;

[0007] Figure 3B An example of generating a segment mask with an object segmentation model is shown in accordance with one or more embodiments;

[0008] Figure 4 An example of generating a bounding box corresponding to a segment mask is shown in accordance with one or more embodiments;

[0009] Figure 5 An example of selecting a potential bounding box for a segment with a bounding box matching model with hit detection is shown in accordance with one or more embodiments;

[0010] Figures 6A to 6C An example of generating and providing a guidance hint for a bounding box associated with a client device input within a graphical user interface is shown in accordance with one or more embodiments;

[0011] Figures 7A to 7C An example of providing adaptive feedback and auto-completing a vector path within a graphical user interface is shown in accordance with one or more embodiments;

[0012] Figure 8 A diagram showing an example architecture of an assisted vector tracing system in accordance with one or more implementations;

[0013] Figure 9 A flowchart showing a series of actions for generating a guidance hint for creating a vector path in accordance with one or more embodiments; and

[0014] Figure 10 A block diagram showing an example computing device in accordance with one or more embodiments. DETAILED DESCRIPTION

[0015] The present disclosure describes one or more embodiments of an assisted vector tracing system that provides a graphical user interface that is tailored to simplify object tracing within a vector-based design application via guidance hints. In particular, in one or more implementations, the assisted vector tracing system utilizes deep learning to perform semantic analysis of a raster image and performs segmentation to generate an object mask for a segment within the raster image. Further, the assisted vector tracing system generates a bounding box with the image mask of the segment. Moreover, the assisted vector tracing system employs a bounding box matching model with hit detection to match a client device input with a generated bounding box that corresponds to a user’s intent. Further, the assisted vector tracing system provides a suggested bounding box that is a possible match to the client device input as a guidance hint within a graphical user interface while adapting the guidance hint in real-time to changes in the client device input. The assisted vector tracing system also generates a vector path in response to a selection of a bounding box from the suggested bounding boxes.

[0016] More specifically, in one or more implementations, the auxiliary vector tracing system determines a set of bounding boxes corresponding to a set of segments (e.g., segment objects) within a raster image. For example, the auxiliary vector tracing system employs a combination of an object detection model (e.g., a deep neural network image segmentation model) and a segmentation model to determine detailed segments within a source raster image. In particular, in some embodiments, the auxiliary vector tracing system utilizes the object detection model to generate object masks that bound semantic objects within the image. Further, in certain embodiments, the auxiliary vector tracing system utilizes the segmentation model to generate a set of segment masks for more detailed segmentation of one or more semantic objects within the image based on the object masks.

[0017] In certain embodiments, the auxiliary vector tracing system converts the one or more segment masks into bounding boxes. For example, to generate the bounding boxes, the auxiliary vector tracing system applies a thresholding technique to the segment masks to generate binary masks. Further, the auxiliary vector tracing system extracts contours that partition the boundaries of the segments within the segment masks. In some cases, the auxiliary vector tracing system approximates the contours to reduce the number of anchor points. In turn, in one or more embodiments, the auxiliary vector tracing system generates the bounding boxes by creating polylines corresponding to the contours of the segment masks.

[0018] As described above, in one or more embodiments, the auxiliary vector tracing system selects one or more bounding boxes (e.g., the set of bounding boxes as candidate bounding boxes) corresponding to client device inputs. For example, the auxiliary vector tracing system receives client device inputs that indicate one or more points corresponding to a path input (e.g., a partial Bezier curve, an input spline, a point, a line primitive) in the input image. In some cases, the auxiliary vector tracing system filters the bounding boxes corresponding to the segment masks with hit detection involving a bounding box of the client device inputs and the bounding boxes. In certain embodiments, the auxiliary vector tracing system performs vector component detection by flattening the client device inputs and the filtered set of bounding boxes into corresponding line primitives. Based on a comparison of the line primitives of the path input and the restricted set of bounding boxes, the auxiliary vector tracing system selects one or more bounding boxes that match the path input as potential bounding boxes.

[0019] Further, in certain embodiments, the auxiliary vector tracing system automatically completes the client device inputs by generating a vector path (e.g., a Bezier curve) that completes the bounding boxes of the segments within the input image. For example, the auxiliary vector tracing system highlights the potential bounding boxes on the client device to visually indicate the matching bounding boxes corresponding to the path input. Upon receiving a client device input that selects one of the highlighted bounding boxes, the auxiliary vector tracing system automatically completes the vector path by generating a Bezier curve corresponding to the geometry of the selected highlighted bounding box and merging (or otherwise matching) the client device input.

[0020] As mentioned above, the auxiliary vector tracing system overcomes the shortcomings of conventional systems that provide tools for vectorizing raster images. Specifically, conventional systems have many technical deficiencies in terms of flexibility and computational efficiency when tracing raster images to generate vector outlines. For example, many existing design systems rigidly provide image tracing using conventional drawing tools that require users to manually perform precise drawing of each desired path based on the source image. Moreover, existing design systems do not provide real-time contextual feedback to client devices when generating Bezier curves to trace complex / detailed images, which often results in users needing to perform multiple tracing iterations or attempts to accurately generate vector paths and / or a high degree of proficiency with the tools.

[0021] Furthermore, while some design systems provide the ability to automatically trace images using conversion tools, these conversion tools are often inflexible. For example, the accuracy of many existing conversion tools is largely dependent on the resolution of the input raster, and often produce significantly complex vector outputs that include unnecessary and / or incorrect details. Notably, these extraneous details often require users to perform a cleaning process before integrating the output into a subsequent workflow. Moreover, existing conversion tools are often complex and require users to manually select, filter, and integrate individual outlines to obtain a vector form of the intended portion of the raster image. In fact, the conversion tools of existing design systems are often cumbersome, which significantly hinders the flexibility and / or adaptability of existing design systems when automatically tracing images.

[0022] Relatedly, many existing design systems are inefficient to operate because they rely on a large amount of manual input. To illustrate, in many existing design systems, users must manually select, trace, and correct vector paths by generating and interacting with a large number of points (e.g., control points), which is both resource- and labor-intensive. Moreover, while some design systems provide automatic tracing features, the automatic tracing features are often highly dependent on the resolution of the raster image and are unable to accurately capture subtle differences in the raster image. This lack of precision requires additional manual corrections to achieve a desired level of detail, resulting in low operational efficiency. Furthermore, many automatic tracing features do not provide the ability to partially vectorize a raster image (e.g., a single vector path), and thus require manual editing to select individual vector paths from an entirely vectorized image. Moreover, the interfaces of some conversion tools are complex and unintuitive, resulting in a steep learning curve that slows down the conversion process, particularly for users unfamiliar with vector-based design applications.

[0023] As described above, embodiments of the assisted vector tracing system provide various advantages over conventional design systems. For example, one or more embodiments of the assisted vector tracing system improve operational flexibility as compared to conventional design systems. Unlike existing conversion tools that rely on substantial manual editing of paths and path points, the assisted vector tracing system provides assisted guidance for completing vector paths without requiring the user to draw complete curves. For example, based on a selection of an outline from a set of potential outlines corresponding to a client device input, the assisted vector tracing system automatically generates a complete vector path corresponding to the geometry of the selected outline. Moreover, unlike existing automated tools, the assisted vector tracing system provides the ability to partially vectorize a raster image. Furthermore, the assisted vector tracing system provides contextual feedback in real-time by adding or removing displays of assisted guidance representing potential highlighted outlines as the client device input is modified.

[0024] Moreover, in one or more embodiments, the assisted vector tracing system provides a streamlined process that improves operational efficiency. For example, as compared to the complex user interfaces of existing design systems, the assisted vector tracing system reduces the complexity of image tracing by providing a simple user interface that reacts to client device input based on content extracted from a raster image. In particular, the assisted vector tracing system provides an intuitive method for user interaction with a raster image, provides contextual feedback to assist in the tracing workflow, and automatically completes vector paths based on client device input.

[0025] For example, rather than requiring the user to manually draw individual outlines representing intended portions of a raster image, the assisted vector tracing system provides assisted guidance to highlight a selection of potential outlines (e.g., partial Bezier curves) associated with client device input. In particular, based on one or more points drawn in a graphical user interface, the assisted vector tracing system provides a selection of potential outlines from an extracted segment within a raster image corresponding to client device input. In this way, the assisted vector tracing system provides a technique for partially vectorizing a raster image based on matching client device input to outlines extracted from the raster image.

[0026] Further details regarding the assisted vector tracing system will now be provided with reference to the accompanying drawings. For example, Figure 1 A schematic diagram illustrating an example system environment (“environment”) 100 in which the assisted vector tracing system 106 operates is shown. As Figure 1 As shown, the environment 100 includes server device(s) 102, a network 108, and client device(s) 110.

[0027] Although Figure 1Environment 100 is described as having a specific number of components, but environment 100 can have any number of additional or alternative components (e.g., any number of servers, client devices, or other components communicating with the auxiliary vector tracking system 106 via network 108). Similarly, although Figure 1 A specific arrangement of (multiple) server devices 102, network 108 and (multiple) client devices 110 is shown, but various additional arrangements are also possible.

[0028] Multiple server devices 102, network 108, and multiple client devices 110 communicate and couple with each other directly or indirectly (e.g., through the following combination). Figure 10 (Network 108 is discussed in more detail). Furthermore, (multiple) server devices 102 and (multiple) client devices 110 include one of a variety of computing devices (including, for example, regarding...). Figure 10 (One or more computing devices discussed in more detail).

[0029] like Figure 1 As shown, environment 100 includes server devices 102 and a digital design creation and editing system 104. Server devices 102 utilize the digital design creation and editing system 104 to generate, track, store, process, receive, and transmit electronic data, including outlines, images, path inputs, and vector paths. For example, server devices 102 receive or monitor interactions between client devices 110. In some implementations, server devices 102 transmit content to client devices 110 to cause client devices 110 to display content associated with a vector path. For example, server devices 102 present images, path inputs, vector paths, and / or outlines to client devices 110 and display images, path outputs, vector paths, and / or outlines on client devices 110, the displayed images, path outputs, vector paths, and / or outlines corresponding to system requirements (e.g., providing vector paths for display via client applications 112).

[0030] Furthermore, the server devices 102 include all or part of the auxiliary vector tracking system 106. For example, the auxiliary vector tracking system 106 operates on the server devices 102 to access digital content (including images, path inputs, vector paths, and / or outlines), determine changes in the digital content, and provide localization of content changes to the client devices 110. In one or more embodiments, via the server devices 102, the auxiliary vector tracking system 106 generates and displays images, path inputs, vector paths, and / or outlines based on input from the client devices 110. Reference will be made below. Figure 10Example components of the assisted vector tracing system 106 are described.

[0031] Further, as Figure 1 illustrated, the illustrated system includes client device(s) 110. In some embodiments, the client device(s) 110 include, without limitation, a mobile device (e.g., a smartphone, a tablet), a laptop, a desktop computer, or another type of computing device, including those explained below with reference to Figure 10 Some embodiments of the client device(s) 110 are operated by a user to perform various functions via respective client application(s) 112, such as generating and modifying vector paths. The client device(s) 110 include one or more applications (e.g., client application(s) 112) that access, edit, modify, store, and / or provide digital image content for display. For example, in some embodiments, the client application(s) 112 include a software application installed on the client device(s) 110. However, in other cases, the client application(s) 112 include a web browser or other application that accesses a software application hosted on the server device(s) 102.

[0032] In one or more embodiments, the assisted vector tracing system 106 is implemented in whole or in part by individual elements of the environment 100. Indeed, as Figure 1 illustrated, the assisted vector tracing system 106 is implemented for the server device(s) 102 and the client device(s) 110. In particular embodiments, the assisted vector tracing system 106 on the client device(s) 110 includes a web application, a native application installed on the client device(s) 110 (e.g., a mobile application, a desktop application, a plug-in application, etc.), or a cloud-based application, where a portion of the functionality is performed by the server device(s) 102.

[0033] In additional or alternative embodiments, the assisted vector tracing system 106 on the client device(s) 110 represents and / or provides the same or similar functionality described herein in connection with the assisted vector tracing system 106 on the server device(s) 102. In some implementations, the assisted vector tracing system 106 on the server device(s) 102 supports the assisted vector tracing system 106 on the client device(s) 110.

[0034] In some embodiments, the assistive vector tracing system 106 includes a web hosting application that allows the client device(s) 110 to interact with content and services hosted on the server device(s) 102. To illustrate, in one or more implementations, the client device(s) 110 access a web page or computing application supported by the server device(s) 102. The client device(s) 110 provide input (e.g., a selected content item) to the server device(s) 102. In response, the assistive vector tracing system 106 on the server device(s) 102 generates / modifies digital content. The server device(s) 102 then provide the digital content to the client device(s) 110.

[0035] In another implementation, the assistive vector tracing system 106 on the server device(s) 102 supports the assistive vector tracing system 106 on the client device(s) 110. For example, in some cases, the assistive vector tracing system 106 on the server device(s) 102 generates or learns parameters of one or more machine learning models (e.g., an object detection model and / or an object segmentation model). The assistive vector tracing system 106 then provides the one or more trained machine learning models to the client device(s) 110 via the server device(s) 102. In other words, the client device(s) 110 obtain (e.g., download) the one or more machine learning models (e.g., with any learned parameters) from the server device(s) 102. After the download, the one or more machine learning models on the client device(s) 110 utilize the one or more trained machine learning models to generate the bounding box independent of the server device(s) 102.

[0036] In some embodiments, although Figure 1 Not shown in FIG. 1, the environment 100 has different arrangements of components and / or different numbers or sets of components in total. For example, in certain embodiments, the client device(s) 110 bypass the network 108 and communicate directly with the server device(s) 102. As another example, the environment 100 includes a third-party server that includes a content server and / or a data collection server.

[0037] As previously mentioned, in one or more embodiments, the assistive vector tracing system 106 utilizes machine learning and assistive guidelines for path tracing to generate digital design content that includes vector paths. For example, Figure 2 An overview diagram of the assistive vector tracing system 106 generating vector paths based on a raster image is shown in accordance with one or more embodiments. Additional details regarding various actions of Figure 2 are provided with reference to subsequent figures thereafter.

[0038] As Figure 2As shown, the auxiliary vector tracking system 106 generates a vector path 260 based on the raster image 210 and the path input 240 using the disclosed methods. Specifically, in one or more embodiments, the auxiliary vector tracking system 106 receives or determines the raster image 210 (e.g., through client device interaction). For example, the raster image 210 includes an image composed of pixels such as JPEG, GIF, or PNG, etc. As shown, the raster image 210 contains one or more identifiable objects or elements within the image. For example, the raster image 210 contains explicitly identified semantic objects including, but not limited to, semantic objects such as people, animals, buildings, books, tools, and / or symbols, etc.

[0039] As further shown, in one or more embodiments, the auxiliary vector tracking system 106 segments the raster image 210 into segment mask(s) 220. Specifically, the auxiliary vector tracking system 106 processes the raster image 210 to identify segments within the raster image 210 and generates segment mask(s) 220 corresponding to the segments. To illustrate, the auxiliary vector tracking system 106 classifies each pixel in the raster image 210 as belonging to a semantic object or background using an object detection model 222 (e.g., a deep neural network image segmentation model). Further, the object detection model 222 generates one or more object masks, where each object mask demarcates the boundaries of a semantic object within the raster image 210. As further shown, the auxiliary vector tracking system 106 further segments the semantic objects into constituent segments (e.g., visually distinct parts of the semantic objects) using an object segmentation model 224. Further, the auxiliary vector tracking system 106 generates segment mask(s) 220 for the constituent segments of the semantic objects within the raster image 210 using the object segmentation model.

[0040] As further shown, the auxiliary vector tracking system 106 generates outer frame(s) 230 from the segment mask(s) 220. Specifically, the auxiliary vector tracking system 106 generates outer frame(s) 230 that correspond to paths that demarcate boundaries around shapes or segments or other contours within the raster image 210. For example, to generate the outer frame(s) 230, the auxiliary vector tracking system 106 creates binary mask(s) for the segment mask(s) 220 by applying thresholding to the segment mask(s) 220. Further, the auxiliary vector tracking system 106 extracts contours to determine the boundaries of the segments represented by the segment mask(s) 220. More specifically, the auxiliary vector tracking system 106 generates the outer frame(s) 230 by converting the extracted contours into the outer frame(s) 230. In one or more embodiments, the auxiliary vector tracking system 106 generates the outer frame(s) 230 as polylines corresponding to the contours.

[0041] As further shown, the assisted vector tracing system 106 determines potential bounding box(es) 250. Specifically, the assisted vector tracing system 106 utilizes a bounding box matching model 252 with hit detection to determine potential bounding box(es) 250 within a threshold distance of the path input 240. For example, the assisted vector tracing system 106 compares the bounding box(es) 230 to the path input 240 to select one or more potential bounding boxes 250 of the bounding box(es) 230. To illustrate, the assisted vector tracing system 106 receives a client device input that includes the path input 240, which includes one or more points that represent a path or partial path (e.g., partial Bezier curve, input spline, point, line primitive) within the raster image 210 that is traced using a tool (e.g., a pen, a brush, or a selection tool) in an image editing application. In some cases, the assisted vector tracing system 106 flattens the path input 240 to a path input line primitive and flattens the bounding box(es) 230 to bounding box line primitives. Based on the comparison of the line primitives, the assisted vector tracing system 106 selects one or more potential bounding boxes of the bounding box(es) 230 that match the path input 240 (e.g., within a threshold distance of the path input 240).

[0042] Further, in certain embodiments, the assisted vector tracing system 106 generates a vector path 260. Specifically, the assisted vector tracing system 106 generates the vector path 260 by generating a vector path that corresponds to the path input 240 using one of the potential bounding boxes 250 of the bounding box(es) 230. To illustrate, the assisted vector tracing system 106 highlights the potential bounding box(es) 250 on the client device to visually indicate the potential bounding box(es) 250 that correspond to the path input 240. Upon receiving an indication from the client device that selects one or more of the potential bounding boxes 250, the assisted vector tracing system automatically completes the path input 240 by fitting a smooth vector path (e.g., a B-spline, a Bezier curve) to the selected potential bounding box to generate the vector path 260 that corresponds to the geometry of the selected potential bounding box.

[0043] As mentioned above, the assisted vector tracing system 106 performs segmentation of the input raster image to generate an object mask and a segment mask. Figure 3A An example of generating an object mask using an object detection model is shown in accordance with one or more embodiments. Figure 3B An example of generating a segment mask using an object segmentation model is shown in accordance with one or more embodiments.

[0044] As Figure 3AAs shown, the auxiliary vector tracking system 106 generates object(s) mask(s) 330 from the raster image 310 using the object detection model 320. In certain embodiments, the auxiliary vector tracking system 106 utilizes the object detection model 320 (e.g., a deep neural network image segmentation model) to delineate semantic objects within the raster image 310. In one or more implementations, the object detection model 320 is a machine learning model (e.g., a neural network) or a collection of machine learning models designed for a semantic segmentation task (e.g., segmenting an image into multiple segments).

[0045] Relatedly, in certain embodiments, a machine learning model includes or refers to a computer algorithm or a collection of computer algorithms that automatically improve for a particular task through iterations of output or predictions based on the use of data. For example, a machine learning model utilizes one or more learning techniques to improve accuracy and / or effectiveness via training data and one or more loss functions. Along these lines, a neural network includes or refers to a machine learning model that is trained and / or tuned based on inputs to determine a digital content item, a key element, or an approximation of an unknown function. For example, a neural network includes a model of interconnected artificial neurons (e.g., organized in layers) that communicate and learn to approximate a complex function and generate an output (e.g., an image segment) based on a plurality of inputs provided to the neural network. In some cases, a neural network refers to an algorithm (or collection of algorithms) that implements a deep learning technique to model high-level abstractions in data. In certain embodiments, a neural network includes various layers, such as an input layer, one or more hidden layers, and an output layer, each layer performing a task for processing data. For example, a neural network includes a deep neural network, a convolutional neural network, a transformer neural network, a diffusion neural network, a recurrent neural network (e.g., LSTM), a graph neural network, or a generative adversarial neural network.

[0046] To illustrate, in one or more embodiments, the object detection model 320 includes a convolutional neural network. For example, the object detection model 320 incorporates an encoder with several layers of convolution and pooling operations to reduce the spatial dimensions of an image while increasing the depth (number of features or channels). In this way, the object detection model 320 extracts features from the raster image 310 and encodes the input into a compressed form. In one or more embodiments, the object detection model 320 also incorporates a decoder that uses transposed convolutions (e.g., deconvolutions) to expand the spatial dimensions and reduce the depth (number of features or channels) to construct a segmentation map. The object detection model 320 uses the segmentation map to predict a class (e.g., an object type) for each pixel. In some cases, the object detection model 320 also uses skip connections to connect features from the encoder to corresponding layers in the encoder. In this way, the object detection model 320 maintains spatial information to determine precise boundaries of objects.

[0047] In some cases, the auxiliary vector tracking system 106 utilizes a transformer-based neural network architecture. Specifically, the auxiliary vector tracking system 106 employs a self-attention mechanism to segment objects within an image. For example, the auxiliary vector tracking system 106 integrates a transformer model with a streamlined multilayer perceptron (MLP) decoder to efficiently handle the segmentation task and capture the fine details and broader contextual information required for accurate image segmentation. For instance, the object detection model 320 uses a transformer-based neural network to classify pixels in the raster image 310 into background pixels or semantic object pixels to generate object masks(s)(s). In this way, the auxiliary vector tracking system 106 employs hierarchical features of a neural network organized in a pyramid configuration with a location encoding scheme tailored to the requirements of the segmentation task, and generates object masks(s)(s).

[0048] For example, object detection model 320 classifies each pixel in raster image 310 as belonging to a semantic object or background. Furthermore, object detection model 320 generates multiple object masks 330 corresponding to semantic objects within raster image 310, where each object mask 330 defines the boundary of the semantic object detected by object detection model 330.

[0049] like Figure 3B As further shown, the auxiliary vector tracking system 106 utilizes the object segmentation model 350 to generate fragment masks for individual segments of semantic objects in the raster image 310. In one or more implementations, the object segmentation model 350 is a machine learning model (e.g., a neural network) or a collection of machine learning models designed for semantic segmentation tasks (e.g., segmenting an image into multiple segments). Similar to the object detection model 320, in some embodiments, the object segmentation model 350 includes convolutional neural networks and / or transformer-based neural network architectures.

[0050] For illustration, in one or more embodiments, object segmentation model 350 includes a neural network that generates masks for more detailed segmentation of objects. For example, object segmentation model 350 incorporates an image encoder, a cue encoder, and a mask decoder. Object segmentation model 350 utilizes the image encoder to generate image embeddings from raster image 310 to capture various aspects of the image, such as texture, color, shape, and spatial hierarchy. Object segmentation model 350 utilizes the cue encoder to encode object mask 340 as a positional cue embedding. Furthermore, object segmentation model 350 uses the mask decoder to map the combined embeddings from raster image 310 and object mask 340 to generate a fragment mask 360. In some cases, object segmentation model 350 uses a transform block to perform cue self-attention and cross-attention operations between cue embeddings and image embeddings to update all embeddings.

[0051] For example, the auxiliary vector tracking system 106 further segments the object mask(s) 330 using the object segmentation model 350. For example, the object segmentation model 350 further segments the object mask(s) using the object segmentation model 350 and determines visually distinct segments of semantic objects within the raster image 310. In particular, the object segmentation model 350 generates segment masks 360, where each segment mask 360 demarcates the boundaries of a segment detected by the object segmentation model 350.

[0052] As previously described, the auxiliary vector tracking system 106 converts the segment masks in the raster image to bounding boxes corresponding to the segments within the raster image. Figure 4 An example of generating bounding boxes corresponding to one or more contours in the segment masks is shown in accordance with one or more embodiments.

[0053] As Figure 4 shown, the auxiliary vector tracking system 106 generates the bounding boxes 440 based on the segment masks 410. In particular, the auxiliary vector tracking system 106 generates binary masks 420 from the segment masks 410. To illustrate, the auxiliary vector tracking system 106 transforms the segment masks 410 (e.g., alpha masks having a range of alpha values) to binary images through thresholding. For example, the auxiliary vector tracking system 106 sets a threshold value as a cut-off point to distinguish between foreground (e.g., semantic objects) and background. When thresholding the segment masks 410, the auxiliary vector tracking system 106 assigns a first color value (e.g., white, representing a value of “1”) to pixels on one side of the threshold and a second color value (e.g., black, representing a value of “0”) to pixels on the other side. After thresholding, the binary masks 420 clearly demarcate objects, with pixels of the first color value defining semantic objects and pixels of the second color value indicating the absence of semantic objects.

[0054] As shown, the auxiliary vector tracking system 106 generates contours 430 corresponding to the binary masks 420. In particular, the auxiliary vector tracking system 106 utilizes a contour extraction algorithm to identify and extract the bounding boxes or boundaries of the segments within the binary masks 420. For example, the auxiliary vector tracking system 106 utilizes one or more boundary tracking algorithms to perform a topological analysis of the binary masks 420. To illustrate, the auxiliary vector tracking system 106 utilizes a first boundary tracking algorithm to determine the enclosing relationship between the boundaries of the binary masks 420 and generate a representation of the binary masks 420 for feature extraction. Further, in some embodiments, the auxiliary vector tracking system 106 utilizes a second boundary tracking algorithm to trace the outermost boundaries (e.g., not enclosed by holes) of the binary masks 420.

[0055] In some cases, the auxiliary vector tracking system 106 applies a contour extraction algorithm to simplify the contour data by eliminating points that contribute little to the overall shape. In this way, the auxiliary vector tracking system 106 reduces the computational load and simplifies the data for further processing (e.g., by reducing the number of anchor points to below a certain limit or threshold). By approximating the contour with a contour extraction algorithm, the auxiliary vector tracking system 106 reduces the number of anchor points, making the representation of the shape that is visually similar to the original bounding box more compact and easier to manage.

[0056] As shown, the auxiliary vector tracking system 106 generates a bounding box 440 from the contour 430. In particular, the auxiliary vector tracking system 106 converts the contour 430 from the contour points of the contour 430 to a polygonal line (e.g., a sequence of connected line primitives). For example, the auxiliary vector tracking system 106 converts a set of contour points representing a closed boundary of a segment (e.g., the contour 430) to a polygon (e.g., a closed polygonal line). More specifically, the auxiliary vector tracking system 106 connects the contour points to generate a polygonal line (or polygon) and generate a bounding box 440 of the segment represented by the segment mask 410.

[0057] As also mentioned, the auxiliary vector tracking system 106 utilizes a bounding box matching model to match the bounding box of an object in a raster image to a client device input (e.g., a path input). Figure 5 An example of utilizing a bounding box matching model is shown in accordance with one or more embodiments. Notably, in certain embodiments, the auxiliary vector tracking system 106 iteratively performs the actions of Figure 5 and re-determines the potential bounding box based on one or more changes to the client device input.

[0058] As Figure 5 shown, the auxiliary vector tracking system 106 performs an action 520 to select filtered bounding boxes 522 from the bounding boxes 510 extracted in conjunction with one or more segments in the raster image. As described above, in one or more embodiments, the auxiliary vector tracking system 106 generates a set of bounding boxes corresponding to segment masks of semantic objects within a raster image (e.g., the bounding boxes 510 corresponding to segments within a cat object in the raster image). Further, based on the location of the path input 512, the auxiliary vector tracking system 106 limits the potential bounding boxes by excluding bounding boxes in the bounding boxes 510 that are not within a threshold distance of the path input 512. In this way, the auxiliary vector tracking system 106 filters the bounding boxes 510 to select the filtered bounding boxes 522.

[0059] For example, the assisted vector tracing system 106 uses bounding boxes to perform action 520 to filter bounding boxes and determine filtered bounding boxes 522. Specifically, the assisted vector tracing system 106 determines the bounding box corresponding to the path input 512. The assisted vector tracing system 106 also determines the bounding box corresponding to the bounding box 510. Furthermore, the assisted vector tracing system 106 selects the filtered bounding box 522 as a subset of the bounding boxes 510 that have bounding boxes intersecting with the bounding boxes of the path input 512. In this way, the assisted vector tracing system 106 eliminates irrelevant bounding boxes that are outside a threshold distance of the path input 512.

[0060] In one or more embodiments, the assisted vector tracking system 106 uses hit detection to filter bounding boxes 510 to perform action 520. Specifically, the assisted vector tracking system 106 uses hit detection to detect whether the path input 512 intersects with any bounding box 510. In fact, by utilizing hit detection, the assisted vector tracking system 106 performs action 520 to filter bounding boxes, including those that the client device might be attempting to track, consistent with user intent. Furthermore, in some embodiments, the assisted vector tracking system 106 extends or expands the path input 512. In this way, even if the vector bounding box is slightly offset, if the bounding box falls within the extended area, the hit detection employed by the assisted vector tracking system 106 will still merge the bounding box (in bounding box 510) into the filtered bounding box 522.

[0061] like Figure 5 As shown, the auxiliary vector tracing system 106 performs action 530 to generate polylines. For example, the auxiliary vector tracing system 106 converts the filtered outline 522 into an outline polyline and the path input 512 into a path polyline. Specifically, the auxiliary vector tracing system 106 converts each filtered outline 522 into an outline polyline and the path input 512 into a sequence of straight line segments (e.g., line primitives) that approximate a curve. Furthermore, the auxiliary vector tracing system 106 normalizes the polylines to ensure that the parameterization of the filtered outline 522 and the path input 512 is consistent. In some cases, the auxiliary vector tracing system 106 normalizes the polylines by adjusting them so that a point on one polyline corresponds to a point on a second polyline at the same relative distance along the polyline.

[0062] In some embodiments, the assisted vector tracing system 106 utilizes the following algorithm to generate polylines, thereby reducing the complexity of the filtered outline 522 and path input 512 while maintaining the integrity of the original shape:

[0063]

[0064]

[0065]

[0066] In particular, as described above, the auxiliary vector tracking system 106 utilizes Algorithm 1 to flatten a quadratic Bezier curve into a line segment and Algorithm 2 to flatten a cubic Bezier curve into a line segment.

[0067] As shown, the auxiliary vector tracking system 106 performs act 540 to simplify the points of the polyline generated in act 530, thereby generating a simplified polyline. In particular, in some cases, when the auxiliary vector tracking system 106 flattens the filtered outline 522 and the path input 512 to generate the polyline, a large number of points can be generated. For example, when the auxiliary vector tracking system 106 implements the fine details of the filtered outline 522 and the path input 512 using line primitives, redundant and / or intermediate points (e.g., superfluous points within a straight line) can be generated. In one or more embodiments, to remove the redundant points from the polyline, the auxiliary vector tracking system 106 utilizes a simplification algorithm, such as the following algorithm:

[0068]

[0069]

[0070] As Figure 5 Further shown, the auxiliary vector tracking system 106 performs act 560 to compare the simplified polyline generated in act 540. In particular, the auxiliary vector tracking system 106 performs act 570 to select the potential outline(s) 572 from the filtered outline 522 that match the path input 512 based on comparing the points of the simplified polyline. To illustrate, the auxiliary vector tracking system 106 compares the simplified polyline of the filtered outline 522 to the simplified polyline of the path input 512 to select the potential outline(s) 572. In fact, based on the comparison of the points of the polyline satisfying a matching tolerance threshold, the auxiliary vector tracking system 106 selects the potential outline(s) 572 from the filtered outline 522. In this way, the auxiliary vector tracking system 106 selects the potential outline(s) 572 that correspond to the outline that the user intended to track.

[0071] To further illustrate, the auxiliary vector tracking system 106 provides an efficient, intuitive graphical user interface for creating an outline that corresponds to a client device input. Figures 6A to 6C An example of generating and providing an outline associated with a client device input within a graphical user interface is shown in accordance with one or more embodiments. Moreover, as Figures 6A to 6C shown, the auxiliary vector tracking system 106 provides adaptive feedback within the graphical user interface based on changes to the client device input.

[0072] As Figure 6AAs shown, the assisted vector tracking system 106 interacts with the client device 600 using a vector-based application (e.g., an image editing application for generating or editing vector images) to modify the raster image 602. As shown, based on the client device input 604a, the assisted vector tracking system 106 provides adaptive feedback to the client device and displays a first highlighted potential bounding box 606, a second highlighted potential bounding box 608, and a third highlighted potential bounding box 610. As described in conjunction with the previous figures, the assisted vector tracking system 106 determines a potential bounding box corresponding to the client device input 604a (which includes a portion of the path the user wants to track). For example, the assisted vector tracking system 106 determines that the bounding boxes of the first highlighted potential bounding box 606, the second highlighted potential bounding box 608, and the third highlighted potential bounding box 610 intersect with the bounding box of the client device input 604a. Therefore, the assisted vector tracking system 106 determines that the first highlighted potential bounding box 606, the second highlighted potential bounding box 608, and the third highlighted potential bounding box 610 match the path the user wants to track.

[0073] like Figure 6B As further illustrated, the assisted vector tracing system 106 provides adaptive feedback and updates the display of potential bounding boxes on the client device 600 in response to changes in client device input 604a to remove one or more potential bounding boxes. Specifically, in response to changes in client device input 604a to add points, as shown in client device input 604b, the assisted vector tracing system 106 redetermines the potential bounding box corresponding to client device input 604b (which includes the updated portion of the path the user intends to trace). For example, the assisted vector tracing system 106 filters potential bounding boxes (e.g., reducing the search space for bounding boxes), simplifies the polyline between client device input 604b and the filtered bounding boxes, and compares the polylines to redetermine the potential bounding box associated with client device input 604b.

[0074] As shown in the figure, based on filtering of potential outlines, the auxiliary vector tracking system 106 removes the first highlighted potential outline 606 from the display on the client device. Also as shown, the auxiliary vector tracking system 106 determines that the bounding boxes of the second highlighted potential outline 608 and the third highlighted potential outline 610 intersect with the bounding box of the client device input 604b. Furthermore, based on the client device input 604b, the auxiliary vector tracking system 106 displays (or does not remove from the display) the second highlighted potential outline 608 and the third highlighted potential outline 610.

[0075] In some embodiments, in response to a change to the client device input 604b, the assisted vector pursuit system 106 determines a change to the shape of the path input. In this case, if the bounding box of the client device input 604b does not change, the assisted vector pursuit system 106 does not further update and / or filter the potential hulls (the initial search space does not change). In this case, to update the potential hulls for display on the client device, the assisted vector pursuit system 106 generates and simplifies the polylines for the edited client device input 604b, and compares the polylines to filter the potential hulls associated with the edited client device input 604b.

[0076] As Figure 6C Further shown, in some embodiments, the assisted vector pursuit system 106 provides adaptive feedback and updates the display of potential hulls on the client device to add one or more potential hulls in response to additional changes to the client device input. For example, based on modifying the client device input 604c to remove a point from the path input (e.g., removing a recently added point), the assisted vector pursuit system 106 updates the display of potential hulls on the client device to add one or more potential hulls. For example, the assisted vector pursuit system 106 filters the potential hulls (e.g., to expand the search space of the hulls), simplifies the polylines of the client device input 604c and the filtered hulls, and compares the polylines to re-determine the potential hulls associated with the client device input 604c.

[0077] As shown, based on the filtering of the potential hulls, the assisted vector pursuit system 106 adds (or adds back) the first highlighted potential hull 606 to the display. Also shown, the assisted vector pursuit system 106 determines that the bounding boxes of the second highlighted potential hull 608 and the third highlighted potential hull 610 intersect the bounding box of the client device input 604a. As shown, based on the client device input 604a, the assisted vector pursuit system 106 displays the first highlighted potential hull 606, the second highlighted potential hull 608, and the third highlighted potential hull 610.

[0078] In one or more embodiments, when a point is added to the client device input, the assisted vector pursuit system 106 maintains a history of potential hulls based on the search space corresponding to the client device input. Thus, as points are added or removed, the assisted vector pursuit system 106 quickly narrows or expands the set of potential hulls to provide assisted guidance in the graphical user interface. Further, in some embodiments, a change in the location of a point in the client device input causes the assisted vector pursuit system 106 to determine a new search / filtering of potential hulls corresponding to the client device input.

[0079] As described above, the assisted vector tracing system 106 provides adaptive feedback to the client device and automatically completes the vector path. Figures 7A to 7C An example of providing adaptive feedback within a graphical user interface and automatically completing a vector path is shown in accordance with one or more embodiments.

[0080] As Figure 7A shown, the assisted vector tracing system 106 interacts with the vector-based application on the client device 700 to modify the raster image 702. For example, the assisted vector tracing system 106 receives client device input 704a indicating a point corresponding to a segment boundary within the raster image 702. In certain embodiments, as described in connection with Figures 6A to 6C the assisted vector tracing system 106 displays potential outlines corresponding to segment boundaries within the raster image within a threshold distance of the client device input 704a.

[0081] As Figure 7B further shown, the assisted vector tracing system 106 receives client device input 704b indicating additional points corresponding to a segment boundary of segment 710 and a segment boundary of segment 712 within the raster image. As further shown, the client device input 704b corresponds to an incomplete or partially traced vector path. In certain embodiments, based on the selection of the additional points, the assisted vector tracing system 106 displays potential outlines corresponding to segment boundaries within the raster image within a threshold distance of the client device input 704b.

[0082] To illustrate, the assisted vector tracing system 106 provides a highlighted indication of the outlines corresponding to the client device input 704b. Specifically, the assisted vector tracing system 106 generates and displays one or more outlines that include the incomplete vector path of the client device input 704b and correspond to potential outlines of segments within the raster image. To illustrate, based on the client device input 704b, the assisted vector tracing system 106 displays highlighted outline 706 and highlighted outline 708a. As shown, the highlighted outline 706 includes the segment boundary of segment 710, including the portion of the segment boundary traced by the client device input 704b. As shown, the highlighted outline 708a includes the segment boundary of segment 712, including the portion of the segment boundary traced by the client device input 704b.

[0083] As Figure 7CFurther shown, based on the client device interaction to select the highlight box 708a, the assisted vector tracing system 106 generates a complete vector path 708b. In particular, the assisted vector tracing system 106 automatically generates and displays the complete vector path 708b without requiring additional client device input. To illustrate, the assisted vector tracing system 106 receives the client device interaction (e.g., pressing down on the stylus, such as the stylus 104, on the touch screen 102) to select the highlight box 708a. In response, the assisted vector tracing system 106 automatically generates and displays the complete vector path 708b. In this example, the complete vector path 708b is generated based on the vector path 708a and the vector path 706a. In this example, the complete vector path 708b is generated based on the vector path 708a and the vector path 706a. <shift>(e.g., modification keys), and generate a complete vector path 708b that outlines segment 712 and corresponds to the geometry of the highlighted outline 708a. In some embodiments, the auxiliary vector tracking system 106 corrects any defects in the partial curve of the client device input 704b by generating a complete vector path 708b corresponding to the outline of segment 712 (and replacing the client device input 704b).

[0084] In one or more embodiments, to generate the complete vector path 708b, the auxiliary vector tracing system 106 converts the polyline of the highlighted outline 708a into a vector curve to generate the complete vector path 708b. Specifically, the auxiliary vector tracing system 106 converts the polyline of the highlighted outline 708a into a vector curve for editing within vector-based applications while maintaining the accuracy and aesthetics of the highlighted outline 708a. In some embodiments, the auxiliary vector tracing system 106 fits a cubic Bézier curve to the highlighted outline 708a to generate the complete vector path 708b. In some embodiments, the auxiliary vector tracing system 106 uses B-spline fitting to fit a B-spline curve to approximate the highlighted outline 708a (e.g., by adjusting control points to reduce one or more predefined metrics). In some embodiments, the auxiliary vector tracing system 106 utilizes partial differential equations to preserve the salient geometric features and smooth low-level details of the highlighted outline 708a. Furthermore, in some embodiments, in order to generate the potential outline (e.g., highlight outline 706 and highlight outline 708a), the auxiliary vector tracing system 106 converts the polyline into a vector curve, as described before displaying the potential outline in the graphical user interface.

[0085] Now go to Figure 8 Additional details regarding the various components and capabilities of the auxiliary vector tracking system 106 will now be provided. Specifically, Figure 8 This illustrates a computing device 800 (e.g., referenced above). Figure 1 The auxiliary vector tracing system 106 is implemented in one of the (multiple) server devices 102 and / or one of the (multiple) client devices 110 discussed. Furthermore, the auxiliary vector tracing system 106 is also part of the digital design creation and editing system 104. Figure 8 As shown, the auxiliary vector tracking system 106 includes, but is not limited to, a segmentation manager 802, an outer frame manager 808, a vector curve manager 814, and a data storage manager 816.

[0086] As mentioned above, such as Figure 8 As shown, the assisted vector tracing system 106 includes a segmentation manager 802. In one or more embodiments, the segmentation manager 802 manages segmentation of objects within a raster image. The segmentation manager 802 utilizes an object detection model 804 to segment an image based on semantic objects within the image. Further, the segmentation manager 802 utilizes an object segmentation model 806 to further segment the semantic objects into visually distinct segments. The segmentation manager 802 utilizes the object detection model 804 and the object segmentation model 806 to generate segment masks corresponding to segments within the raster image.

[0087] Further, as shown, the assisted vector tracing system 106 includes a bounding box manager 808. The bounding box manager 808 manages generation and selection of bounding boxes corresponding to segments within a raster image. In particular, the bounding box manager 808 utilizes a bounding box generation model 810 to generate bounding boxes corresponding to the segment masks. For example, the bounding box manager 808 utilizes the bounding box generation model 810 to generate binary masks and determine outlines of segments within the raster image to generate the bounding boxes. Further, the bounding box manager 808 utilizes a bounding box matching model 812 to convert the bounding boxes to polylines (e.g., line primitives) and compare the bounding boxes to client device inputs. Further, the bounding box manager 808 provides a highlighted bounding box that matches a client device input for display on the client device. Figure 8

[0088] As shown, the assisted vector tracing system 106 includes a bounding box manager 808. The bounding box manager 808 manages generation and selection of bounding boxes corresponding to segments within a raster image. In particular, the bounding box manager 808 utilizes a bounding box generation model 810 to generate bounding boxes corresponding to the segment masks. For example, the bounding box manager 808 utilizes the bounding box generation model 810 to generate binary masks and determine outlines of segments within the raster image to generate the bounding boxes. Further, the bounding box manager 808 utilizes a bounding box matching model 812 to convert the bounding boxes to polylines (e.g., line primitives) and compare the bounding boxes to client device inputs. Further, the bounding box manager 808 provides a highlighted bounding box that matches a client device input for display on the client device. Figure 8 Further, as shown, the assisted vector tracing system 106 includes a vector curve manager 814. In particular, the assisted vector tracing system 106 utilizes the vector curve manager 814 to generate a vector path corresponding to a selected bounding box. In particular, the vector curve manager 814 receives a selection of a bounding box from the highlighted bounding boxes. Further, the vector curve manager 814 generates a vector path (e.g., a Bezier curve) corresponding to the geometry of the bounding box, incorporates boundaries traced by the client device input, and outlines a segment of the raster image.

[0089] Further, as shown, the assisted vector tracing system 106 includes a data storage manager 816. In particular, the data storage manager 816 (implemented by one or more memory devices) stores digital design documents, including raster images. The data storage manager 816 facilitates use of the digital design documents by the assisted vector tracing system 106.

[0090] ​Each of components 802 to 816 of the assisted vector tracking system 106 includes software, hardware, or both. For example, components 802 to 816 include one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices, such as client devices or server devices. When executed by one or more processors, the computer-executable instructions of the assisted vector tracking system 106 cause the computing devices(s) to perform the methods described herein. Alternatively, components 802 to 816 include hardware, such as dedicated processing devices that perform specific functions or groups of functions. Alternatively, components 802 to 816 of the assisted vector tracking system 106 include a combination of computer-executable instructions and hardware.

[0091] Furthermore, components 802 to 816 of the assisted vector tracking system 106 are implemented as one or more operating systems, one or more standalone applications, one or more modules of an application, one or more plugins, one or more library functions or functions called by other applications, and / or cloud computing models. Therefore, in some embodiments, components 802 to 816 of the assisted vector tracking system 106 are implemented as standalone applications, such as desktop or mobile applications. Furthermore, in some embodiments, components 802 to 816 of the assisted vector tracking system 106 are implemented as one or more web-based applications hosted on a remote server. Alternatively or supplementarily, components 802 to 816 of the assisted vector tracking system 106 are implemented in a suite of mobile device applications or "apps". For example, in one or more embodiments, the assisted vector tracking system 106 includes or operates in conjunction with digital software applications, such as: EXPRESS XD, and CREATIVE The above items are registered trademarks or trademarks of Adobe Inc. in the U.S. and / or other countries.

[0092] Figures 1 to 8 The accompanying text and examples provide various methods, systems, devices, and non-transitory computer-readable media for the auxiliary vector tracking system 106. In addition to the foregoing, one or more embodiments are described with respect to flowcharts of actions for achieving specific results, such as… Figure 9 As shown. In some embodiments, Figure 9 The actions shown are performed by combining more or fewer actions. Furthermore, these actions may be performed in different orders. Additionally, in various embodiments, the actions described herein are repeated, performed in parallel with each other, or performed in parallel with different instances of the same or similar actions. The non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause a computing device to perform. Figure 9 the actions of FIG. 8. In some embodiments, the system is configured to perform the actions of FIG. 9. Figure 9 Alternatively, the actions of FIG. 9 are performed as part of a computer- implemented method. Figure 9 Alternatively, the actions of FIG. 9 are performed as part of a computer- implemented method.

[0093] Figure 9 A flow diagram illustrating a series of actions 900 for modifying a digital document using an auxiliary vector tracing system 106 in accordance with one or more embodiments is shown. While Figure 9 actions in accordance with one embodiment are shown, alternative embodiments can omit, add to, reorder, and / or modify any of the actions shown. Figure 9 actions in accordance with one embodiment are shown, alternative embodiments can omit, add to, reorder, and / or modify any of the actions shown.

[0094] Figure 9 A series of example actions 900 for generating a vector path with an auxiliary vector tracing system 106 is shown. Specifically, in certain embodiments, the series of actions 900 includes an action 902 of determining a set of bounding boxes. Specifically, in one or more embodiments, the action 902 includes determining a set of bounding boxes corresponding to boundaries of a set of segments within a raster image. Specifically, in certain embodiments, the series of actions 900 includes an action 904 of selecting a bounding box from the set of bounding boxes, a sub-action 904a of selecting the bounding box in response to a client device input, and a sub-action 904b of determining points located within a threshold distance of the bounding box. Specifically, in one or more embodiments, the action 904 includes selecting a bounding box corresponding to a segment of the set of segments from the set of bounding boxes in response to the client device input indicating one or more points located within a threshold distance of the bounding box. As shown, in some embodiments, the series of actions 900 also includes an action 906 of providing a highlighted indication of the bounding box. Specifically, in one or more embodiments, the action 906 includes providing a highlighted indication of the bounding box corresponding to the segment for display within a graphical user interface of the client device. Further, in certain embodiments, the series of actions 900 includes an action 908 of generating a vector path based on the bounding box. Specifically, in one or more embodiments, the action 908 includes generating a vector path within a vector image based on the bounding box corresponding to the segment in response to a selection of the bounding box via the graphical user interface.

[0095] In addition to (or instead of) the above actions, in certain embodiments, the series of actions 900 by the auxiliary vector tracking system includes determining a first bounding box of the outer frame. In some embodiments, the series of actions 900 further includes determining a second bounding box of the client device input. Further, in one or more embodiments, the series of actions 900 by the auxiliary vector tracking system 106 includes selecting the outer frame based on the first bounding box intersecting the second bounding box. Further, in some embodiments, the series of actions 900 by the auxiliary vector tracking system 106 includes generating, with the object detection model, an object mask for objects within the raster image with semantic analysis. Further, in one or more embodiments, the series of actions 900 by the auxiliary vector tracking system includes generating, with the object segmentation model, a set of segments within the raster image by segmenting the objects based on the object mask.

[0096] Further, in one or more embodiments, the series of actions 900 includes selecting, from the set of outer frames, an additional outer frame corresponding to an additional segment of the set of segments in response to determining that one or more points of the client device input are within a threshold distance of the additional outer frame. Further, in one or more embodiments, the series of actions 900 includes providing an additional highlight indication of the additional outer frame corresponding to the additional segment for display within the graphical user interface of the client device, where the additional highlight indication is visually distinct from the highlight indication of the outer frame. Further, in one or more embodiments, the series of actions 900 includes selecting the outer frame in response to determining that one or more points of the client device input intersect the outer frame with the hit detection operation.

[0097] In certain embodiments, the series of actions 900 further includes generating a binary mask for the segment. Further, in one or more embodiments, the series of actions 900 includes extracting one or more contours from the binary mask to determine a boundary of the segment. Further, in one or more embodiments, the series of actions 900 includes generating the outer frame from the binary mask including an outer frame polygon corresponding to the one or more contours. Further, in one or more embodiments, the series of actions 900 includes generating an input polygon corresponding to one or more points of the client device input. In one or more embodiments, the series of actions 900 includes selecting the outer frame from the set of outer frames based on a comparison of the input polygon and the outer frame polygon to a matching tolerance threshold. Further, in one or more embodiments, the series of actions 900 includes removing redundant points within the outer frame polygon based on a distance between points in the outer frame polygon with the outer frame matching model.

[0098] Further, in one or more embodiments, the series of acts 900 includes, for the raster image, determining a first bounding box corresponding to a boundary of the first segment and a second bounding box corresponding to a boundary of the second segment. In one or more embodiments, the series of acts 900 further includes receiving, from the client device, a client device input indicating a point in the raster image within a threshold distance of the first segment and the second segment. Further, in one or more embodiments, the series of acts 900 includes providing, in response to the client device input being within the threshold distance of the first segment and the second segment, a first highlight indication of the first bounding box and a second highlight indication of the second bounding box for display within the graphical user interface of the client device. Further, in one or more embodiments, the series of acts 900 includes generating, within the vector image, a vector path having a geometric shape associated with the first bounding box based on a selection of the first bounding box.

[0099] Further, in one or more embodiments, the series of acts 900 includes receiving an additional client device input indicating an additional point in the raster image. Further, in one or more embodiments, the series of acts 900 includes determining that the first segment is not within a threshold distance of the additional point. In one or more embodiments, the series of acts 900 includes removing the first highlight indication of the first bounding box from the display within the graphical user interface. Further, in one or more embodiments, the series of acts 900 includes determining that the first segment is outside the threshold distance of the additional point by limiting potential bounding boxes with a hit detection operation and excluding the first segment.

[0100] In some embodiments, the series of acts 900 further includes receiving a supplemental client device input indicating a removal of the additional point in the raster image. Further, in one or more embodiments, the series of acts 900 of the auxiliary vector tracing system 106 includes providing, in response to the removal of the additional point in the raster image, the first highlight indication of the first bounding box for display within the graphical user interface of the client device. Further, in some embodiments, the series of acts 900 of the auxiliary vector tracing system 106 includes generating a bounding box polyline including a line primitive corresponding to the first bounding box. Further, in one or more embodiments, the series of acts 900 of the auxiliary vector tracing system includes removing redundant points within the bounding box polyline according to a distance between points in the bounding box polyline.

[0101] Further, in one or more embodiments, the series of acts 900 includes generating, with an object detection model, a plurality of object masks for a plurality of objects within the raster image. Further, in one or more embodiments, the series of acts 900 includes generating the first segment and the second segment within the raster image by segmenting, with an object segmentation model, objects based on a mask of the plurality of object masks to generate a set of segment mask masks and generating a plurality of bounding boxes based on contours extracted from the set of segment mask masks.

[0102] Further, in one or more embodiments, the series of acts 900 includes determining a first bounding box corresponding to the first outer frame. In certain embodiments, the series of acts 900 further includes determining a second bounding box corresponding to the second outer frame. Further, in one or more embodiments, the series of acts 900 includes selecting the first outer frame and the second outer frame in response to determining that a third bounding box corresponding to the client device input intersects the first bounding box and the second bounding box.

[0103] Further, in one or more embodiments, the series of acts 900 includes segmenting one or more objects in the raster image into a set of segments using the object segmentation model. Further, in one or more embodiments, the series of acts 900 includes generating a set of outer frames corresponding to outlines extracted from the set of segments. In one or more embodiments, the series of acts 900 includes selecting, in response to the client device input drawing one or more points within a threshold distance of an outer frame, an outer frame corresponding to a segment in the set of segments from the set of outer frames. Further, in one or more embodiments, the series of acts 900 includes providing a highlighted indication of the outer frame corresponding to the segment for display within a graphical user interface of the client device. Further, in one or more embodiments, the series of acts 900 includes generating a vector path within the vector image based on the outer frame corresponding to the segment in response to selection of the outer frame via the graphical user interface.

[0104] In one or more embodiments, the series of acts 900 further includes determining a modified client device input based on receiving an indication of removal of a point from the one or more points. Further, in one or more embodiments, the series of acts 900 includes selecting, in response to the modified client device input being within a threshold distance of an additional outer frame, an additional outer frame corresponding to an additional segment in the set of segments from the set of outer frames. Further, in one or more embodiments, the series of acts 900 includes providing an additional highlighted indication of the additional outer frame for display within a graphical user interface of the client device. Further, in one or more embodiments, the series of acts 900 includes determining an input spline corresponding to the client device input, the input spline including the one or more points. Further, in one or more embodiments, the series of acts 900 includes generating a corresponding input polyline by reducing the input spline to line primitives. Further, in one or more embodiments, the series of acts 900 includes selecting the outer frame corresponding to the segment based on a comparison of the input polyline and the outer frame.

[0105] In certain embodiments, the series of actions 900 further includes determining a set of bounding boxes corresponding to the set of outer boxes. Further, in one or more embodiments, the series of actions 900 includes determining an input bounding box corresponding to the client device input. Further, in one or more embodiments, the series of actions 900 includes selecting a subset of the set of outer boxes based on a subset of the set of bounding boxes that intersect the input bounding box. Further, in one or more embodiments, the series of actions 900 includes selecting an outer box from the subset of the set of outer boxes. In one or more embodiments, the series of actions 900 includes generating an inflated input by inflating the client device input by an amount of pixels. Further, in one or more embodiments, the series of actions 900 includes selecting the outer box in response to determining that the inflated input intersects the outer box.

[0106] Embodiments of the present disclosure can include or utilize special-purpose or general-purpose computers including computer hardware, such as, for example, one or more processors and system memory, as discussed in greater detail below. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for carrying or storing computer- executable instructions and / or data structures. In particular, one or more of the processes described herein can be implemented at least in part as instructions embodied in a non-transitory computer- readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). Generally, a processor (e.g., a microprocessor) receives instructions, from a non-transitory computer-readable medium (e.g., a memory), and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.

[0107] Computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer. Computer-readable media that store computer- executable instructions are non-transitory computer-readable storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Thus, for example, and without limitation, embodiments of the present disclosure can include at least two distinct computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

[0108] Non-transitory computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid state drives ("SSDs") (e.g., based on RAM), Flash memory, phase- change memory ("PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer- executable instructions or data structures and which can be accessed by a general- purpose or special-purpose computer.

[0109] A "network" is defined as one or more data links that enable the transport of electronic data between computer systems and / or modules and / or other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmission media can include a network and / or data links which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of computer-readable media.

[0110] Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to non-transitory computer-readable storage media (devices) (or vice versa). For example, computer-executable instructions or data structures received by way of network or data link can be buffered in RAM within a network interface module (e.g., a "NIC"), and then eventually transferred to computer system RAM and / or to less volatile computer storage media (devices) at a computer system. Accordingly, it should be understood that non-transitory computer-readable storage media (devices) can be included in computer system components that also (or even primarily) utilize transmission media.

[0111] Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor, cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. In some embodiments, computer-executable instructions are executed by a general purpose computer to transform the general purpose computer into a special purpose computer that implements elements of the present disclosure. Computer-executable instructions can be, for example, binaries, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.

[0112] Those skilled in the art will appreciate that the disclosure can be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, and the like. The disclosure can also be practiced in distributed system environments where local or remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules can be located in both local and remote memory storage devices.

[0113] Embodiments of the disclosure can also be implemented in a cloud computing environment. As used herein, the term "cloud computing" refers generally to the use of a shared pool of configurable computing resources (e.g., networks, servers, storage, processes, applications, etc.) to perform tasks for multiple users. For example, cloud computing can be used to provide on-demand access to shared pools of configurable computing resources (e.g., networks, servers, storage, processes, applications, etc.). The shared pools of configurable computing resources can be rapidly provisioned and released with minimal management effort or service provider interaction, and then scaled out or in accordingly.

[0114] A cloud computing model can be composed of various characteristics such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, and the like. A cloud computing model can also expose various service models such as, for example, Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). A cloud computing model can also be deployed using different deployment models such as, for example, private cloud, community cloud, public cloud, hybrid cloud, and the like. Furthermore, as used herein, the term "cloud computing environment" refers to an environment in which cloud computing is employed.

[0115] Figure 10 A block diagram illustrating an example computing device 1000 that can be configured to perform one or more of the processes described above is shown. It should be understood that one or more computing devices, such as the computing device 1000, can represent the computing devices described above (e.g., the server device(s) 102, the client device(s) 110, and the computing device 1000). In one or more embodiments, the computing device 1000 can be a mobile device (e.g., a mobile phone, a smartphone, a PDA, a tablet, a laptop, a camera, a tracker, a watch, a wearable device, and the like). In some embodiments, the computing device 1000 can be a non-mobile device (e.g., a desktop computer or another type of client device). Moreover, the computing device 1000 can be a server device that includes cloud-based processing and storage capabilities.

[0116] As Figure 10 As shown, computing device 1000 can include one or more processors 1002, memory 1004, storage 1006, input / output interface(s) 1008 (or "I / O interface(s) 1008"), and communication interface(s) 1010, which can be communicatively coupled via a communication infrastructure, such as a bus 1012. While Figure 10 The computing device 1000 is illustrated as including one or more components, but the computing device 1000 can include additional or fewer components in other embodiments. Figure 10 The components illustrated for the computing device 1000 are not intended to be limiting. Additional or alternative components can be used in other embodiments. Furthermore, in certain embodiments, the computing device 1000 includes fewer components than those shown. The components illustrated for the computing device 1000 will now be described in more detail. Figure 10 The components illustrated for the computing device 1000 are not intended to be limiting. Additional or alternative components can be used in other embodiments. Furthermore, in certain embodiments, the computing device 1000 includes fewer components than those shown. The components illustrated for the computing device 1000 will now be described in more detail. Figure 10 The components illustrated for the computing device 1000 are not intended to be limiting. Additional or alternative components can be used in other embodiments. Furthermore, in certain embodiments, the computing device 1000 includes fewer components than those shown. The components illustrated for the computing device 1000 will now be described in more detail.

[0117] In particular embodiments, the processor(s) 1002 include hardware for executing instructions, such as those that make up a computer program. As an example and not by way of limitation, to execute instructions, the processor(s) 1002 can retrieve (or fetch) the instructions from an internal register, an internal cache, the memory 1004, or the storage 1006, decode and execute them, and then dispose of them. Although not shown, the processor(s) 1002 can be multi-core processor(s) that include two or more cores.

[0118] The computing device 1000 includes memory 1004 coupled to the processor(s) 1002. The memory 1004 can be used for storing data, metadata, and programs for execution by the processor(s). The memory 1004 can include one or more of volatile and non-volatile memory such as random access memory ("RAM"), read only memory ("ROM"), solid state disk ("SSD"), flash memory, phase change memory ("PCM"), or other types of data storage. The memory 1004 can be internal or distributed.

[0119] The computing device 1000 includes storage 1006 that includes storage for storing data or instructions. As an example and not by way of limitation, the storage 1006 can include non-transitory storage media as described above. The storage 1006 can include a hard disk drive (HDD), flash memory, a universal serial bus (USB) drive, or a combination of these or other storage devices.

[0120] As shown, the computing device 1000 includes one or more I / O interface(s) 1008, which provide a means for allowing a user to provide input to, receive output from, and otherwise transfer data to and from the computing device 1000. These I / O interface(s) 1008 can include a mouse, a mini- or full-size keyboard or keyboard, a touchscreen, a camera, an optical scanner, a network interface, a modem, other well-known I / O devices, or a combination of such I / O interface(s) 1008. The touchscreen can be activated with a stylus or a finger.

[0121] The I / O interface 1008 can include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In certain embodiments, the I / O interface 1008 is configured to provide graphical data to a display for presentation to a user. The graphical data can be representative of one or more graphical user interfaces and / or any other graphical content serving a particular implementation.

[0122] The computing device 1000 can also include a communications interface 1010. The communications interface 1010 can include hardware, software, or both. The communications interface 1010 provides one or more interfaces for communication (such as packet-based communication) between the computing device and one or more other computing devices or one or more networks. As an example but not by way of limitation, the communications interface 1010 can include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wired network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI. The computing device 1000 can also include a bus 1012. The bus 1012 can include hardware, software, or both that couples components of the computing device 1000 to each other.

[0123] In the foregoing specification, the disclosure has been described with reference to specific exemplary embodiments thereof. Various embodiments of the disclosure have been described with reference to details described herein, accompanying drawings, and specific language evoking the details. The description above and the drawings are illustrative of various embodiments of the disclosure and are not intended to limit the scope or options for claims. Numerous specific details are described to provide a thorough understanding of various embodiments of the disclosure.

[0124] The disclosure can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The described embodiments are to be considered in all respects only as illustrative and not restrictive. For example, the methods described herein can be performed with fewer or additional steps / actions, or in different orders. Also, the steps / actions described herein can be repeated or performed in parallel with each other or with different instances of the same or similar steps / actions. Accordingly, the scope of the application is indicated by the appended claims, rather than by the description above. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.< / shift>

Claims

1. A computer-implemented method, comprising: Determine the set of bounding boxes corresponding to the boundaries of the set of segments within the raster image; In response to a client device input indicating one or more points within a threshold distance of the outer frame, select an outer frame from the outer frame set that corresponds to a segment in the segment set; For displays within the graphical user interface of the client device, a highlighting indication of the outer frame corresponding to the fragment is provided; as well as In response to the selection of the outline via the graphical user interface, a vector path is generated within the vector image based on the outline corresponding to the segment.

2. The computer-implemented method according to claim 1, wherein selecting the outer frame includes: Determine a first bounding box for the outer frame; Determine the second bounding box input for the client device; as well as The outer frame is selected based on the intersection of the first bounding box and the second bounding box.

3. The computer-implemented method according to claim 1 further includes: An object detection model is used to generate an object mask for objects within the raster image using semantic analysis. as well as The object is segmented based on the object mask, and the set of fragments is generated within the raster image using an object segmentation model.

4. The computer-implemented method according to claim 1 further includes: In response to determining that the one or more points are within the threshold distance of an additional bounding box corresponding to an additional fragment in the fragment set, the additional bounding box is selected from the bounding box set; as well as For the display within the graphical user interface of the client device, an additional highlighting indication is provided for the additional outline corresponding to the additional fragment, wherein the additional highlighting indication is visually different from the highlighting indication of the outline.

5. The computer-implemented method according to claim 1, wherein selecting the outer frame includes: In response to determining, using a hit detection operation, that one or more points input by the client device intersect with the outer frame, the outer frame is selected.

6. The computer-implemented method according to claim 1, wherein determining the outer frame set comprises: Generate a binary mask for the segment; Extract one or more contours from the binary mask to determine the boundaries of the segment; as well as The outer frame, including outline polylines corresponding to one or more contours, is generated based on the binary mask.

7. The computer-implemented method of claim 6, wherein selecting the outer frame comprises: Generate an input polyline corresponding to one or more points input by the client device; as well as The outer frame is selected from the set of outer frames based on a comparison between the input polyline and the outer frame polyline with respect to a matching tolerance threshold.

8. The computer-implemented method according to claim 6 further includes using an outer frame matching model to remove redundant points within the outer frame polyline based on the distance between points within the outer frame polyline.

9. A system comprising: One or more memory devices; as well as One or more processors, the one or more processors being configured to cause the system to: For a raster image, a first bounding box corresponding to the boundary of the first segment and a second bounding box corresponding to the boundary of the second segment are determined; Receive client device input from the client device, the client device input indicating a point in the raster image within a threshold distance between the first segment and the second segment; In response to the client device input being within the threshold distance between the first segment and the second segment, a first highlight indication for the first outline and a second highlight indication for the second outline are provided for display within the graphical user interface of the client device; as well as Within the vector image, a vector path with geometry associated with the first outer frame is generated based on the selection of the first outer frame.

10. The system of claim 9, wherein the one or more processors are further configured to cause the system to: Receive additional client device input indicating additional points in the raster image; It is determined that the first segment is not within the threshold distance of the additional point; and Remove the first highlighted indicator from the first outer frame of the display within the graphical user interface.

11. The system of claim 10, wherein the one or more processors are further configured to: determine that the first segment is outside the threshold distance of the additional point by utilizing a hit detection operation to limit the outline and exclude the first segment.

12. The system of claim 10, wherein the one or more processors are further configured to: Receive supplementary client device input, the supplementary client device input indicating the removal of the additional points in the raster image; and In response to the removal of the additional point in the raster image, a first highlight indication of the first outline is provided for display within the graphical user interface of the client device.

13. The system of claim 9, wherein the one or more processors are further configured to generate the vector path in such a way as: Generate a frame polyline including line primitives corresponding to the first outer frame; and Based on the distance between points in the outer frame polyline, remove redundant points within the outer frame polyline.

14. The system of claim 9, wherein the one or more processors are further configured to: Using an object detection model, multiple object masks are generated for multiple objects within the raster image; and The first segment and the second segment are generated within the raster image in the following manner: Using an object segmentation model, objects are segmented based on masks among the multiple object masks to generate a fragment mask set; and Multiple bounding boxes are generated based on the contours extracted from the set of fragment masks.

15. The system of claim 9, wherein the one or more processors are further configured to select the first outer frame and the second outer frame in such a way that: Determine the first bounding box corresponding to the first outer frame; Determine the second bounding box corresponding to the second outer frame; and In response to determining that a third bounding box corresponding to the client device input intersects with the first bounding box and the second bounding box, the first outer box and the second outer box are selected.

16. A non-transitory computer-readable medium storing executable instructions, which, when executed by a processing device, cause the processing device to perform operations, the operations including: Using an object segmentation model to segment one or more objects in a raster image into a set of segments; Generate a set of bounding boxes corresponding to the contours extracted from the set of fragments; In response to client device input, draw one or more points located within a threshold distance of the outer frame, and select an outer frame corresponding to a segment in the segment set from the outer frame set; For display within the graphical user interface of the client device, a highlighting indication of the outer frame corresponding to the fragment is provided; as well as In response to the selection of the outline via the graphical user interface, a vector path is generated within the vector image based on the outline corresponding to the segment.

17. The non-transitory computer-readable medium of claim 16, further comprising: Based on the received instruction to remove a point from one or more of the points, the modified client device input is determined; In response to the modified client device input being within a threshold distance of an additional bounding box corresponding to an additional bounding box in the set of bounding boxes, the additional bounding box is selected from the set of bounding boxes; as well as Additional highlighting indications are provided for the additional outline for display within the graphical user interface of the client device.

18. The non-transitory computer-readable medium of claim 16, wherein selecting the outer frame includes: Determine an input spline corresponding to the input from the client device, the input spline comprising one or more points; The corresponding input polyline is generated by reducing the input spline to line primitives; as well as The outer frame corresponding to the segment is selected based on the comparison between the input polyline and the outer frame.

19. The non-transitory computer-readable medium of claim 16, wherein selecting the outer frame includes: Determine the set of bounding boxes corresponding to the outer frame set; Determine the input bounding box corresponding to the input from the client device; Based on the subset of bounding boxes in the bounding box set that intersect with the input bounding box, select a subset of the outer box set; as well as Select the outer frame from the subset of the outer frame set.

20. The non-transitory computer-readable medium of claim 19, wherein selecting the outer frame from the subset of the set of outer frames comprises: Dilated input is generated by dilating the client device input by pixel amount. as well as The outer frame is selected in response to determining the intersection of the expansion input and the outer frame.

Images