Gating of contextual attention features and convolutional features

Abstract

A method includes determining a mask defining a portion of a perceptual representation, a convolutional feature associated with the perceptual representation, and a contextual attention feature associated with the perceptual representation. The method also includes determining a concatenation of the mask, the convolutional feature, and the contextual attention feature. The method also includes determining a gating value for the portion, wherein determining the gating value includes processing the concatenation using a machine learning model. The method also includes determining a gated convolutional feature based on the convolutional feature and the gating value, and determining a gated contextual attention feature based on the contextual attention feature and the gating value. The method also includes generating a refined value for the portion based on the gated convolutional feature and the gated contextual attention feature.

Description

Technical Field

[0001] The present application relates to the filling of perceptual representations. Background Art

[0002] In-filling a perceptual representation, such as an image, involves replacing a damaged, degraded, removed, and / or otherwise missing portion of the perceptual representation. The replaced portion may be based on, and therefore may be consistent with, the remaining portion of the perceptual representation so that the resulting perceptual representation appears natural, non-synthetic, and / or has other desired qualities. Various machine learning models, algorithms, and / or architectures may be configured to facilitate in-filling of perceptual representations. Summary of the invention

[0003] Filling in a portion of a perceptual representation such as an image can be improved by including a gating layer that is configured to explicitly determine the relative proportions of generated features and replicated features used to determine fine sample values ​​for the portion. Specifically, a mask defining the portion of the perceptual representation, convolutional (generated) features associated with the perceptual representation, and contextual attention (replicated) features associated with the perceptual representation can be concatenated. The concatenation can be processed by a machine learning model, and the output of the machine learning model can be used to determine multiple gating values ​​for the perceptual representation. Multiple gating values ​​can be used to weight convolutional features and contextual attention features to generate gated convolutional features and gated contextual attention features. The gated convolutional features and gated contextual attention features can be processed by a further machine learning model to generate fine sample values ​​for the portion.

[0004] In a first example embodiment, a computer-implemented method may include determining (i) a mask defining a portion of a perceptual representation, (ii) a plurality of convolutional features associated with the perceptual representation, and (iii) a plurality of contextual attention features associated with the perceptual representation. The computer-implemented method may also include determining a concatenation of the mask, the plurality of convolutional features, and the plurality of contextual attention features. The computer-implemented method may additionally include determining a plurality of gating values ​​for the portion of the perceptual representation. Determining the plurality of gating values ​​may include processing the concatenation using a machine learning model. The computer-implemented method may also include determining (i) a plurality of gated convolutional features based on the plurality of convolutional features and the plurality of gating values, and determining (ii) a plurality of gated contextual attention features based on the plurality of contextual attention features and the plurality of gating values. The computer-implemented method may also include generating a plurality of refined values ​​for the portion of the perceptual representation based on the plurality of gated convolutional features and the plurality of gated contextual attention features.

[0005] In a second example embodiment, a system is provided, the system comprising a processor and a non-transitory computer-readable medium having instructions stored thereon, which, when executed by the processor, cause the processor to perform operations. The operations may include determining (i) a mask defining a portion of a perceptual representation, (ii) a plurality of convolutional features associated with the perceptual representation, and (iii) a plurality of contextual attention features associated with the perceptual representation. The operations may also include determining a concatenation of the mask, the plurality of convolutional features, and the plurality of contextual attention features. The operations may additionally include determining a plurality of gating values ​​for the portion of the perceptual representation. Determining the plurality of gating values ​​may include processing the concatenation using a machine learning model. The operations may also include determining (i) a plurality of gated convolutional features based on the plurality of convolutional features and the plurality of gating values, and determining (ii) a plurality of gated contextual attention features based on the plurality of contextual attention features and the plurality of gating values. The operations may also include generating a plurality of refined values ​​for the portion of the perceptual representation based on the plurality of gated convolutional features and the plurality of gated contextual attention features.

[0006] In a third example embodiment, a non-transitory computer-readable medium may have instructions stored thereon that, when executed by a computing device, cause the computing device to perform operations. The operations may include determining (i) a mask defining a portion of a perceptual representation, (ii) a plurality of convolutional features associated with the perceptual representation, and (iii) a plurality of contextual attention features associated with the perceptual representation. The operations may also include determining a concatenation of the mask, the plurality of convolutional features, and the plurality of contextual attention features. The operations may additionally include determining a plurality of gating values ​​for the portion of the perceptual representation. Determining the plurality of gating values ​​may include processing the concatenation using a machine learning model. The operations may also include determining (i) a plurality of gated convolutional features based on the plurality of convolutional features and the plurality of gating values, and determining (ii) a plurality of gated contextual attention features based on the plurality of contextual attention features and the plurality of gating values. The operations may also include generating a plurality of refined values ​​for the portion of the perceptual representation based on the plurality of gated convolutional features and the plurality of gated contextual attention features.

[0007] In a fourth example embodiment, a system may include components for determining (i) a mask defining a portion of a perceptual representation, (ii) a plurality of convolutional features associated with the perceptual representation, and (iii) a plurality of contextual attention features associated with the perceptual representation. The system may also include components for determining a concatenation of the mask, the plurality of convolutional features, and the plurality of contextual attention features. The system may additionally include components for determining a plurality of gating values ​​for the portion of the perceptual representation. Determining the plurality of gating values ​​may include processing the concatenation using a machine learning model. The system may also include components for determining (i) a plurality of gated convolutional features based on the plurality of convolutional features and the plurality of gating values ​​and determining (ii) a plurality of gated contextual attention features based on the plurality of contextual attention features and the plurality of gating values. The system may also include components for generating a plurality of refined values ​​for the portion of the perceptual representation based on the plurality of gated convolutional features and the plurality of gated contextual attention features.

[0008] These and other embodiments, aspects, advantages and alternatives will become apparent to those of ordinary skill in the art by reading the detailed description below and referring appropriately to the accompanying drawings. In addition, this overview and other descriptions and drawings provided herein are intended to illustrate the embodiments only by way of example, and as such, many variations are possible. For example, structural elements and process steps may be rearranged, combined, distributed, eliminated or otherwise changed while remaining within the scope of the claimed embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A computing device according to examples described herein is illustrated.

[0010] Figure 2 A computing system according to examples described herein is illustrated.

[0011] Figure 3 A system according to examples described herein is illustrated.

[0012] Figure 4A , 4C and 4D illustrate perceptual representations according to the examples described herein.

[0013] Figure 4B A mask according to examples described herein is illustrated.

[0014] Figure 5 A system according to examples described herein is illustrated.

[0015] Figure 6 Illustrated is the processing of contextual attention features and convolutional features according to the examples described herein.

[0016] Figure 7 is a flow chart according to examples described herein. DETAILED DESCRIPTION

[0017] Example methods, devices, and systems are described herein. It should be understood that the words "example" and "exemplary" as used herein mean "serving as an example, instance, or illustration." Unless so stated, any embodiment or feature described herein as "example," "exemplary," and / or "illustrative" is not necessarily to be construed as preferred or advantageous over other embodiments or features. Therefore, other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein.

[0018] Accordingly, the example embodiments described herein are not meant to be limiting.It will be readily appreciated that aspects of the present disclosure, as generally described herein and illustrated in the accompanying drawings, may be arranged, substituted, combined, separated, and designed in a variety of different configurations.

[0019] In addition, unless the context suggests otherwise, the features illustrated in each figure may be used in combination with each other. Therefore, the figures should generally be viewed as constituent aspects of one or more overall embodiments, with it being understood that not all illustrated features are necessary for each embodiment.

[0020] In addition, any enumeration of elements, blocks, or steps in this specification or claims is for the purpose of clarity. Therefore, such enumeration should not be interpreted as requiring or implying that these elements, blocks, or steps follow a specific arrangement or are performed in a specific order. Unless otherwise specified, the drawings are not drawn to scale.

[0021] I. Overview

[0022] In various contexts and applications, it may be desirable to replace damaged, degraded, removed, and / or otherwise missing portions of a perceptual representation with sample values ​​that match the rest of the perceptual representation. For example, when the perceptual representation is an image, it may be desirable to fill in missing areas of the image with new pixel values ​​that visually blend with and / or otherwise match the rest of the image. When the perceptual representation is an audio waveform, it may be desirable to fill in missing segments of the audio waveform with new sample values ​​that audibly blend with and / or otherwise match the rest of the audio waveform. A machine learning model may be configured to perform such replacement of portions of a perceptual representation.

[0023] Specifically, the machine learning model can be configured to determine refined sample values ​​through a combination of generation and replication to replace initial sample values ​​for missing portions of the perceptual representation. For example, the machine learning model can be configured to determine (i) convolutional features that provide a basis for synthesizing new sample values ​​and (ii) contextual attention features that provide a basis for copying sample values ​​from other portions of the perceptual representation. Using contextual attention features (replication) in combination with convolutional features (synthesis) can produce perceptual representations that appear natural, unaltered, and / or otherwise have improved quality. However, computing contextual attention features can be computationally expensive due to the relatively large number of mathematical operations involved. Therefore, contextual attention features can be computed at a reduced resolution (e.g., 32×32 instead of 256×256), thereby reducing the associated computational cost.

[0024] Due to the reduced resolution of the context-attention features, the refined sample values ​​determined based on the corresponding context-attention features may appear unnatural, synthetic, and / or otherwise have relatively low quality. For example, a sample value determined based on a context-attention feature representing a straight line may appear blocky and / or may include translation due to the reduced resolution of the context-attention features. In other words, in some cases, when a sample value is replicated using a context-attention feature, the context-attention feature may not preserve the semantic meaning of the underlying feature.

[0025] To compensate for these and other shortcomings, a gating architecture can be provided for explicitly determining the relative proportions of generated and replicated features at different samples in the perceptual representation. The gating architecture can be configured to determine the extent to which contextually-attentive (replicated) features are combined with corresponding convolutional (generated) features so that the resulting refined sample values ​​appear natural, realistic, and / or otherwise of relatively high quality. The relative proportions of generated and replicated features can be determined on a sample-by-sample basis so that, for example, replication is favored for features that suffer relatively insignificant distortion under size scaling, while generation is preferred for features that suffer relatively more significant distortion under size scaling.

[0026] The gating architecture may include a gate configured to determine a plurality of gating values. The gate may be configured to determine a concatenation of a mask indicating a portion of a perceptual representation for which a refined value is to be determined, a convolutional (generated) feature associated with the perceptual representation, and a contextual attention (replicated) feature associated with the perceptual representation. A machine learning model (i.e., a gating model) may be configured to process the concatenation to generate a plurality of gating values. That is, the gating model may be trained to explicitly determine the relative proportions of the convolutional features and the contextual attention features at different samples in the perceptual representation based on their semantic content.

[0027] Multiple gating values ​​can be associated with, for example, a convolutional feature. Therefore, the fusion function can be configured to multiply the convolutional feature and the multiple gating values ​​to generate a gated convolutional feature. In addition, the fusion function can be configured to determine multiple residual gating values ​​for the contextual attention feature by subtracting each gating value from a predetermined scaling value. The scaling value can define the upper limit of the range (e.g., 0 to 1, where the predetermined scaling value corresponds to 1) to which the gating model maps the gating value. Subtracting a given gating value from the predetermined scaling value can therefore indicate the proportion of corresponding samples generated by the corresponding contextual attention feature. Therefore, the fusion function can be further configured to multiply the contextual attention feature and the multiple residual gating values ​​to generate a gated contextual attention feature.

[0028] The fusion function can be configured to combine the gated convolutional features and the gated contextual attention features and provide the combination to a machine learning model (i.e., a filling model) that has been trained to generate refined values ​​for at least the masked portion of the perceptual representation based on the combination. For example, the gated convolutional features and the gated contextual attention features can be combined by determining their sum and / or their concatenation. Since the relative proportions of the generated (convolutional) features and the replicated (contextual attention) features are explicitly represented by the combination, it may not be necessary to train the filling model to implicitly make this determination. Therefore, the size of the filling model can be reduced while improving the overall quality of the refined values ​​generated thereby.

[0029] II. Example Computing Devices and Systems

[0030] Figure 1 An example computing device 100 is illustrated. Computing device 100 is shown in the form of a mobile phone. However, computing device 100 may alternatively be implemented as a laptop computer, a tablet computer, or a wearable computing device (e.g., a watch), among other possibilities. Computing device 100 may include various elements, such as body 102, display 106, and buttons 108 and 110. Computing device 100 may also include one or more cameras, such as front-facing camera 104 and rear-facing camera 112.

[0031] The front-facing camera 104 may be located on a side of the body 102 that generally faces the user when in operation (e.g., on the same side as the display 106). The rear-facing camera 112 may be located on a side of the body 102 opposite the front-facing camera 104. Referring to the cameras as front-facing and rear-facing is arbitrary, and the computing device 100 may include multiple cameras located on various sides of the body 102.

[0032] Display 106 may represent a cathode ray tube (CRT) display, a light emitting diode (LED) display, a liquid crystal (LCD) display, a plasma display, an organic light emitting diode (OLED) display, or any other type of display known in the art. In some examples, display 106 may display a current image captured by front camera 104 and / or rear camera 112, an image that may be captured by one or more of these cameras, an image that was recently captured by one or more of these cameras, and / or a digital representation of a modified version of one or more of these images. Thus, display 106 may act as a viewfinder for the camera. Display 106 may also support touch screen functionality that enables adjustment of settings and / or configuration of any aspect of computing device 100.

[0033] The front camera 104 may include an image sensor and associated optical elements, such as a lens. The front camera 104 may provide a zoom capability, or may have a fixed focal length. In other examples, interchangeable lenses may be used with the front camera 104. The front camera 104 may have a variable mechanical aperture and a mechanical and / or electronic shutter. The front camera 104 may also be configured to capture still images, video images, or both. In addition, the front camera 104 may represent, for example, a single-view camera. The rear camera 112 may be arranged similarly or differently. In addition, one or more of the front camera 104 and / or the rear camera 112 may be an array of one or more cameras.

[0034] One or more of the front camera 104 and / or the rear camera 112 may include or be associated with an illumination component that provides a light field to illuminate the target object. For example, the illumination component may provide a flash or constant illumination of the target object. The illumination component may also be configured to provide a light field that includes one or more of structured light, polarized light, and light with a specific spectral content. In the context of the examples herein, other types of light fields that are known and used to recover a three-dimensional (3D) model from an object are possible.

[0035] The computing device 100 may also include an ambient light sensor that may continuously or from time to time determine the ambient brightness of a scene that may be captured by the cameras 104 and / or 112. In some implementations, the ambient light sensor may be used to adjust the display brightness of the display 106. Additionally, the ambient light sensor may be used to determine an exposure length for one or more of the cameras 104 or 112, or to assist in making such a determination.

[0036] The computing device 100 may be configured to capture images of a target object using the display 106 and the front camera 104 and / or the rear camera 112. The captured images may be a plurality of still images or a video stream. Image capture may be triggered by activating a button 108, pressing a soft key on the display 106, or by some other mechanism. Depending on the implementation, images may be automatically captured at specific time intervals, for example, upon pressing the button 108, under appropriate lighting conditions for the target object, upon moving the digital camera device 100 a predetermined distance, or according to a predetermined capture schedule.

[0037] Figure 2 2 is a simplified block diagram illustrating some components of an example computing system 200. By way of example and not limitation, computing system 200 may be a cellular mobile phone (e.g., a smart phone), a computer (e.g., a desktop, notebook, tablet, or handheld computer), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, a gaming console, a robotic device, a vehicle, or some other type of device. Computing system 200 may represent aspects of computing device 100, for example.

[0038] like Figure 2 As shown, computing system 200 may include communication interface 202, user interface 204, processor 206, data storage device 208, and camera assembly 224, all of which may be communicatively linked together via a system bus, network, or other connection mechanism 210. Computing system 200 may be equipped with at least some image capture and / or image processing capabilities. It should be understood that computing system 200 may represent a physical image processing system, a specific physical hardware platform on which image sensing and processing applications run in software, or other combinations of hardware and software configured to perform image capture and / or processing functions.

[0039] The communication interface 202 may allow the computing system 200 to communicate with other devices, access networks, and / or transport networks using analog or digital modulation. Thus, the communication interface 202 may facilitate circuit-switched and / or packet-switched communications, such as plain old telephone service (POTS) communications and / or Internet Protocol (IP) or other packetized communications. For example, the communication interface 202 may include a chipset and an antenna arranged for wireless communication with a radio access network or access point. Furthermore, the communication interface 202 may take the form of or include a wired interface, such as an Ethernet, Universal Serial Bus (USB), or High Definition Multimedia Interface (HDMI) port. The communication interface 202 may also take the form of or include a wireless interface, such as Wi-Fi, The communication interface 202 may include a global positioning system (GPS) or a wide area wireless interface (e.g., WiMAX or 3GPP Long Term Evolution (LTE)). However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used on the communication interface 202. In addition, the communication interface 202 may include multiple physical communication interfaces (e.g., Wi-Fi interfaces, interface and wide area wireless interface).

[0040] User interface 204 can be used for allowing computing system 200 to interact with the human or non-human user, such as receiving input from the user and providing output to the user.Therefore, user interface 204 can include input components, such as keypad, keyboard, touch-sensitive panel, computer mouse, trackball, joystick, microphone, etc.User interface 204 can also include one or more output components, such as display screen, which can be combined with touch-sensitive panel, for example.Display screen can be based on CRT, LCD and / or LED technology, or other technologies known now or developed later.User interface 204 can also be configured to generate (multiple) audible outputs via loudspeaker, speaker jack, audio output port, audio output device, earphone and / or other similar devices.User interface 204 can also be configured to receive and / or capture (multiple) audible speech, (multiple) noise and / or (multiple) signal by microphone and / or other similar devices.

[0041] In some examples, user interface 204 may include a display that serves as a viewfinder for still camera and / or video camera functions supported by computing system 200. In addition, user interface 204 may include one or more buttons, switches, knobs, and / or dials that facilitate configuration and focusing of camera functions and capture of images. Some or all of these buttons, switches, knobs, and / or dials may be implemented via a touch-sensitive panel.

[0042] Processor 206 may include one or more general purpose processors, such as a microprocessor, and / or one or more special purpose processors, such as a digital signal processor (DSP), a graphics processing unit (GPU), a floating point unit (FPU), a network processor, or an application specific integrated circuit (ASIC). In some cases, the special purpose processor may be capable of image processing, image alignment, and merging images, among other possibilities. Data storage device 208 may include one or more volatile and / or non-volatile storage components, such as magnetic, optical, flash memory, or organic storage devices, and may be integrated with processor 206 in whole or in part. Data storage device 208 may include removable and / or non-removable components.

[0043] The processor 206 may be capable of executing program instructions 218 (e.g., compiled or non-compiled program logic and / or machine code) stored in the data storage device 208 to perform various functions described herein. Therefore, the data storage device 208 may include a non-transitory computer-readable medium having program instructions stored thereon, which, when executed by the computing system 200, causes the computing system 200 to perform any method, process, or operation disclosed in this specification and / or the accompanying drawings. The execution of the program instructions 218 by the processor 206 may cause the processor 206 to use the data 212.

[0044] As an example, program instructions 218 may include an operating system 222 (e.g., an operating system kernel, (multiple) device drivers, and / or other modules) and one or more application programs 220 (e.g., a camera function, an address book, email, web browsing, social networking, an audio-to-text function, a text translation function, and / or a gaming application) installed on computing system 200. Similarly, data 212 may include operating system data 216 and application data 214. Operating system data 216 may be primarily accessible to operating system 222, and application data 214 may be primarily accessible to one or more application programs 220. Application data 214 may be arranged in a file system that is visible or hidden to a user of computing system 200.

[0045] Applications 220 may communicate with operating system 222 via one or more application programming interfaces (APIs). These APIs may facilitate, for example, application 220 reading and / or writing application data 214, sending or receiving information via communication interface 202, receiving and / or displaying information on user interface 204, and the like.

[0046] In some cases, application programs 220 may be referred to simply as "applications." In addition, application programs 220 may be downloaded to computing system 200 through one or more online application stores or application markets. However, application programs may also be installed on computing system 200 in other ways, such as via a web browser or through a physical interface on computing system 200 (e.g., a USB port).

[0047] Camera assembly 224 may include, but is not limited to, an aperture, a shutter, a recording surface (e.g., photographic film and / or an image sensor), a lens, a shutter button, an infrared projector, and / or a visible light projector. Camera assembly 224 may include components configured to capture images in the visible light spectrum (e.g., electromagnetic radiation having a wavelength of 380-700 nanometers) and components configured to capture images in the infrared light spectrum (e.g., electromagnetic radiation having a wavelength of 701 nanometers-1 millimeter). Camera assembly 224 may be controlled, at least in part, by software executed by processor 206.

[0048] III. Example Replacement and Refinement of Perceptual Representations

[0049] Figure 3 An example system configured to replace and / or refine one or more portions of a perceptual representation is illustrated. In this example, system 320 includes a coarse machine learning model 304 and a fine machine learning model 308, each of which can be based on and / or include one or more machine learning algorithms, architectures, and / or components. For example, coarse machine learning model 304 and fine machine learning model 308 can each include a convolutional neural network (CNN) configured to apply various convolution and / or deconvolution operations.

[0050] Coarse machine learning model 304 can be configured to generate coarse perceptual representation 306 based on perceptual representation 300 and mask 302. Perceptual representation 300 can be a two-dimensional image, a three-dimensional image, a point cloud, a waveform, text data, and / or other types of data that can be processed by a machine learning model. Mask 302 can indicate one or more portions of perceptual representation 300 to be removed and replaced with new values, and / or one or more portions of perceptual representation 300 to remain unmodified. Mask 302 can be manually specified by a user, determined using a machine learning model, and / or determined using a rule-based algorithm, among other possibilities.

[0051] In one example, the perceptual representation 300 may be an image, and the mask 302 may indicate areas of the image to be removed and filled with new values ​​that are visually consistent with the remaining (unmodified) contents of the image. In another example, the perceptual representation 300 may be an audio waveform, and the mask 302 may indicate segments of the audio waveform to be removed and replaced with new values ​​that are audibly consistent with the remaining contents of the audio waveform.

[0052] The coarse machine learning model 304 may be configured to replace initial sample values ​​within the portion of the perceptual representation 300 indicated by the mask 302 with the coarse sample values, thereby generating a coarse perceptual representation 306. For example, the coarse machine learning model 304 may be configured to generate the coarse sample values ​​by generating a latent space representation of the unmasked (unmodified) portion of the perceptual representation 300 using an encoder (e.g., a neural network arranged as an encoder). Based on the latent space representation, the coarse machine learning model 304 may be configured to generate coarse sample values ​​of the masked portion of the perceptual representation 300 using a decoder (e.g., a neural network arranged as a decoder). The coarse sample values ​​may be "coarse" in that (i) their degree of visual, auditory, textual, and / or other consistency with the unmasked portion of the perceptual representation 300 may be below a threshold consistency value, (ii) they may not be perceived as real / natural by a human observer, and / or (iii) they may not be perceived as real / natural by a machine learning-based discriminator, among other possibilities.

[0053] Fine machine learning model 308 may be configured to generate fine perceptual representation 310 based on coarse perceptual representation 306. Specifically, fine machine learning model 308 may be configured to update coarse sample values ​​within the portion of perceptual representation 300 indicated by mask 302 with fine sample values, thereby generating fine perceptual representation 310.

[0054] The fine machine learning model 308 may generate fine sample values ​​by, for example, generating (i) a plurality of context-attention features and (ii) a plurality of convolutional features, wherein each of the plurality of context-attention features and the plurality of convolutional features may be generated using a corresponding encoder based on coarse sample values ​​of an unmasked portion of the perceptual representation 300 and / or a masked portion of the perceptual representation 300. Based on (i) the plurality of context-attention features and (ii) the plurality of convolutional features, the fine machine learning model 308 may be configured to generate fine sample values ​​of the masked portion of the perceptual representation 300 using a decoder. The fine sample values ​​may be "refined" in that (i) they may be visually, auditorily, textually, and / or otherwise consistent with the unmasked portion of the perceptual representation 300 above a threshold consistency value, (ii) they may be perceived as realistic / natural by a human observer, and / or (iii) they may be perceived as realistic / natural by a machine learning-based discriminator, among other possibilities. For example, the fine sample values ​​may be perceptually improved at least relative to the coarse sample values.

[0055] Figure 4A , 4B4C and 4D provide visual examples of perceptual representation 300, mask 302, coarse perceptual representation 306, and fine perceptual representation 310, respectively. For example, perceptual representation 300 may be a two-dimensional (e.g., red, green, blue (RGB)) image, and may include representations of subject 400 (a first person), subject 404 (a second person), and scene 402 (a mountain) therein. For example, perceptual representation 300 may be an image captured by subject 400 (e.g., a "selfie"), and the presence of subject 404 in the image may be undesirable.

[0056] Mask 302 may be generated to indicate that the representation of subject 404 is to be removed from perceptual representation 300 and replaced by pixel values ​​that are visually consistent with the rest of the image. Mask 302 may be, for example, a binary image in which a first value (e.g., 0, represented by black) is used to specify pixel values ​​to remain unmodified (i.e., unmasked portions) and a second value (e.g., 1, represented by white) is used to specify pixel values ​​to be replaced and / or modified (i.e., masked portions). The portion of mask 302 corresponding to subject 404 may be determined, for example, by a machine learning model configured to determine a segmentation map for identifying pixels in perceptual representation 300 associated with subject 404.

[0057] like Figure 4C As shown, reference box 406 may be overlaid on coarse perceptual representation 306 to approximately indicate portions of perceptual representation 300 that are masked by mask 302 and, therefore, modified by coarse machine learning model 304. Reference box 406 is not part of coarse perceptual representation 306 and is shown for reference purposes only. Specifically, all portions of coarse perceptual representation 306 outside of reference box 406 (and / or outside of the white portions of mask 302) may be identical to those portions of perceptual representation 300. Portions inside reference box 406 (and / or inside of the white portions of mask 302) may include new pixel values ​​that roughly replace the representation of subject 404 with lines and textures that approximate the previously hidden portions of mountain 402. Thus, reference box 406 includes therein various lines and / or textures that are similar to lines and / or textures that represent mountain 402 in other portions of the image but that do not accurately fill in the previously hidden portions of mountain 402.

[0058] like Figure 4D As shown, refined perceptual representation 310 includes image content within reference box 406 of coarse perceptual representation 306 that has been refined to more accurately fill in previously hidden portions of mountain 402. Thus, refined perceptual representation 310 provides a realistic and / or visually plausible replacement for the representation of subject 404 with lines, textures, and / or other features that complete the representation of mountain 402.

[0059] However, when context-attention features are determined at a relatively low resolution (e.g., 32×32 pixels, and / or another resolution low enough to allow operation on a computationally constrained computing device), some portions of the lines, textures, and / or other features that complete the representation of mountain 402 may appear jagged, pixelated, and / or otherwise unnatural. Accordingly, a gating architecture may be provided as part of refined machine learning model 308 to avoid such undesirable artifacts in refined perceptual representation 310 while allowing context-attention features to be computed at a relatively low resolution.

[0060] IV. Example System for Gating Contextual Attention Features and Convolutional Features

[0061] Figure 5 An example system for gating contextual attention features and convolutional features is illustrated. Specifically, Figure 5 A system 506 is illustrated that is configured to generate a refined value 540 based on a perceptual representation 500. The perceptual representation 500 may correspond to the coarse perceptual representation 306, and the system 506 may represent and / or form a part of the refined machine learning model 308. The system 506 includes a convolutional neural network 508, a contextual attention calculator 512, a gating 518 including a splicer 520 and a machine learning model 524, a fusion function 536, and a machine learning model 538.

[0062] The convolutional neural network 508 may be configured to generate convolutional features 510 based on the perceptual representation 500. For example, each respective convolutional feature of the convolutional features 510 may be generated by processing the perceptual representation 500 using a corresponding convolutional kernel defined by the convolutional neural network 508. For example, when the perceptual representation 500 is an image, each convolutional feature of the convolutional features 510 may be a two-dimensional feature map. The convolutional features 510 may be divided into a plurality of feature vectors, each respective feature vector in the plurality of feature vectors representing the information content of at least one corresponding sample of the perceptual representation 500. Thus, the values ​​of the plurality of feature vectors along a given dimension thereof may define a corresponding convolutional feature of the convolutional features 510.

[0063] Convolutional features 510 may be determined for at least one or more portions of perceptual representation 500 marked as replaced and / or refined by mask 516. In some implementations, convolutional features 510 may be determined for all portions of perceptual representation 500. Mask 516 may correspond to Figure 3The convolution features 510 may be based on (i) coarse sample values ​​associated with one or more (masked) portions of the perceptual representation marked as replaced and / or refined by the mask 516, and / or (ii) initial sample values ​​associated with unmasked portions of the perceptual representation 500. For example, a convolution kernel with a relatively wide receptive field may be used to generate the convolution features 510 based on the coarse sample values ​​of the masked portions and the initial sample values ​​of the unmasked portions. The convolution features 510 may provide a basis for generating new sample values ​​for masked samples of the perceptual representation 500, rather than copying sample values ​​from the unmasked portions of the perceptual representation 500.

[0064] Contextual attention calculator 512 may be configured to generate contextual attention features 514 based on perceptual representation 500. Each respective contextual attention feature of contextual attention features 514 may be associated with at least one corresponding sample of perceptual representation 500. Contextual attention features 514 may be determined for samples of perceptual representation 500 marked as replaced and / or refined by mask 516, and / or for samples of the entire perceptual representation 500. Each respective contextual attention feature of contextual attention features 514 may be generated by comparing at least one feature vector associated with at least one corresponding sample with a plurality of other feature vectors associated with corresponding other samples of perceptual representation 500.

[0065] For example, each respective context-attention feature may be based on a comparison of (i) a feature vector associated with the corresponding sample(s) and (ii) a plurality of other feature vectors associated with corresponding other samples of the unmasked portion of the perceptual representation 500. In some cases, each respective context-attention feature may be based on a comparison of (i) a feature vector associated with the corresponding sample(s) and (ii) each feature vector associated with each sample of the unmasked portion of the perceptual representation 500. The comparison may be made with the unmasked portion of the perceptual representation 500 rather than the masked portion because the masked portion contains sample values ​​that may have been synthesized by the coarse machine learning model 304 (and thus may not appear realistic or natural), whereas the unmasked portion contains sample values ​​that have not been altered by the coarse machine learning model 304.

[0066] The comparison performed by the contextual attention calculator 512 may include an inner product between (i) a feature vector associated with the corresponding sample and (ii) each respective feature vector of the other feature vectors, followed by a softmax function (e.g., a scaled softmax) along a dimension corresponding to the plurality of other feature vectors. Alternatively or additionally, the comparison performed by the contextual attention calculator 512 may include a convolution between (i) two or more feature vectors associated with two or more corresponding samples and (ii) each respective set of two or more feature vectors of the other feature vectors, followed by a softmax function along a dimension corresponding to the plurality of other feature vectors. Thus, each respective contextual attention feature of the contextual attention features 514 may represent a degree of similarity between (i) a feature vector (s) associated with at least one corresponding sample and (ii) each of a plurality of other feature vectors associated with other samples of the perceptual representation 500. The convolved features 510 may provide a basis for copying sample values ​​from an unmasked portion of the perceptual representation 500 to a masked sample of the perceptual representation 500, rather than generating new sample values ​​for the masked sample of the perceptual representation 500.

[0067] For example, when the perceptual representation 500 is an image, each of the context-attention features 514 may represent the similarity between (i) corresponding pixels and / or pixel patches in the image (e.g., pixels and / or pixel patches in the masked portion and having coarse sample values ​​(multiple) assigned to it, and / or pixels and / or pixel patches in the unmasked portion and having original, unmodified sample values ​​(multiple)) and (ii) pixels and / or pixel patches in the unmasked portion of the image. For example, an image may be associated with x*y feature vectors corresponding to the masked image portion (thus representing coarse sample values) and i*j feature vectors corresponding to the unmasked image portion, where * represents multiplication, x and i correspond to the horizontal dimension of the image, and y and j correspond to the vertical dimension of the image. Each of the x*y and i*j feature vectors may be associated with one or more corresponding pixels. Thus, for example, determining a contextual attention feature corresponding to a particular feature vector among the x*y feature vectors may involve determining a dot product between (i) a given feature vector and (ii) each of the i*j feature vectors, followed by a softmax function with respect to the resulting i*j dot products, thereby generating a contextual attention feature having i*j values. Determining a contextual attention feature corresponding to a particular feature vector among the i*j feature vectors may involve determining a dot product between (i) a given feature vector and (ii) each of the i*j feature vectors, followed by a softmax function with respect to the resulting i*j dot products, thereby generating a contextual attention feature having i*j values.

[0068] The feature vector used to determine the contextual attention features 514 can be generated by processing the perceptual representation 500 using one or more convolution kernels defined by the contextual attention calculator 512 (e.g., as discussed above with respect to the convolution features 510). In some cases, at least a portion of the convolution kernels provided by the convolutional neural network 508 can be shared with the contextual attention calculator 512. In other cases, the convolutional neural network 508 and the contextual attention calculator 512 can each use different sets of convolution kernels. In an example, the contextual attention calculator 512 can include (and therefore the contextual attention features 514 can be based on) aspects of a global attention mechanism and / or model, a local attention mechanism and / or model, a hard attention mechanism and / or model, a soft attention mechanism and / or model, and / or a self-attention mechanism and / or model.

[0069] The stitcher 520 can be configured to generate a stitching 522 based on the convolutional features 510, the contextual attention features 514, and the mask 516. For example, each of the convolutional features 510, the contextual attention features 514, and the mask 516 can share at least one dimension, or can be scaled to a shared dimension, and can be stacked together, where the shared dimensions are aligned. For example, each of the convolutional features 510, the contextual attention features 514, and the mask 516 can be a tensor with a shared width and height, and can be stacked together by depth to form the stitching 522.

[0070] The machine learning model 524 can be configured to process the splice 522 to generate the gating value 526. Therefore, the machine learning model 524 can alternatively be referred to as a gating model. The machine learning model 524 can be, for example, a CNN that defines a filter with a depth equal to the depth of the splice 522. Therefore, the machine learning model 524 can be configured to reduce the three-dimensional tensor representing the splice 522 to a two-dimensional matrix representing the gating value 526 along its depth dimension.

[0071] Rather than being used to explicitly select a masked subset of the convolutional features 510 and / or the contextual attention features 514, the mask 516 can form part of the stitching 522 to allow the machine learning model 524 to adjust the portion of the perceptual representation 500 to be filled in. That is, the machine learning model 524 can be trained to "trust" different portions of the mask 516 to varying degrees, thereby allowing the use of the gating value 526 as part of the filling process to implicitly correct a mask that is based on, for example, an inaccurate segmentation map and / or an inaccurate manually specified contour. For example, the machine learning model 524 can be configured to determine, based on the stitching 522, that a particular unmasked portion of the perceptual representation 500 is to be regenerated, rather than keeping that portion without modification, and can therefore assign a relatively high gating value to that particular unmasked portion.

[0072] The machine learning model 524 can be configured to determine the gating values ​​526 along a predetermined range. For example, each gating value 526 can be selected from a range of zero to a scaling value 530 (e.g., a predetermined value, such as 1). Thus, the output layer of the machine learning model 524 can be configured to map the results of its earlier layers to a predetermined range (e.g., to a range of 0 to 1 using a sigmoid function).

[0073] The gate value 526 may indicate the relative contribution of the convolution feature 510 to the refined value 540 in a per-sample manner. Therefore, the fusion function 536 may be configured to multiply the convolution feature 510 by the gate value 526 to generate the gated convolution feature 532. In addition, the fusion function 536 may be configured to subtract each gate value 526 from the scaled value 530 to generate a residual gated value 528. The residual gated value 528 may indicate the relative contribution of the contextual attention feature 514 to the refined value 540 in a per-sample manner. Therefore, the fusion function 536 may be configured to multiply the contextual attention feature 514 by the residual gated value 528 to generate the gated contextual attention feature 534.

[0074] In other implementations, the gate values ​​526 may instead correspond to the contextual attention features 514 and may be multiplied therewith to generate the gated contextual attention features 534. Thus, the residual gate values ​​528 may instead correspond to the convolutional features 510 and may be multiplied therewith to generate the gated convolutional features 532. However, for a given image, training the machine learning model 524 in this alternative arrangement may produce a nearly equivalent set of gated convolutional features 532 and gated contextual attention features 534.

[0075] The fusion function 536 can also be configured to add the gated convolutional features 532 to the gated contextual attention features 534 and provide the resulting sum as an input to the machine learning model 538. The machine learning model 538 can be configured to generate a refined value 540 based on the output of the fusion function 536. Therefore, the machine learning model 538 can be alternatively referred to as a filling model. In some implementations, the fusion function 536 can be configured to concatenate the gated convolutional features 532 and the gated contextual attention features 534 and provide the concatenation as an input to the machine learning model 538 instead of adding the gated convolutional features 532 and the gated contextual attention features 534.

[0076] Each respective gating value of gating values ​​526, and thus each respective residual gating value of residual gating values ​​528, may be associated with at least one corresponding sample of perceptual representation 500, and thus with at least one corresponding fine value of refinement value 540. Thus, the respective gating values ​​may indicate the relative contribution of the respective convolutional features of convolutional features 510 to the respective fine value(s), and the respective residual gating values ​​may indicate the relative contribution of the respective contextual attention features of contextual attention features 514 to the respective fine value(s). The respective gating values ​​and their respective residual gating values ​​sum to scaled value 530. Thus, the respective gating values ​​indicate how much of the respective fine value is produced by the generation of new values ​​(based on convolutional features 510), and how much of the respective fine value is produced by the copying of other portions of perceptual representation 500 (based on contextual attention features 514).

[0077] Because the gate value 526 is generated by the machine learning model 524, the machine learning model 524 can be trained to determine the relative per-sample weighting between the replicated features and the generated features. Specifically, the machine learning model 524 can be explicitly trainable to determine the per-sample weighting as part of the encoding of the perceptual representation 500, rather than allowing the machine learning model 538 to implicitly select the relative contributions of the convolutional features 510 and the contextual attention features 514 as part of the decoding process to determine the refined value 540. Thus, the latent space representation of the perceptual representation 500 (e.g., the sum and / or concatenation of features 532 and 534) can be based on the per-sample weighting. This can allow the machine learning model 538 to include fewer parameters, thereby reducing the size of the system 506, while improving the quality of the resulting refined perceptual representation.

[0078] The system 506 can be additionally or alternatively adapted to operate at multiple different size scales. For example, the convolutional neural network 508, the contextual attention calculator 512, the splicer 520, the gating 518, and / or the fusion function 536 can be configured to process and / or generate corresponding data at least for a first size scale and a second size scale different from the first size scale. Therefore, the system 506 can generate corresponding instances of the convolutional features 510, the contextual attention features 514, the splicing 522, the gating value 526, the residual gating value 528, the gated convolutional features 532, and / or the gated contextual attention features 534 at the first size scale and the second size scale. The machine learning model 538 can therefore be configured to generate a refined value 540 based on the gated convolutional features 532 of the first size scale, the gated convolutional features 532 of the second size scale, the gated contextual attention features 534 of the first size scale, and / or the gated contextual attention features 534 of the second size scale.

[0079] System 506 and / or aspects thereof may be implemented as part of various architectures configured to generate refined sample values ​​based on (i) features generated / synthesized using convolution or other similar techniques and (ii) features copied from an unmodified portion of a perceptual representation using contextual attention or other similar techniques. For example, system 506 and / or aspects thereof may be implemented as part of the architecture discussed in (i) a paper entitled “Generative Image Inpainting with Contextual Attention” by Jiahui Yu et al. and published as arXiv:1801.07892, (ii) a paper entitled “Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting” by Zili Yi et al. and published as arXiv:2005.09704, and / or (iii) a paper entitled “MUSICAL: Multi-Scale Image Contextual Attention Learning for Inpainting” by Ning Wang et al. and published in Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19), among other possibilities.

[0080] Figure 6 A visual illustration and example dimensions of data processed and generated by system 506 are provided. Specifically, perceptual representation 500 is shown as having a width U, a height V, and a depth Y. For example, in the case of an RGB image, Y may be equal to 3. Some perceptual representations may have a width of 100, a height of 100, and a depth of 100. Figure 6 A mask 516 corresponding to the perceptual representation 500 has a width U, a height V, and a depth 1. Each of the Y dimensions of the perceptual representation 500 can be masked with a single depth layer of the mask 516. A portion of the mask 516 is shown in white to illustrate the corresponding portion of the perceptual representation 500 to be removed and / or filled, and the remainder of the mask 516 is shown in black to illustrate the portion of the perceptual representation 500 that is not modified.

[0081] Convolutional features 510 and contextual attention features 514 can each have a width W, a height H, and a depth D. In some cases, width W and height H can be equal to width U and height V, respectively. Thus, each respective convolutional feature vector of convolutional features 510 can be associated with and / or represent a corresponding sample of perceptual representation 500, and each respective contextual attention feature vector of contextual attention features 514 can be associated with and / or represent a corresponding sample of perceptual representation 500.

[0082] In other cases, the width W and the height H may be different from the width U and the height V, respectively. In one example, the width W and the height H may be smaller than the width U and the height V, respectively, and may correspond to the entirety of the UxV perceptual representation 500. Thus, each respective convolutional feature vector of the convolutional features 510 may be associated with and / or represent two or more corresponding samples of the perceptual representation 500, and each respective contextual attention feature vector of the contextual attention features 514 may be associated with and / or represent two or more corresponding samples of the perceptual representation 500. In another example, the width W and the height H may be smaller than the width U and the height V, respectively, and may correspond to a masked portion of the UxV perceptual representation 500. Thus, each respective convolutional feature vector of the convolutional features 510 may be associated with and / or represent one or more corresponding samples of the masked portion of the perceptual representation 500, and each respective contextual attention feature vector of the contextual attention features 514 may be associated with and / or represent one or more corresponding samples of the masked portion of the perceptual representation 500.

[0083] In an implementation where the gated convolutional features 532 are added to the gated contextual attention features 534 via the fusion function 536, the convolutional features 510 and the contextual attention features 514 may have the same depth dimension D. In an implementation where the gated convolutional features 532 are concatenated with the gated contextual attention features 534 via the fusion function 536, the convolutional features 510 may have the same depth dimension D as the contextual attention features 514. CONTEXTUAL Different depth dimensions D CONVOLUTIONAL .

[0084] Splice 522 may have a width W, a height H, and a depth (2D+1). In implementations where the width W and height H of convolutional features 510 and / or contextual attention features 514 are different from the width U and height V of mask 516, mask 516 may be resized (e.g., by splicer 520) from UxVx1 to WxHx1. In implementations where the width W and height H of convolutional features 510 and / or contextual attention features 514 are different from the width U and height V of mask 516, mask 516 may be resized (e.g., by splicer 520) from UxVx1 to WxHx1. CONTEXTUALDifferent depth dimensions D CONVOLUTIONAL In the implementation of the present invention, the stitching 522 can be changed to have a width W, a height H and a depth (D CONVOLUTIONAL +D CONTEXTUAL +1).

[0085] The gate value 526 and the residual gate value 528 may each have a width W, a height H, and a depth 1. Accordingly, the machine learning model 524 may include a model having a depth of (2D+1) or (D CONVOLUTIONAL +D CONTEXTUAL +1). Thus, the same set of gating values ​​526 may be used to gate each of the D depth layers of the convolutional features 510, and the same set of residual gating values ​​528 may be used to gate each of the D depth layers of the contextual attention features 514. In implementations where the convolutional features 510 and the contextual attention features 514 have the same depth dimension D, the gated convolutional features 532 and the contextual attention features 534 may each have a width W, a height H, and a depth D. In implementations where the convolutional features 510 have the same depth dimension D as the contextual attention features 514, the gated convolutional features 532 and the contextual attention features 534 may each have a width W, a height H, and a depth D. CONTEXTUAL Different depth dimensions D CONVOLUTIONAL In an implementation of , the gated convolutional feature 532 may have a width W, a height H, and a depth D CONVOLUTIONAL , and the contextual attention feature 534 may have a width W, a height H, and a depth D CONTEXTUAL .

[0086] Alternatively, in some implementations, the gated value 526 and the residual gated value 528 may each have a width W, a height H, and a depth D different from 1. GATE For example, depending on the implementation, D GATE Can be equal to (2D+1) or (D CONVOLUTIONAL +D CONTEXTUAL +1). Accordingly, depending on the implementation, the machine learning model 524 may include (2D+1) or (D CONVOLUTIONAL +D CONTEXTUAL +1) filters, each with a depth of 1. Therefore, the D (or D) of the convolutional features 510 can be gated using the corresponding subset of gating values ​​526. CONVOLUTIONAL ) depth layers, and the corresponding subset of the residual gating values ​​528 can be used to gate the D (or D CONTEXTUAL ) depth layers. The gated convolutional features 532 may have a width W, a height H, and a depth D (or D CONVOLUTIONAL ), the gated contextual attention feature 534 may have a width W, a height H, and a depth D (or D CONTEXTUAL ).

[0087] The fine perceptual representation 600 may have a width U, a height V, and a depth Y. At least a portion of the fine perceptual representation 600 (eg, the portion indicated by the mask 516) may include the fine value 540. The fine perceptual representation 600 may correspond to Figure 3 310 of the refined perceptual representation. The refined perceptual representation 600 may thus correspond to the perceptual representation 500 whose masked portions are replaced by refined sample values. The gating architecture provided by the system 506 may allow the refined perceptual representation 600 to be of higher quality (e.g., visually, aurally, textually, etc.) and / or appear more natural (rather than synthetic) than a refined perceptual representation generated without using such a gating architecture.

[0088] V. Additional Sample Operations

[0089] Figure 7 A flow diagram of operations related to generating a refined value for a portion of a perceptual representation is illustrated. These operations may be performed by computing device 100, computing system 200, system 320, and / or system 506, among other possibilities. Figure 7 The embodiments of the present invention may be simplified by removing any one or more features shown therein. In addition, these embodiments may be combined with any of the features, aspects and / or implementations described in any previous figures or otherwise herein.

[0090] Block 700 may involve determining (i) a mask defining a portion of a perceptual representation, (ii) a plurality of convolutional features associated with the perceptual representation, and (iii) a plurality of contextual attention features associated with the perceptual representation.

[0091] Block 702 may involve determining a concatenation of a mask, a plurality of convolutional features, and a plurality of contextual attention features.

[0092] Block 704 may involve determining a plurality of gating values ​​for the portion of the perceptual representation.Determining the plurality of gating values ​​may include processing the splicing using a machine learning model.

[0093] Block 706 may involve determining (i) a plurality of gated convolutional features based on the plurality of convolutional features and the plurality of gating values, and determining (ii) a plurality of gated contextual attention features based on the plurality of contextual attention features and the plurality of gating values.

[0094] Block 708 may involve generating a plurality of refined values ​​for the portion of the perceptual representation based on the plurality of gated convolutional features and the plurality of gated contextual attention features.

[0095] In some embodiments, the portion of the perceptual representation may include a plurality of samples. For each respective sample in the plurality of samples, a corresponding coarse sample value may be generated by the coarse machine learning model. Each of the plurality of convolutional features and the plurality of contextual attention features may be determined based on the corresponding coarse sample value of each respective sample.

[0096] In some embodiments, each corresponding sample in the plurality of samples may be associated with (i) a corresponding convolutional feature in a plurality of convolutional features representing a generated sample value of the corresponding sample and (ii) a corresponding contextual attention feature in a plurality of convolutional features representing a replicated sample value of the corresponding sample. Determining a plurality of gating values ​​may include determining a corresponding gating value in a plurality of gating values ​​for each corresponding sample. Determining (i) a plurality of gated convolutional features may include determining a corresponding gated convolutional feature in a plurality of gated convolutional features for each corresponding sample, and determining (ii) a plurality of gated contextual attention features may include determining a corresponding gated contextual attention feature in a plurality of gated contextual attention features for each corresponding sample. Generating a refined value for the portion of the perceptual representation may include generating, for each corresponding sample, a corresponding refined value in a plurality of refined values ​​based on the corresponding gated convolutional feature and the corresponding gated contextual attention feature.

[0097] In some embodiments, determining the plurality of gating values ​​may include determining a convolution of the concatenation and one or more convolution kernels of the machine learning model.

[0098] In some embodiments, determining (i) a plurality of gated convolutional features may include determining a product of the plurality of convolutional features and a plurality of gating values.

[0099] In some embodiments, determining (ii) multiple gated contextual attention features may include determining multiple residual gating values ​​based on multiple differences between a predetermined value and multiple gating values, and determining the product of (i) multiple contextual attention features and (ii) multiple residual gating values.

[0100] In some embodiments, each respective gating value in the plurality of gating values ​​may be generated by mapping to a range between zero and a predetermined value by a machine learning model.

[0101] In some embodiments, generating the plurality of refined values ​​may include determining a sum of a plurality of gated convolutional features and a plurality of gated contextual attention features, and generating the plurality of refined values ​​based on the sum.

[0102] In some embodiments, generating the plurality of refined values ​​may include determining a second concatenation of the plurality of gated convolutional features and the plurality of gated contextual attention features, and generating the plurality of refined values ​​based on the second concatenation.

[0103] In some embodiments, the portion of the perceptual representation may be a first portion of the perceptual representation. The mask may include a first region indicating that the first portion of the perceptual representation is to be replaced with the plurality of refined values, and a second region defining a second portion of the perceptual representation and indicating that the second portion of the perceptual representation remains unmodified.

[0104] In some embodiments, the perceptual representation may include one or more of: (i) a two-dimensional image, (ii) a three-dimensional image, (iii) a point cloud, (iv) a waveform, or (v) textual data.

[0105] In some embodiments, a second mask defining the portion of the perceptual representation at a second size scale may be determined, the second size scale being different from a first size scale associated with the mask, the plurality of convolutional features, and the plurality of contextual attention features. A plurality of second convolutional features associated with the perceptual representation at a second size scale may be determined. A plurality of second contextual attention features associated with the perceptual representation at a second size scale may be determined. A further concatenation of the second mask, the plurality of second convolutional features, and the plurality of second contextual attention features may be determined. A plurality of second gating values ​​at a second size scale may be determined for the portion of the perceptual representation. Determining the plurality of second gating values ​​may include processing the further concatenation using a second machine learning model associated with the second size scale. A plurality of second gated convolutional features at a second size scale may be determined based on the plurality of second convolutional features and the plurality of second gating values. A plurality of second gated contextual attention features at a second size scale may be determined based on the plurality of second contextual attention features and the plurality of second gating values. A plurality of refined values ​​for the portion of the perceptual representation may be generated further based on the plurality of second gated convolutional features and the plurality of second gated contextual attention features.

[0106] In some embodiments, a machine learning model can be trained to generate multiple gating values, and the training of the machine learning model can include obtaining a training perceptual representation and removing a portion of the training perceptual representation. Multiple refined values ​​can be generated for the removed portion of the training perceptual representation based on processing by the machine learning model of a concatenation of: (i) a training mask defining the removed portion of the training perceptual representation, (ii) multiple training convolutional features associated with the training perceptual representation, and (iii) multiple training contextual attention features associated with the training perceptual representation. A loss value can be determined using a loss function based on a comparison of the multiple refined values ​​of the removed portion with corresponding values ​​of the training perceptual representation. One or more parameters of the machine learning model can be adjusted based on the loss value and the loss function.

[0107] VI. Conclusion

[0108] The present disclosure is not limited to the specific embodiments described in this application, which are intended to illustrate various aspects. It will be apparent to those skilled in the art that many modifications and variations may be made without departing from the scope thereof. In addition to the methods and apparatus described herein, it will be apparent to those skilled in the art that functionally equivalent methods and apparatus within the scope of the present disclosure will be apparent from the foregoing description. Such modifications and variations are intended to fall within the scope of the appended claims.

[0109] The above detailed description describes various features and operations of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the accompanying drawings, similar symbols generally identify similar components unless the context indicates otherwise. The example embodiments described herein and in the accompanying drawings are not meant to be limiting. Other embodiments may be utilized and other changes may be made without departing from the scope of the subject matter presented herein. It will be readily understood that the various aspects of the present disclosure, as generally described herein and illustrated in the accompanying drawings, may be arranged, replaced, combined, separated, and designed in a variety of different configurations.

[0110] With respect to any or all message flow diagrams, scenarios, and flow diagrams in the accompanying drawings, and as discussed herein, each step, block, and / or communication may represent the processing of information and / or the transmission of information, according to example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, the operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages may not be performed in the order shown or discussed, including substantially simultaneously or in reverse order, depending on the functionality involved. In addition, more or fewer blocks and / or operations may be used with any message flow diagrams, scenarios, and flow diagrams discussed herein. Figure 1 The present invention relates to a method for carrying out the present invention and the method for carrying out the present invention. The present invention relates to a method for carrying out the present invention and the method for carrying out the present invention.

[0111] The steps or blocks representing information processing may correspond to circuits that may be configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, the blocks representing information processing may correspond to modules, fragments, or a portion (including associated data) of program code. The program code may include one or more instructions executable by a processor for implementing specific logical operations or actions in a method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including a random access memory (RAM), a disk drive, a solid-state drive, or other storage medium.

[0112] Computer readable media may also include non-transitory computer readable media, such as computer readable media for short-term storage of data, such as register memory, processor cache, and RAM. Computer readable media may also include non-transitory computer readable media for long-term storage of program code and / or data. Therefore, computer readable media may include secondary or permanent long-term storage devices, such as read-only memory (ROM), optical or magnetic disks, solid-state drives, compact disk read-only memory (CD-ROM). Computer readable media may also be any other volatile or non-volatile storage system. For example, computer readable media may be considered to be computer readable storage media, or tangible storage devices.

[0113] In addition, the steps or blocks representing one or more information transfers may correspond to information transfers between software and / or hardware modules in the same physical device. However, other information transfers may be performed between software modules and / or hardware modules in different physical devices.

[0114] The specific arrangements shown in the accompanying drawings should not be considered restrictive. It should be understood that other embodiments may include more or less of each element shown in a given figure. In addition, some illustrated elements may be combined or omitted. In addition, example embodiments may include elements not shown in the figures.

[0115] Although various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and not limitation, with the true scope being indicated by the appended claims.

Claims

1. A computer-implemented perceptual representation filling method, comprising: determining (i) a mask defining a portion of a perceptual representation, (ii) a plurality of convolutional features associated with the perceptual representation, and (iii) a plurality of contextual attention features associated with the perceptual representation; determining a concatenation of the mask, the plurality of convolutional features, and the plurality of contextual attention features; determining a plurality of gating values ​​for the portion of the perceptual representation, wherein determining the plurality of gating values ​​comprises processing the splicing using a machine learning model; Determining (i) a plurality of gated convolutional features based on the plurality of convolutional features and the plurality of gating values, and determining (ii) a plurality of gated contextual attention features based on the plurality of contextual attention features and the plurality of gating values, wherein determining (i) the plurality of gated convolutional features comprises determining a product of the plurality of convolutional features and the plurality of gating values, and wherein determining (ii) the plurality of gated contextual attention features comprises: determining a plurality of residual gating values ​​based on a plurality of differences between a predetermined value and the plurality of gating values; and determining a product of (i) the plurality of contextual attention features and (ii) the plurality of residual gating values; and generating a plurality of refined values ​​for the portion of the perceptual representation based on the plurality of gated convolutional features and the plurality of gated contextual attention features, Wherein, the perceptual representation includes one or more of (i) an image, (ii) a point cloud, (iii) a waveform, or (iv) text data.

2. The computer-implemented method of claim 1 , wherein the portion of the perceptual representation comprises a plurality of samples, and wherein the computer-implemented method further comprises: Through a coarse machine learning model, a corresponding coarse sample value is generated for each corresponding sample in the multiple samples, wherein each of the multiple convolutional features and the multiple contextual attention features is determined based on the corresponding coarse sample value of each corresponding sample.

3. The computer-implemented method of claim 2, wherein: Each corresponding sample of the plurality of samples is associated with (i) a corresponding convolution feature of the plurality of convolution features representing a generated sample value of the corresponding sample and (ii) a corresponding contextual attention feature of the plurality of contextual attention features representing a replicated sample value of the corresponding sample; Determining the plurality of gating values ​​comprises determining a corresponding gating value in the plurality of gating values ​​for each respective sample; Determining (i) the plurality of gated convolutional features comprises determining, for each respective sample, a corresponding gated convolutional feature of the plurality of gated convolutional features, and determining (ii) the plurality of gated contextual attention features comprises determining, for each respective sample, a corresponding gated contextual attention feature of the plurality of gated contextual attention features; and Generating the refined values ​​for the portion of the perceptual representation includes, for each respective sample, generating a corresponding refined value from among the plurality of refined values ​​based on the corresponding gated convolutional features and the corresponding gated contextual attention features.

4. The computer-implemented method of any one of claims 1-3, wherein determining the plurality of gating values ​​comprises: Determining a convolution of the concatenation with one or more convolution kernels of the machine learning model.

5. The computer-implemented method of claim 1 , wherein: Each respective gating value of the plurality of gating values ​​is generated by mapping by the machine learning model to a range between zero and the predetermined value.

6. The computer-implemented method of any one of claims 1 to 3, wherein: Generating the plurality of refined values ​​comprises: determining a sum of the plurality of gated convolutional features and the plurality of gated contextual attention features; and The plurality of refined values ​​are generated based on the sum.

7. The computer-implemented method of any one of claims 1 to 3, wherein: Generating the plurality of refined values ​​comprises: determining a second concatenation of the plurality of gated convolutional features and the plurality of gated contextual attention features; and The plurality of refined values ​​are generated based on the second splicing.

8. A computer-implemented method according to any one of claims 1-3, wherein the portion of the perceptual representation is a first portion of the perceptual representation, and wherein the mask includes a first region indicating replacement of the first portion of the perceptual representation with the plurality of refined values, and a second region defining a second portion of the perceptual representation and indicating that the second portion of the perceptual representation remains unmodified.

9. The computer-implemented method of any one of claims 1 to 3, further comprising: determining (i) a second mask defining the portion of the perceptual representation at a second size scale different from a first size scale associated with the mask, the plurality of convolutional features, and the plurality of contextual attention features, (ii) a second plurality of convolutional features associated with the perceptual representation at the second size scale, and (iii) a second plurality of contextual attention features associated with the perceptual representation at the second size scale; determining a further concatenation of the second mask, the plurality of second convolutional features, and the plurality of second contextual attention features; determining a plurality of second gating values ​​at the second size scale for the portion of the perceptual representation, wherein determining the plurality of second gating values ​​comprises processing the further concatenation using a second machine learning model associated with the second size scale; Determining (i) a plurality of second gated convolutional features at the second size scale based on the plurality of second convolutional features and the plurality of second gating values, and determining (ii) a plurality of second gated contextual attention features at the second size scale based on the plurality of second contextual attention features and the plurality of second gating values; and Further based on the plurality of second gated convolutional features and the plurality of second gated contextual attention features, a plurality of refined values ​​are generated for the portion of the perceptual representation.

10. The computer-implemented method of any one of claims 1-3, wherein the machine learning model is trained to generate the plurality of gating values, and wherein the training of the machine learning model comprises: obtaining trained perceptual representations; removing a portion of the trained perceptual representation; generating a plurality of refined values ​​for the removed portion of the training perceptual representation based on processing by the machine learning model a concatenation of (i) a training mask defining the removed portion of the training perceptual representation, (ii) a plurality of training convolutional features associated with the training perceptual representation, and (iii) a plurality of training contextual attention features associated with the training perceptual representation; determining a loss value based on a comparison of the plurality of refined values ​​of the removed portion with corresponding values ​​of the trained perceptual representation using a loss function; and One or more parameters of the machine learning model are adjusted based on the loss value and the loss function.

11. A computing system comprising: processor; and A non-transitory computer readable medium having stored thereon instructions which, when executed by the processor, cause the processor to perform the method according to any one of claims 1-3.

12. A non-transitory computer readable medium having stored thereon instructions which, when executed by a computing device, cause the computing device to perform the method of any one of claims 1-3.

Images