Use semantic analysis and target identifiers to support object movement
Through semantic analysis of source objects and target destinations and enhanced representation on interfaces, the problem of inefficient object movement in the prior art is solved, and efficient object transmission on small screen devices is realized.
Patent Information
- Application Number
- CN201980022786.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-03-28
- Filing Date
- 2019-03-21
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2039-03-21
AI Technical Summary
The prior art is inefficient when moving objects from one application to another, especially on computing devices with small screens, where users need to perform a multi-step clipboard process to accomplish this.
By performing semantic analysis of the source object and the target destination, a score reflecting the strength of the relationship between the object and the target is generated, and an enhanced target representation is provided on the user interface, including changing the characteristics of the target identifier or displaying a transient target identifier. So that the user can move the object to the destination more intuitively.
The steps that users need to perform are greatly reduced, and the efficiency of moving objects is improved, especially on small display devices, simplifying the transmission process of objects.
Smart Images

Figure CN111971680B_ABST
Abstract
Description
Background Art
[0001] In recent years, manufacturers have increased the power and versatility of smaller-sized computing devices (such as smartphones, tablet computing devices, etc.). However, interaction with these types of devices is sometimes challenging. For example, consider a situation where a user wishes to move an object from a first application to a second application. The computing device may have a smaller screen size that makes it impossible or inefficient to simultaneously present application windows associated with both the first application and the second application. Instead, the user typically transfers the object by performing a multi-step process that utilizes the operating system's clipboard repository. In this process, the user: (1) activates the first application; (2) selects the desired object and stores it in the clipboard repository; (3) activates the second application; (4) pastes the object into the second application at the desired location. Although this solution achieves the desired result, it is cumbersome and inefficient. Summary of the invention
[0002] According to an illustrative implementation, a technique for helping a user move a source object that appears on a user interface presentation to a target destination is described herein. The technique involves performing semantic analysis on both the source object and at least one candidate target destination. The semantic analysis produces a score that reflects the strength of the relationship between the source object and the candidate target destination. The technique then presents a representation of the candidate target destination that is enhanced to a degree that depends on the output score of the candidate target destination. The technique then allows the user to move the source object to a representation of a selected target destination, such as the above representation of the candidate target destination, whereupon the source object is associated with the candidate target destination.
[0003] According to one illustrative aspect, the semantic analysis can consider multiple features related to the source object and the candidate target destinations, including content-based features, features related to user history, features related to object history, and the like.
[0004] According to another illustrative aspect, semantic analysis can use a machine-trained model.
[0005] According to one mode of operation, before the user begins to move the source object, the target identifier associated with the candidate target destination has appeared on the user interface presentation. Such a target identifier is referred to herein as an on-screen target identifier. In this case, the technology provides an enhanced representation of the candidate target destination by changing at least one characteristic of the on-screen target identifier (such as by enlarging the target identifier, making it glow, etc.). According to another mode of operation, the technology provides an enhanced representation of the candidate target destination by showing a transient target identifier (which does not appear on the user interface presentation when the user begins the move operation).
[0006] The technology has various technical advantages. For example, the technology allows a user to efficiently move a source object to a target destination, for example, by reducing the number of steps that the user needs to perform to achieve that destination. This is particularly beneficial for devices with small displays, but the technology improves efficiency when applied to any display device of any size.
[0007] The techniques outlined above may be embodied in various types of systems, apparatus, components, methods, computer-readable storage media, data structures, graphical user interface presentations, articles of manufacture, and the like.
[0008] This summary is provided to introduce some concepts in a simplified form; these concepts are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Figure 1 An illustrative user interface presentation is shown that includes a source object that a user intends to move to a target destination before the user moves the source object.
[0010] Figure 2 An illustrative user interface presentation of generating a mobility management component (MMC) when a user initiates moving a source object according to a first implementation is shown.
[0011] Figure 3 Shown in Figure 2 A variation of the user interface presentation shown in Figure 2 This variation is suitable for presentation on smaller sized display devices compared to the example shown in FIG.
[0012] Figure 4 Another user interface presentation of an MMC according to a second implementation is shown that can generate an MMC when a user starts to move a source object.
[0013] Figure 5Another user interface presentation of an MMC according to a third implementation is shown that can generate an MMC when a user starts to move a source object.
[0014] Figure 6 Shown in Figure 5 A continuation of the user interface experience shown in .
[0015] Figure 7 One illustrative computing system for implementing the MMC is shown.
[0016] Figure 8 One way in which the MMC may interact with a source component (which supplies a source object) and a target component (which receives the source object and is associated with a target destination) is shown.
[0017] Fig. 9 A more detailed illustrative implementation of an MMC is shown.
[0018] Fig.10 One implementation of a configuration user interface presentation that an MMC may use to configure the operation of the MMC is shown.
[0019] Fig.11 An implementation of a target ranking component is shown, which is a component of the MMC.
[0020] Fig.12 and Fig.13 Together, a process representing one mode of operation of the MMC is shown.
[0021] Fig.14 is a process that illustrates one way to invoke enhancements to a target identifier when a user moves a source object.
[0022] Fig.15 The case where the MMC enhances the on-screen target identifier (which already exists on the user interface presentation when the move operation begins) is shown. Figure 12 to Figure 13 the realization of the process.
[0023] Fig.16 1 shows a case where a transient target identifier is displayed for the MMC (the transient target identifier is not yet present on the user interface presentation when the move operation starts). Figure 12 to Figure 13 the realization of the process.
[0024] Fig.17 Illustrative types of computing devices are shown that may be used to implement any aspects of the features shown in the preceding figures.
[0025] Throughout this disclosure and the drawings, the same reference numerals are used to refer to the same components and features. Figure 1Series 200 reference numerals refer to features originally found in Figure 2 Series 300 reference numerals refer to features originally found in Figure 3 Features found in , and so on. DETAILED DESCRIPTION
[0026] The present disclosure is organized as follows. Section A describes a computing system for assisting a user in moving a source object to a target destination in an efficient manner. Section B sets forth an illustrative method for explaining the operation of the computing system in Section A. And Section C describes illustrative computing functionality that can be used to implement any aspect of the features described in Sections A and B.
[0027] As a reference, the term "hardware logic circuit device" corresponds to one or more hardware processors (e.g., CPU, GPU, etc.) that execute machine-readable instructions stored in a memory, and / or one or more other hardware logic components (e.g., FPGA) that use a task-specific set of fixed and / or programmable logic gates to perform operations. Section C provides additional information about one implementation of the hardware logic circuit device.
[0028] The terms "component", "unit", "element", etc. refer to a portion of a hardware logic circuit device that performs a specific function. In one case, the illustrated separation of various components in the drawings into different units may reflect the use of corresponding different physical and tangible components in actual implementation. Alternatively, or additionally, any single component illustrated in the drawings may be implemented by multiple actual physical components. Alternatively, or additionally, the depiction of any two or more separate components in the drawings may reflect different functions performed by a single actual physical component.
[0029] Other figures describe concepts in the form of flow charts. In this form, certain operations are described as constituting different blocks that are executed in a characteristic order. This implementation is illustrative and not restrictive. Certain blocks described herein can be gathered together and executed in a single operation, certain blocks can be divided into multiple constituent blocks, and certain blocks can be executed in an order different from the order illustrated herein (including a parallel manner of executing blocks). In one implementation, the blocks related to processing-related functions shown in the flow chart can be implemented by the hardware logic circuit device described in Section C, which can be implemented by one or more hardware processors and / or other logic components including a set of task-specific logic gates.
[0030] With respect to terminology, the phrase "configured to" encompasses various physical and tangible mechanisms for performing the identified operations. These mechanisms may be configured to perform the operations using the hardware logic circuitry in Section C. The term "logic" likewise encompasses various physical and tangible mechanisms for performing tasks. For example, each processing-related operation illustrated in a flowchart corresponds to a logic component for performing that operation. The logic component may perform its operation using the hardware logic circuitry in Section C. When implemented by a computing device, a logic component represents an electrical component that is a physical part of a computing system, regardless of how it is implemented.
[0031] Any storage resource or any combination of storage resources described herein may be considered a computer-readable medium. In many cases, a computer-readable medium represents a physical and tangible entity of some form. The term "computer-readable medium" also encompasses, for example, propagation signals transmitted or received via physical conduits and / or air or other wireless media, etc. However, the specific term "computer-readable storage medium" explicitly excludes the propagation signal itself, but includes all other forms of computer-readable media.
[0032] The following explanation may identify one or more features as "optional." This type of statement should not be interpreted as an exhaustive indication of features that may be considered optional; that is, other features may be considered optional even though they are not explicitly identified herein. Further, any description of a single entity is not intended to exclude the use of multiple such entities; similarly, a description of multiple entities is not intended to exclude the use of a single entity. Further, while the description may explain certain features as alternative ways of implementing an identified function or realizing an identified mechanism, these features may also be combined together in any combination. Finally, the terms "exemplary" or "illustrative" refer to one implementation among potentially many implementations.
[0033] A. Illustrative Computing Systems
[0034] A.1. Illustrative User Interface Experience
[0035] This subsection describes an illustrative user interface presentation provided by a computing device. The computing device displays the user interface presentation on a display device having any size. In one implementation, the operating system of the computing device implements a mobile management component (MMC). The MMC is the logic that, at least in part, generates the user interface presentation described below. The next subsection (Subsection A.2) describes an illustrative implementation of the MMC itself.
[0036] from Figure 1Initially, the figure shows a user interface presentation 102 that generally corresponds to a desktop-type presentation. The user interface presentation 102 includes a source presentation A 104 generated by an application A. The source presentation A 104 is associated with one or more moveable objects 106. For example, application A may correspond to a photo sharing application that includes one or more pages of digital photos. Each digital photo corresponds to a moveable object. In another example, application A corresponds to a word processing application that provides one or more documents. Each document may correspond to a moveable object. In addition, each portion of a separate document (such as each page, paragraph, sentence, word, letter, etc.) may correspond to a moveable object. More generally, a moveable object may include content of any type (or combination of types) and thus have any scope.
[0037] The user interface presentation 102 may also include zero, one, or more other presentations 108 provided by other corresponding applications. Figure 1 In the state shown in , application A corresponds to the in-focus application with which the user is currently interacting. Therefore, the user interface presentation 102 shows that the source presentation A 104 is overlaid on the other presentations 108. (The subsequent figures will omit the depiction of the other presentations 108 to support the explanation.)
[0038] The user interface presentation 102 may optionally include one or more on-screen target identifiers 110 in a main portion of the user interface presentation 102. Additionally or alternatively, the user interface presentation 102 may include one or more on-screen target identifiers 112 in a taskbar area 114 of the user interface presentation 102. Each on-screen target identifier represents a candidate target destination. Figure 1 The on-screen target identifiers (110, 112) shown in the figure represent corresponding applications. Alternatively, or in addition, the target identifiers may represent other resources, such as folders, remote storage sites, people, etc. The qualifier "on-screen" means that each target identifier has a stable on-screen presence on the user interface presentation 102. For example, each on-screen target identifier may correspond to an icon that persists on the user interface presentation 102 (i.e., unless explicitly removed by the user).
[0039] In the above setting, it is assumed that the user intends to move the source object 116 from the source domain presented on the user interface presentation 102 to an appropriate target destination, such as a specific application, folder, repository, person, etc. Figure 1 In the specific example of , the source object 116 corresponds to a digital photo, and the source domain corresponds to the source presentation A 104. In other cases, the source domain may correspond to the desktop application itself.
[0040] The user initiates the move operation by selecting the source object 116 and dragging it across the user interface presentation 102. Figure 1 In the merely illustrative case of FIG. 1 , a user uses a mouse device (not shown) to perform a drag operation. The user uses the mouse device to control a cursor 118. In other implementations, the computing device displays the user interface presentation 102 on a touch-sensitive display device. The touch-sensitive display device includes a touch-sensitive surface implemented using capacitive technology, resistive technology, optical technology, etc., or any combination thereof. In this context, the user can use a finger (or multiple fingers), a pen, a stylus, etc., applied to the touch-sensitive surface of the display device to perform the drag operation. Note that this explanation sets forth various mouse-based user interface operations; any such examples should be interpreted as having a corresponding touch-based drag operation, even if not explicitly stated in each instance.
[0041] In some cases, the user begins the drag operation with a target destination already in mind. The user interface presentation 102 may already show a target identifier associated with the intended target destination, e.g., corresponding to one of the on-screen target identifiers (110, 112). Alternatively, at the beginning of the drag operation, the user interface presentation 102 may not display a target identifier associated with the intended target destination. In other cases, the user may begin the drag operation without having decided where to place the source object 116.
[0042] In response to the user's movement of source object 116, the MMC determines the relationship between the source object and each of the plurality of candidate target destinations, including target destinations that currently have a corresponding on-screen target identifier and target destinations that currently do not have any on-screen target identifier. This produces an output score for each candidate destination. Subsection A.2 explains a technique for performing this evaluation.
[0043] The MMC then enhances each target identifier based on the output score of its corresponding target destination. For example, the MMC may determine whether the output score of the target destination is above a specified threshold. If so, the MMC may enhance its corresponding target identifier to a certain extent, which depends on the magnitude of the output score. The enhancement may involve modifying the on-screen target identifier that has already appeared on the user interface presentation 102. In other cases, the enhancement involves displaying a transient target identifier associated with the candidate target destination.
[0044] The user can configure the MMC to identify when the processing outlined above will begin. In one case, the MMC can be configured to start the above processing immediately after the user begins to move the source object 116. In another case, the MMC can be configured to start the above processing when the user explicitly requests the above processing, for example, by issuing a voice command of "show recommended targets". In another case, the MMC can be configured to start processing when the user presses the source object 116 (using a finger, a pen, etc.) for more than a specified amount of time or when the user performs a similar holding gesture. In yet another case, the MMC can be configured to start the above processing when the user fails to move the source object 116 to the target destination within a specified amount of time, which indicates that the user has difficulty selecting the desired target destination.
[0045] Figure 2 104, for example, by clicking and holding source object 116 with a mouse device, or by touching source object 116; thereafter, the user moves source object 116 away from object set 106. Depending on how the drag operation is configured, the operation may result in the removal of source object 116 from object set 106; or, it may involve the creation of a copy of source object 116, leaving the original source object 116 in place in source presentation A 104. Figure 2 At the depicted juncture, the user has dragged the source object 116 over the path 202. However, at this juncture, the user has not yet moved the source object 116 to its final target destination.
[0046] The MMC generates enhanced representations of four target identifiers in response to its processing. The first two target identifiers (204, 206) correspond to existing on-screen target identifiers, e.g., to two icons associated with two respective local applications. The other two target identifiers (208, 210) correspond to transient target identifiers that did not appear on the user interface presentation 102 before the user performed the drag operation. More generally, the MMC may present any number of transient target identifiers 212. Figure 2 In a non-limiting example, transient target identifier 212 corresponds to a target icon having the same style and appearance as the on-screen target identifier (110, 112).
[0047] The MMC may enhance the appearance of the target identifier in one or more ways. Consider any on-screen target identifier scenario. Without limitation, the MMC may: change the size of the target identifier; change the color of the target identifier; change the intensity level of the target identifier; change the transparency level of the target identifier; present the target identifier in a special mode (such as a glow mode, a visual vibration / shake mode, etc.); add a rating label to the target identifier; change the position of the target identifier on the user interface presentation 102 relative to other target identifiers, etc.
[0048] Now consider the case of a transient target identifier. It can be said that the MMC enhances the visual representation of such a transient target identifier simply by the act of displaying the visual representation of such a transient target identifier (which inherently emphasizes such a transient target identifier relative to those transient target identifiers that are not displayed because those transient target identifiers that are not displayed are not sufficiently relevant to the source object 116). In addition, the MMC can use any of the techniques described above for on-screen target identifiers to emphasize the relevance of the transient target identifier, for example, by selecting an appropriate size, position, color, intensity, etc. of the transient target identifier.
[0049] exist Figure 2 In the merely illustrative example of , MMC uses star rating labels to highlight the relevance of each of the selected target identifiers (204, 206, 208, 210). As shown, transient target identifier H 208 has the most stars, indicating that MMC has evaluated it as the most relevant to the source object. In addition, MMC modifies the intensity level applied to the on-screen target identifiers (204, 206) based on the corresponding output scores of the on-screen target identifiers (204, 206). Note that on-screen target identifier 206 is shown as having a higher intensity level than on-screen target identifier 204. In addition, MMC modifies the size of on-screen target identifier 206.
[0050] The MMC may optionally display an explanation 214 that alerts the user to the reason why it has enhanced the above-mentioned set of target identifiers (204, 206, 208, 210). The MMC may present the explanation at any level of granularity. For example, the MMC may notify the user that it has selected the transient target identifier H 208 because the theme of the source object (Venice, Italy) matches the theme of the target destination associated with the target identifier 208 (e.g., the theme corresponds to a photo journal of an Italian vacation). The MMC may notify the user that it has selected the on-screen target identifier 206 (the on-screen target identifier 206 is a photo editing application) because users often drag digital photos from the source presentation A 104 as a prelude to operating on the photos using the photo editing application. The MMC may create the explanation in the manner set forth in Subsection A.2.
[0051] In the next stage, the user moves the source object 116 to the target destination. The user can perform this task by dragging the source object 116 to any target identifier, including any identifier that the MMC has enhanced or any identifier that the MMC has not enhanced. Figure 2 In the example of FIG. 1 , assume that the user drags the source object 116 to the transient target identifier H 208, i.e., by verbally “dropping” the source object 116 onto the icon associated with the transient target identifier H 208. Alternatively, the user may abandon the drag operation, upon which the source object 116 may remain in whatever position the user left it on the user interface presentation 102. Alternatively, the MMC may snap the source object 116 back to its original position in the object array 106.
[0052] The user's move operation causes the source object 116 to be associated with the target destination H associated with the transient target identifier H 208. For example, again assume that the target destination H corresponds to an online photo diary. The user's dragging behavior causes the photo diary to store the source object 116 in a default location within the photo diary, such as on the last page of the photo diary.
[0053] In one implementation, the MMC removes the transient target identifier 212 after the user drops the source object 116 onto the transient target identifier H208. It also removes any enhancements applied to the on-screen target identifier 110. In another implementation, the MMC removes the transient target identifier 212 only after receiving an explicit instruction from the user. In either case, the target identifier 212 is referred to as "transient" because, unlike the on-screen target identifiers (110, 112), they are not considered a persistent part of the user interface presentation 102.
[0054] The user interface presentation 102 may optionally present another type of identifier that may be considered a learned on-screen target identifier. In operation, assume that a user frequently moves a source object to a particular transient target identifier corresponding to a social media application. With respect to each of these previous examples, further assume that the MMC presents the transient target identifier associated with the social media application in the same portion of the user interface presentation 102. Figure 2 In the example of , assume that the MMC presents a transient identifier in the upper right corner of the user interface presentation 102. This behavior allows the user to create a mental association between the social media application and the upper right corner of the user interface presentation 102. The MMC determines when the frequency of the above behavior exceeds a threshold value specific to the environment. When this occurs, the MMC can display a currently persistent on-screen target identifier 216 in the upper right corner of the user interface presentation 102, which is associated with the social media application. The on-screen target identifier 216 can be considered a learned target identifier because the MMC presents the on-screen target identifier 216 after learning the user's behavioral patterns with respect to the social media application. Thereafter, the user can move the source object to the learned on-screen target identifier 216 in the same manner as any other on-screen target identifier.
[0055] In some cases, the MMC may mark the learned on-screen target identifier 216; in other cases, the MMC does not display any information identifying the corner of the user interface presentation 102 as being associated with the social media application (in Figure 2 ). In another case, the MMC may provide a transient label in a corner of the user interface presentation 102 that identifies the social media application as the user moves the source object on the user interface presentation 102; this MMC behavior confirms to the user that the corner of the user interface presentation 102 is still associated with the social media application. In general, using a learned target identifier is beneficial to the user because it takes advantage of the user's established drag-related behavior patterns without having to clutter the user interface presentation 102 with transient target identifiers.
[0056] In other cases, the user may interact with the configuration component to manually establish a link between a particular portion of the user interface presentation 102 and a target destination. The user may also provide a setting that determines whether the identified portion of the user interface presentation will include a label. When not labeled, such a target identifier may be referred to as a user-defined "invisible" target identifier.
[0057] Figure 3 Shows that for Figure 2Compared to the case of computing device 302 having a smaller display device, how can the Figure 2 Assume that Figure 3 The display device shown has a touch-sensitive surface. Figure 3 In the case of the source presentation A 104, the MMC fills the display space provided by the display device. Assume again that the source presentation A is generated by application A. It includes a set of movable objects 106. And assume again that the user intends to move the source object 116 to the desired target destination. However, in this case, the MMC does not immediately show any on-screen target identifier.
[0058] In operation, the MMC determines that the user has begun dragging the source object 116 using his or her finger. In response, the MMC determines that the most relevant target destinations include target destinations F, H, and I. In response, the MMC displays transient target identifiers 304 for these three target destinations. Assume that the user then drags the source object 116 to the transient target identifier H 306, which causes the source object 116 to be associated with the corresponding target destination H. For example, the user's gesture causes the source object 116 to be stored on the last page of the online photo diary (which is the target destination associated with the transient target identifier H 306). Figure 2 In the same manner described, after the user drops the source object 116 onto the transient target identifier H 306 , the MMC may remove the transient target identifier 304 .
[0059] Figure 4 Shown in Figure 2 102. In this case, the MMC displays a transient target identifier 402 for the target destination H (corresponding to the online photo diary), which is similar to the one in FIG. Figure 2 Compared to the transient target identifier H 208 shown in FIG. 4 , the transient target identifier 402 is more detailed. Figure 2 In FIG. 2 , the transient target identifier H 208 corresponds only to the target icon associated with the target destination H. Figure 4 In the example, when the MMC determines that the user is moving the source object 116, the MMC still displays the transient target 402. Figure 4 The transient target identifier 402 in includes a representation of the content provided by the transient target destination H. For example, the transient target identifier 402 shows the last page of the online photo diary.
[0060] The user may terminate the drag operation by dropping the source object 116 at the desired location in the transient target identifier 402. The drop also defines the location of the source object 116 within the corresponding target destination (e.g., by dropping the source object 116 at a specific location on the last page of a photo journal). Figure 2 Unlike the example Figure 2 In the example of , the MMC places the source object 416 at a default location within the target destination. In a configurable mode of operation, when the user places the source object 116 at a selected location within the transient target identifier 402, the MMC removes the transient target identifier 402.
[0061] Figure 4 Only a single transient target identifier 402 is shown. However, in other scenarios, the MMC may be present in Figure 4 . The MMC may arrange the transient target identifiers in the user interface presentation 102 in any manner, such as by showing an array of the identifiers, or by creating a partially overlapping stack of transient target identifiers. The MMC may sort the transient target identifiers in the array or stack based on the assessed relevance to the source object 116.
[0062] Figure 5 Shown in Figure 2 and Figure 4 . In this case, it is assumed that at least some of the target destinations include two or more sub-destination. For example, an online photo diary may include multiple pages or sections dedicated to different topics. For example, a user's photo diary of his or her trip to Italy may include multiple pages associated with corresponding visits to major cities in Italy (e.g., Rome, Venice, Florence, etc.).
[0063] exist Figure 5 In an implementation of , the MMC displays a first group of transient (sub) target identifiers 502 associated with a target destination H and a second group of transient (sub) target identifiers 504 associated with a target destination I. For example, each group may correspond to a stacked set of partially overlapping target icons associated with a corresponding target sub-destination of a primary target destination. The MMC may sort the transient target identifiers in the group based on their relevance to the source object 116. For example, the MMC may determine that the target sub-destination Venice is most relevant to the user's Venice photos. Therefore, the MMC places a transient target identifier 506 for the Venice page at the top of the transient target identifier stack.
[0064] Go to Figure 6, next assume that the user drags source object 116 onto transient target identifier 506 for Venice, and then maintains his or her touch (or other type of contact) at that location for a specified amount of time. In response, the MMC may generate Figure 6 , including an expanded transient target identifier 604 associated with the city of Venice. The user may then place a source object 116 at a desired location within the transient target identifier 604 (which also places the source object 116 at a corresponding location on the actual Venice page of the photo diary). Note that the MMC may invoke the array 602 of transient target identifiers in response to any other kind of command transmitted by the user (such as a voice command, etc.) (in addition to the above-described press and hold gesture).
[0065] The above implementation is described for the purpose of illustration rather than limitation. Other variations are also possible. For example, in another variation, the MMC may allow a user to simultaneously move a group consisting of multiple source objects. In this case, the MMC will determine the relevance of the source object group to each candidate target destination. In other words, in this case, the description of the group may correspond to a summary or aggregation of the attributes of its constituent sub-objects.
[0066] In a variation of the above implementation, it is assumed that the target destination corresponds to a composite destination representing a set of target destinations. Figure 2 The transient identifier H 208 in identifies an online photo diary, and the online photo diary covers destination pages associated with different Italian cities. Now assume that a user simultaneously places a set of two or more source objects on the transient identifier H208. In one implementation, the MMC automatically distributes the source objects in the set to the appropriate pages of the photo diary based on any combination of factors described in the next subsection. For example, the MMC can generate scores for each pair of separate source objects and candidate pages of the photo diary based on the assessed content-related similarities between the source object and the candidate page. The MMC can then distribute each source object to the page with the highest score, for example, by distributing pictures of canal boats to pages related to the city of Venice. In other examples, a composite destination can represent a heterogeneous group of target destinations, meaning a group that does not conform to a single content-related theme. Optionally, the MMC can ask for the user's permission before distributing the source object to the target destination within the composite destination.
[0067] In one implementation, a user may configure the MMC to define one or more desired composite target destinations. In another implementation, the MMC may automatically create one or more composite target destinations by identifying groups of related target destinations using semantic clustering.
[0068] A.2. Illustrative Implementation of MMC
[0069] Figure 7 An illustrative computing system 702 for implementing MMC is shown. It is assumed that a user interacts with MMC via a local user computing device 704. The local user computing device 704 may correspond to a desktop computing device, a laptop computing device, a game console, a tablet computing device, a smart phone or other type of handheld computing device, a wearable computing device, a mixed reality device (e.g., a head-mounted display), etc. The local user computing device 704 includes a separate or integrated display device 706.
[0070] The source objects and target destinations may be distributed in any manner among different entities in the computing system 702. For example, the user computing device 704 may include a local application or other local resource that hosts one or more source objects 708. The user computing device 704 may also include a target application or other resource corresponding to one or more target destinations 710.
[0071] The computing system 702 also includes one or more remote computing devices (712, 714, ...) that host source objects (716, 718, ...) and target destinations (720, 722, ...). The remote computing devices (712, 714, ...) may correspond to corresponding servers. Further, the computing system 702 may include other user computing devices operated by other users (and / or the same user operating the user computing device 704). For example, the representative user computing device 724 may be associated with a friend or family member of the user operating the user computing device 704. Another user computing device 724 may also host one or more source objects 726 and one or more target destinations 728. The user computing device 704 may interact with any computing device (or resource) (712, 718, 724, ...) identified above via a computer network 730. The computer network 730 may correspond to a local area network, a wide area network (e.g., the Internet), one or more point-to-point links, or any combination thereof. The computer network 730 may be controlled according to any protocol or combination of protocols. Additionally, user computing device 704 may communicate with user computing device 724 in a direct manner, such as via near field communication (NFC) or the like.
[0072] The MMC itself can be distributed in the various components described above. For example, in one implementation, the local user computing device 704 includes an operating system that implements the entire logic of the MMC. In another case, one or more remote servers implement the entire logic of the MMC. In another case, the logic of the MMC is implemented by a distributed combination of local and remote processing resources.
[0073] Figure 7 Four illustrative scenarios in which a source object is moved to a target destination are shown. Each scenario involves the transfer of digital information associated with a source object to a target destination. In scenario (A), the MMC transfers a local source object provided by a user computing device 704 to a local target destination also provided by the user computing device 704. In scenario (B), the MMC transfers a local source object provided by a user computing device 704 to a remote target destination associated with a remote computing resource 712. In scenario (C), the MMC transfers a local source object provided by a user computing device 704 to a target destination provided by another user computing device 724. For example, a user may transfer a source object to a photo diary maintained by his wife's local computing device. In scenario (D), the MMC transfers a remotely stored source object provided by a remote computing resource 712 to a remote target destination provided by the same remote computing resource 712. This non-exhaustive collection of example scenarios is presented for illustration and not limitation.
[0074] Figure 8 802 and the relationship between the source component 804 and the target component 806. The source component 804 supplies the source object. For example, the source component 804 may correspond to the source component 806 provided by the Figure 7 , which supplies a source object (such as a digital photo). The target component 806 implements the target destination. For example, the target component 806 may correspond to the target object provided by any computing device shown in Figure 7 Target applications (e.g., photo diary) provided by any of the computing devices shown in . In other cases, source component 804 and / or target component 806 correspond to components provided by an operating system.
[0075] In one implementation, MMC 802 corresponds to the logic provided by an operating system (such as, the operating system of local user computing device 704). When source component 804 detects that the user has begun to drag the source object, source component 804 can provide a handoff signal to MMC802. This handoff signal includes a description of the source object. Then, MMC 802 processes the behavior related to dragging described in chapter A.1, including enhancing the target identifier on the screen and presenting a transient target identifier. When MMC 802 determines that the user has placed the source object on the target identifier associated with the target destination, MMC 802 sends a handoff signal to target component 806. This handoff signal describes the source object again. Then, target component 806 processes the source object in an application-specific manner, such as, by storing digital photos in the pages of a photo diary.
[0076] In another implementation, the MMC 802 may provide an application programming interface (API) with which the source component 804 and / or the target component 806 may interact. For example, after determining that the user has started a drag operation, the source component 804 may call the API to send a data packet to the MMC 802.
[0077] Fig. 9 A more detailed implementation of the MMC 802 is shown. The MMC 802 includes a configuration component 902 that allows a user to define various aspects of the way in which the MMC 802 operates. The configuration component 902 stores configuration settings in a configuration data repository 904.
[0078] For example, when invoked, configuration component 902 may be presented in Fig.10 1002. Through the user interface presentation 1002, the configuration component 902 allows the user to specify a set of target destinations to which the source object can be directed and / or a set of source destinations through which the source object can be drawn. The configuration component 902 can also allow the user to specify the types of source objects that are allowed to be moved. Fig.10 In the merely exemplary case, configuration component 902 allows a user to specify whether personal content and / or work-related content can be moved; however, other implementations may categorize content into any number of context-specific types.
[0079] Configuration component 902 can also allow the user to specify whether MMC 802 will produce its target enhancement effect for the combination of the target identifier on the screen, the transient target identifier, or the target identifier and the transient target identifier on the screen. Configuration component 902 can also request the input from the user, the control screen target identifier behavior and the transient target identifier behavior. For example, the user can specify that the process will be called, and MMC 802 enhances the gesture of the target identifier thus.
[0080] In addition, the configuration component 902 can allow the user to customize the behavior of the analysis (described below) performed by the MMC 802 in any manner. For example, a user can define preferences that control the manner in which the MMC links source objects to corresponding target destinations. For example, by setting such preferences, the user can ensure that when the user begins manipulating a specified type of source object, the MMC will select the highest ranked target destination using the analysis specified by the preferences expressed by the user. In other implementations, the configuration component 902 can allow the user to create more complex custom rules that control their ranking of target destinations, such as by defining rules of a customized IF-THEN structure.
[0081] Return to Fig. 9, the initiation-detection component 906 provides logic for determining when to call the MMC to process the target destination. For example, when it receives a switching signal from the source component 804, the initiation-detection component 904 can determine that the user has started the drag operation. In one implementation, the initiation-detection component 906 immediately calls the MMC to process the target destination when the user starts the drag operation. In another case, the initiation-detection component 906 calls the MMC to process the target destination only when the additional factor is met. For example, when the user spends more than a specified amount of time to move the source object to the target identifier, or when the user pauses the drag operation at any position other than the target identifier on the user interface presentation 102 for more than a specified amount of time, the initiation-detection component 906 can call the MMC to process the target destination.
[0082] The initiation detection component 906 can detect the above-described gestures by comparing input signals provided by one or more input devices 908 to speech patterns associated with a predetermined set of gestures. Illustrative input devices 908 include touch-based input surfaces, pen input devices, mouse input devices, etc. The initiation detection component 906 can compare input signals to known patterns using discrete algorithms, machine-trained models (e.g., hidden Markov models), etc.
[0083] The object manipulation component 910 provides any user interface experience accompanying the user's drag gesture. For example, when the source object is moved from the source domain to the target identifier, the object manipulation component 910 displays the source object.
[0084] The target set identification component 912 identifies a set of candidate target destinations to be analyzed. In one mode of operation, the target set identification component 912 first consults the settings provided in the configuration data repository 904 to determine which target destinations are allowed to be checked. The target set identification component 912 can then check each authorized source of the target destinations to determine the target destinations it contains. For example, the target set identification component 912 can perform this task by checking the file system of the computing device and identifying the applications or other resources it contains that can constitute the target destinations.
[0085] The feature discovery component 914 identifies a first set of features describing the source object and a second set of features describing each candidate target destination. A feature refers to any characteristic of a source object or a target destination. In some cases, a target destination is composed of multiple target objects or other parts. For example, a target destination associated with an online photo diary is composed of multiple digital photos. In this scenario, the feature discovery component 914 can provide features that collectively describe the target objects associated with the target destination. The above situation applies to the source object in the following scenario: where the source object is composed of multiple secondary objects that the user moves as a group.
[0086] The feature discovery component 914 relies on a collection of feature generation components 916 to generate features. In one mode of operation, the feature generation component 916 operates in an online manner, i.e., by performing analysis in real time in response to a user's selection of a source object. Alternatively, or in addition, the feature discovery component 916 can perform at least some of its work in an offline manner. For example, before the drag operation begins, the feature generation component 916 can perform analysis on digital photos in the user's photo diary in an offline manner. Fig. 9 Illustrated is a possible use of offline data mined by loop 918. More specifically, loop 918 indicates that offline data mining can be run as an ongoing process, for example, periodically or in an on-demand manner. Feature generation components 916 store the features they identify in a feature data store 920.
[0087] With respect to the feature generation component 916 itself, the content inspection component 922 provides one or more features describing the content associated with each item under consideration (where the general term "item" is used hereinafter to refer to a source object or a target destination). For example, the content inspection component 922 may determine the subject of an item based on subject tags associated with the item (such as, tags associated with a digital photo or keywords and titles associated with a text document, etc.). Additionally, or alternatively, the content inspection component 922 may use a rule-based model or a machine-trained model to determine the subject of an item. For example, the content inspection component 922 may use a convolutional neural network (CNN) to provide an output vector that characterizes the subject of an image or text document, for example, by mapping the item to a vector in a low-dimensional semantic space. These techniques are set forth for the purpose of illustration and not limitation; other implementations of the content inspection component 922 may use other techniques to classify the content of an item.
[0088] The history monitoring component 924 performs two recording tasks. As a first function, the history monitoring component 924 generates information identifying the user's habits related to dragging. For example, the history monitoring component 924 can determine which source objects the user has moved in the past and where the user has moved them. The history monitoring component 924 can organize this information in any way, such as by providing a table listing the object types along one dimension and the destination types along a second dimension. The history monitoring component 924 can maintain a running count in each cell of the table, which identifies how many times the user has dragged a specific type of source object to a specific type of target destination. The above information is related to the habits of the user currently interacting with the MMC 802; but the history monitoring component 924 can also maintain information describing the habits of a general user group or the habits of one or more user subgroups, each group including members with similar characteristics.
[0089] The history monitoring component 924 can also store the information that reflects that the user explicitly rejects the target identifier. For example, suppose that in the previous occasion, MMC 802 identifies three transient target identifiers as being relevant to the source object that the user is presenting 102 on the user interface moving. Before the source object is dragged to the expected target identifier, the user can explicitly remove one or more unwanted transient identifiers, such as, by performing a gesture, whereby the user slides the transient identifier out of the screen. By doing this, the user can convey that these transient identifiers are obviously inappropriate, and the user does not wish to see them again in similar situations. This behavior constitutes a strong signal for training the (multiple) model of the MMC.
[0090] As a second function, the history monitoring component 924 can maintain information about the history of any particular item. For example, the history monitoring component 924 can maintain information about the person or entity that created the item; the date the item was created; the source(s) from which the item was obtained; the location where the item has been previously moved; the people with whom the item has been previously shared (if any), etc.
[0091] The trajectory analysis component 926 determines possible target destinations that the user may target based on the following path: the user has moved the source object on the user interface presentation 102 on the path relative to the current state of the path. The trajectory analysis component 926 can perform this task by fitting a line (such as a linear line or a parabola, etc.) that describes the user's current movement path. The trajectory analysis component 926 then extends the line to the edge of the user interface presentation 102. The trajectory analysis component 926 then identifies any on-screen target identifiers within a specified distance from the line; those on-screen target identifiers define the most likely targets of the user's drag movement, where the most relevant targets are associated with the target identifiers closest to the line. The trajectory analysis component 926 can also provide a certainty score that reflects the confidence of its evaluation. The confidence score can depend on the current length of the user's drag trajectory, the number of feasible target identifiers within a specified distance of the line, the user's drag-related habits, etc.
[0092] Feature generation component 916 may include other sources of features, although in Fig. 9 For example, another source may provide information about the type of file associated with the item. Another source may provide information about a rating associated with the item. Another source may provide information about a license associated with the item, etc.
[0093] The target ranking component 928 generates an output score for each pair of a source object (the source object is associated with the first set of features) and a candidate target destination (the candidate target destination is associated with the second set of features). In one implementation, the output score is related to the distance in the semantic space between the source object and the candidate target destination. The target ranking component 928 generates the output score using a rule-based algorithm, a machine-trained model, or some other method. Without limitation, the output score described below is Fig.11 An illustrative method for performing this task using a deep neural network (DNN) is shown.
[0094] The target ranking component 928 performs further processing on the output scores it generates. For example, the target ranking component 928 can select output scores that are above a specified threshold. Additionally, or alternatively, the target ranking component 928 can select the n highest scores, where n is a user-configurable parameter value. This means that the user interface presentation 102 will display at most n enhanced target identifiers. In other cases, the target ranking component 928 can select the k highest scores associated with the on-screen target identifiers and the m highest scores associated with the transient target identifiers.
[0095] A target presentation and interaction component (TPIC) 930 enhances a target identifier associated with a respective target destination based on the respective output score of the respective target destination. For example, the TPIC 930 may perform any combination of the techniques set forth in subsection A.1 to enhance an on-screen target identifier, such as by changing the size, intensity, color, etc. of the on-screen target identifier. The TPIC 930 may also control the appearance and removal of transient target identifiers. The TPIC 930 may also modify the characteristics of transient target identifiers using any of the techniques it applies to on-screen target identifiers, such as by showing a transient target identifier that is particularly relevant using a glow effect.
[0096] The explanation providing component 932 provides optional explanations that may accompany its enhancements to each target identifier, such as in Figure 2, illustrative explanation 214 shown in . For example, consider an illustrative pairing between a source object and a candidate target destination. The explanation providing component 932 can map features associated with the pairing to an explanation. This task can be performed using any feature-to-mapping technique, for example, by using a lookup table or a machine-trained model (such as a recurrent neural network (RNN)) that performs free-form generation on text. For example, in one scenario, the lookup table provides pre-stored text that explains that the pairing is appropriate because the content of the source object matches the content of the target destination well. In another scenario, the lookup table may include the following text: The text explains that the pairing is appropriate because users frequently move source objects of the current type to the target destination, etc.
[0097] Fig.11 One implementation of a target ranking component 928 is shown. The target ranking component 928 receives a first set of features 1102 describing a source object from the feature discovery component 914 and a second set of features 1104 describing a candidate target destination from the feature discovery component 914. It outputs a score identifying a semantic distance between the source object and the candidate target destination.
[0098] The pre-processing component 1106 converts each feature set into a form that can be further analyzed by the analysis engine 1108. For example, the pre-processing component 1106 can map each feature into (multiple) input values. It can then map the (multiple) input values to one or more input vectors, such as one-hot vectors. The pre-processing component 1106 can use a lookup table, a hash function, a separate neural network, etc. to perform this task.
[0099] The analysis engine 1108 may use a machine-trained model 1110 to map input information associated with a source object to a source object result 1112, e.g., corresponding to a first vector in a low-dimensional semantic space. The analysis engine 1108 may use a machine-trained model 1110 to map input information associated with a target destination to a target destination result 1114, e.g., corresponding to a second vector in a semantic space. For example, the machine-trained model 1110 may correspond to a deep neural network (DNN). The comparison component 1116 may determine the distance between the source object result 1112 and the target destination result 1114 to provide an output score. For example, the comparison component 1116 may calculate a cosine similarity distance metric, a Euclidean distance metric, a Manhattan difference metric, etc.
[0100] Typically, a feedforward neural network may include N layers of neurons that map an input vector z1 to an output vector y. For j = 2, ... N, the value in any layer j can be calculated by the formula z j =f(W j zj-1 +b j ) is given. Symbol W j represents the jth weight matrix generated by the training component 1118, and symbol b j refers to the optional j-th bias vector also generated by the training component 1118. The function f(x), known as the activation function, can be formulated in different ways, such as a tanh function. For example, information on the general subject of deep neural networks can be found in U.S. Published Patent Application No. 20150074027 (to Huang et al.), published on March 12, 2015, and entitled “A Deep Structured Semantic Model Produced Using Click-Through Data”.
[0101] The training component 1118 generates the parameter values identified above by processing the training examples in the data repository 1120. For example, the training examples include positive examples and negative examples. Each positive example includes a pair of items (A and B) with a known relationship (i.e., where A is positively correlated with B). Each negative example includes a pair of items (A and B) that do not have a relationship (i.e., where A is not correlated with B). An example mining engine (not shown) can collect positive examples from any source (such as an online document repository). That is, the example mining engine can regard two items extracted from the same document or document portion as related. The example mining engine can simulate each negative example by randomly creating a pair of items, assuming that the items paired together are not related. The training component 1118 can apply appropriate training objectives, such as, by iteratively deriving parameter values that maximize the ability of the model to predict correct relevance assessments and minimize the tendency of the model to produce incorrect relevance assessments. The training component 1118 can use any technology to perform training, such as, gradient descent combined with back propagation.
[0102] The use of DNN is described above for the purpose of illustration rather than limitation. Other implementations may use other machine-trained models, such as linear regression models, support vector machine models, Bayesian network models, decision tree models, clustering-based models, etc.
[0103] B. Illustrative Process
[0104] Figures 12 to 16The process of explaining the operation of MMC 802 in chapter A is shown in the form of a flow chart. Since the basic operating principle of MMC 802 has been described in chapter A, therefore, in this chapter, some operations will be introduced in a summarized manner. As mentioned in the preface of the specific embodiment, each flow chart is represented as a series of operations performed in a specific order. But the order of these operations is only representative, and the order of these operations can be changed in any way.
[0105] Figure 12 to Figure 13 1202 for supporting the movement of a source object to a target destination. In block 1204, the mobility management component (MMC) 802 receives a signal indicating that a user has selected a source object to be moved, the source object being presented in a source domain of the user interface presentation 102 displayed on the display device 706. For example, the MMC 802 may select Figure 8 The source component 804 of the MMC 802 receives such a signal. In block 1206, the MMC 802 identifies a first feature set associated with the source object. In block 1208, the MMC 802 identifies a second feature set associated with the candidate target destination. In block 1210, the MMC 802 uses semantic analysis to determine the relationship of the first feature set to the second feature set to provide an output score. Loop 1212 indicates that the above process is repeated one or more times for other candidate target destinations to generate a plurality of output scores overall. Another implementation may compare the source object to multiple candidate destinations in a parallel manner rather than a serial manner.
[0106] In block 1214, the MMC 802 displays representations of the candidate target destination(s) on the user interface presentation 102, the representation(s) being visually enhanced to a degree that depends on the corresponding output scores of the candidate target destination(s). At one end of the enhancement spectrum, the MMC 802 will not display representations of the target destinations at all. Note that in deciding how and when to enhance representations of target destinations, the MMC 802 may apply various analyses, such as, but not limited to, by comparing the score of the target destination to a threshold, by comparing the score of the target destination to the scores of other target destinations (and selecting the n best scores), etc.
[0107] exist( Fig.13 In block 1216 of the present invention, the MMC 802 receives an instruction from the user to move the source object to a representation of the selected candidate target destination, which may or may not have been visually enhanced. And in block 1218, the MMC 802 moves the source object on the user interface presentation 102 to the representation of the selected target destination, which associates the source object with the selected target destination.
[0108] Fig.14 It shows the call Figure 12 to Figure 13 In block 1404, the MMC 802 determines whether a certain enabling condition has been met, such as the user just started the drag operation, or the user spent more than a specified amount of time to perform the drag operation, or the user paused the drag operation for more than a specified amount of time, etc. If so, the MMC 802 performs the Figure 12 to Figure 13 Process 1202 shown in .
[0109] Fig.15 The method is shown in which the at least one target identifier is a target identifier on the screen. Figure 12 to Figure 13 Process 1502 is an instantiation of process 1202. In block 1504, MMC 802 displays a set of on-screen target identifiers on user interface presentation 102, such as, Figure 1 In block 1506, the MMC 802 changes a characteristic of at least one on-screen target identifier based on the associated output score of the at least one on-screen target identifier. Then, the process 1502 performs Fig.13 Frames 1216 to 1218.
[0110] Fig.16 The method is shown to be executed for a case where at least one target identifier is a transient target identifier. Figure 12 to Figure 13 Another instantiation of process 1202 is shown as process 1602. In block 1604, after executing Fig.12 Following blocks 1204 to 1212 of FIG. 12, the MMC 802 displays at least one transient target identifier associated with the candidate target destination on the user interface presentation 102 (such as, for example, in FIG. 12). Figure 2 ); when the output score of the candidate target destination meets the specified criteria, the MMC 802 performs the operation. In block 1606, after executing Fig.13 Following blocks 1216 through 1218 of FIG. 104 , if the user moves the source object to the target destination, the MMC 802 removes the transient target identifier(s) from the user interface presentation 102 .
[0111] C. Representative Computational Functionality
[0112] Fig.17 A computing device 1702 is shown that can be used to implement any aspect of the mechanisms described in the above figures. Fig.17 The type of computing device 1702 shown in FIG. 1 may be used to implement Figure 71702. In all cases, computing device 1702 represents a physical and tangible processing mechanism.
[0113] The computing device 1702 may include one or more hardware processors 1704. The hardware processor(s) 1704 may include, but are not limited to, one or more central processing units (CPUs) and / or one or more graphics processing units (GPUs) and / or one or more application specific integrated circuits (ASICs), etc. More generally, any hardware processor may correspond to a general purpose processing unit or a special purpose processor unit.
[0114] The computing device 1702 may also include a computer-readable storage medium 1706, corresponding to one or more computer-readable medium hardware units. The computer-readable storage medium 1706 retains any kind of information 1708, such as machine-readable instructions, settings, data, etc. For example, but not limited to: the computer-readable storage medium 1706 may include one or more solid-state devices, one or more magnetic hard disks, one or more optical disks, magnetic tapes, etc. Any instance of the computer-readable storage medium 1706 may use any technology for storing and retrieving information. Further, any instance of the computer-readable storage medium 1706 may represent a fixed component or a removable component of the computing device 1702. Further, any instance of the computer-readable storage medium 1706 may provide volatile retention or non-volatile retention of information.
[0115] The computing device 1702 may utilize any instance of the computer-readable storage medium 1706 in different ways. For example, any instance of the computer-readable storage medium 1706 may represent a hardware memory unit (such as a random access memory (RAM)) for storing transient information during execution of a program by the computing device 1702 and / or a hardware storage unit (such as a hard disk) for more permanent retention / archiving of information. In the latter case, the computing device 1702 also includes one or more drive mechanisms 1710 (such as a hard disk drive mechanism) for storing information and retrieving information from an instance of the computer-readable storage medium 1706.
[0116] When hardware processor(s) 1704 implement computer-readable instructions stored in any instance of computer-readable storage medium 1706, computing device 1702 can perform any of the functions described above. For example, computing device 1702 can implement computer-readable instructions to perform each block of the process described in Section B.
[0117] Alternatively, or in addition, the computing device 1702 may rely on one or more other hardware logic components 1712 to perform operations using a task-specific set of logic gates. For example, the (multiple) hardware logic components 1712 may include hardware logic gates of a fixed configuration, for example, the hardware logic gates are created and set at the time of manufacturing and cannot be changed thereafter. Alternatively, or in addition, the (multiple) other hardware logic components 1712 may include a set of programmable hardware logic gates that can be set to perform different application-specific tasks. The latter class of devices includes, but is not limited to: programmable array logic devices (PALs), general array logic devices (GALs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), etc.
[0118] Fig.17 Hardware logic circuitry 1714 is generally indicated to correspond to any combination of hardware processor(s) 1704, computer-readable storage medium 1706, and / or other hardware logic component(s) 1712. That is, computing device 1702 may employ any combination of hardware processor(s) 1704 that execute machine-readable instructions provided in computer-readable storage medium 1706 and / or one or more other hardware logic component(s) 1712 that perform operations using a fixed and / or programmable set of hardware logic gates.
[0119] In some cases (e.g., where computing device 1702 represents a user computing device), computing device 1702 also includes an input / output interface 1716 for receiving various inputs (via input devices 1718) and for providing various outputs (via output devices 1720). Illustrative input devices include a keyboard device, a mouse input device, a touch screen input device, a digitizer tablet, one or more still image cameras, one or more video cameras, one or more depth camera systems, one or more microphones, a voice recognition mechanism, any movement detection mechanism (e.g., accelerometers, gyroscopes, etc.), etc. A specific output mechanism may include a display device 1722 and an associated graphical user interface presentation (GUI) 1724. Display device 1722 may correspond to a liquid crystal display device, a light emitting diode display (LED) device, a cathode ray tube device, a projection mechanism, etc. Other output devices include a printer, one or more speakers, a tactile output mechanism, an archiving mechanism (for storing output information), etc. The computing device 1702 may also include one or more network interfaces 1726 for exchanging data with other devices via one or more communication pipes 1728. One or more communication buses 1730 communicatively couple the above-mentioned components together.
[0120] The communication conduit(s) 1728 may be implemented in any manner, for example, through a local area computer network, a wide area computer network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication conduit(s) 1728 may include any combination of hardwired links, wireless links, routers, gateway functionality, name servers, etc., controlled according to any protocol or combination of protocols.
[0121] Fig.17 Computing device 1702 is shown as being composed of a discrete collection of separate units. In some cases, the collection of units may correspond to discrete hardware units provided in a computing device rack having any form factor. Fig.17 An illustrative form factor is shown at its bottom. In other cases, computing device 1702 may include an integrated Figure 1 For example, computing device 1702 may include a system on a chip (SoC or SOC), corresponding to a combination of Fig.17 An integrated circuit that incorporates the functionality of two or more of the units shown.
[0122] The following summary provides a non-exhaustive collection of illustrative aspects of the techniques described herein.
[0123] According to a first aspect, one or more computing devices for supporting movement of an object to a target destination are described herein. The device(s) include hardware logic circuitry corresponding to: (a) one or more hardware processors that perform operations by executing machine-readable instructions stored in a memory, and / or (b) one or more other hardware logic components that perform operations using a task-specific set of logic gates. The operations include: for at least one pair of source objects and candidate target destinations: receiving a signal indicating that a user has selected a source object to be moved, the source object being presented in a source domain of a user interface presentation displayed on a display device; identifying a first feature set associated with the source object, at least one feature in the first feature set describing characteristics of content associated with the source object; identifying a second feature set associated with the candidate target destination, at least one feature in the second feature set describing characteristics of content associated with the candidate target destination; using semantic analysis to determine a relationship between the first feature set and the second feature set to provide an output score; displaying a representation of the candidate target destination on the user interface presentation, the representation being visually enhanced to a degree dependent on the output score; receiving an instruction from the user to move the source object to a representation of the selected candidate target destination; and moving the source object on the user interface presentation to the representation of the selected target destination, which associates the source object with the selected target destination.
[0124] According to a second aspect, the operations further include determining scores for a plurality of candidate target destinations and presenting a representation of each candidate target destination that is enhanced by an amount that is a function of the respective score of the candidate target destination.
[0125] According to a third aspect, the operation of displaying a representation of the target destination is performed when it takes a user more than a prescribed amount of time to move the source object to the selected target destination.
[0126] According to a fourth aspect, candidate target destinations are associated with resources local to the user.
[0127] According to a fifth aspect, a candidate target destination is associated with a resource that is remote from the user.
[0128] According to a sixth aspect, the operations further include: prior to the movement of the source object, displaying on the user interface presentation a set of on-screen target identifiers, one of the on-screen target identifiers corresponding to the candidate target destination. In that case, the operation of displaying the presentation includes: changing a characteristic of the on-screen target identifier associated with the candidate target destination.
[0129] According to a seventh aspect, the changed characteristics of the on-screen target identifier control: the appearance of the on-screen target identifier; and / or the rating-related labels associated with the on-screen target identifier; and / or the order in which the on-screen target identifier appears in a user interface presentation relative to other on-screen target identifiers.
[0130] According to an eighth aspect, the operation of displaying a representation of a candidate target destination includes displaying a transient target identifier of the candidate target destination on a user interface presentation. The transient target identifier appears on the user interface presentation in response to a user moving a source object.
[0131] According to a ninth aspect (depending on the eighth aspect), the operations further include: if the user moves the source object to any target destination, removing the transient target identifier from the user interface presentation.
[0132] According to a tenth aspect (depending on the eighth aspect), the candidate target destination includes a plurality of target sub-destinations. Further, the transient target identifier includes a set of sub-target identifiers associated with the corresponding target sub-destinations.
[0133] According to the eleventh aspect (depending on the eighth aspect), the operation also includes: determining that a specific transient target identifier has been previously displayed on a user interface presentation a number of times that meets a specified threshold; and associating the specific transient target identifier with a specific physical portion of the screen presenting the user interface presentation.
[0134] According to the twelfth aspect, the target destination is associated with one or more target objects. Further, the operation of identifying the second feature set includes: identifying features associated with the target object(s).
[0135] According to the thirteenth aspect 13, the operation of identifying the first set of features and / or the operation of identifying the second set of features comprises: identifying at least one feature that depends on a previous behavior of a user when moving a source object to a target destination.
[0136] According to a fourteenth aspect, the operation of identifying a first set of features and / or the operation of identifying a second set of features comprises identifying at least one feature that depends on the history of the object under consideration.
[0137] According to a fifteenth aspect, the operation of identifying a second feature set includes: predicting a target destination to which a user moves a source object; and identifying at least one feature describing the predicted target destination.
[0138] According to a sixteenth aspect, the operation of using semantic analysis involves: using a machine-trained model to provide an output score.
[0139] According to the seventeenth aspect, a method for supporting the movement of an object to a target destination is described. The method includes: for at least one pair of source objects and candidate target destinations: receiving a signal indicating that a user has selected a source object to be moved, the source object being presented in a source domain presented on a user interface displayed on a display device; identifying a first feature set associated with the source object, at least one feature in the first feature set describing the characteristics of content associated with the source object; identifying a second feature set associated with the candidate target destination, at least one feature in the second feature set describing the characteristics of content associated with the candidate target destination; using semantic analysis based on a machine-trained model to determine the relationship between the first feature set and the second feature set to provide an output score; displaying a representation of the candidate target destination on the user interface presentation, the representation being visually enhanced to a degree that depends on the output score; receiving an instruction from the user to move the source object to the representation of the selected target destination; and moving the source object on the user interface presentation to the representation of the selected target destination, which associates the source object with the selected target destination.
[0140] According to an eighteenth aspect, the method (according to the seventeenth aspect) further comprises: before the movement of the source object, displaying a set of on-screen target identifiers on the user interface presentation, one of the on-screen target identifiers corresponding to the candidate target destination. Further, the operation of displaying the presentation comprises: changing the characteristics of the on-screen target identifier associated with the candidate target destination.
[0141] According to a nineteenth aspect (depending on the seventeenth aspect), the operation of displaying a representation of a candidate target destination includes: displaying a transient target identifier of the candidate target destination on a user interface presentation. The transient target identifier appears on the user interface presentation in response to the user moving the source object. Further, the method includes: if the user moves the source object to any target destination, removing the transient target identifier from the user interface presentation.
[0142] According to the twentieth aspect, a computer-readable storage medium for storing computer-readable instructions is described. The computer-readable instructions perform a method when executed by one or more hardware processors. The method includes: for at least one pair of source objects and candidate target destinations: receiving a signal indicating that a user has selected a source object to be moved, the source object being presented in a source domain of a user interface presentation displayed on a display device; identifying a first feature set associated with the source object; identifying a second feature set associated with the candidate target destination; using semantic analysis to determine the distance between the first feature set and the second feature set to provide an output score; when the output score of the candidate target destination reaches a specified standard, displaying a transient target identifier associated with the candidate target destination on the user interface presentation; receiving an instruction from the user to move the source object to a selected transient identifier associated with the selected target destination; placing the source object on the selected transient target identifier; and when the user places the source object on the selected transient target identifier, removing the selected transient target identifier from the user interface presentation.
[0143] The twenty-first aspect corresponds to any combination of the first to twentieth aspects described above (eg, any arrangement or subset that is not logically contradictory).
[0144] The twenty-second aspect corresponds to any method counterpart, device counterpart, system counterpart, component plus function counterpart, computer-readable storage medium counterpart, data structure counterpart, product counterpart, graphical user interface presentation counterpart, etc. associated with the first to twenty-first aspects.
[0145] Finally, the functionality described herein may employ various mechanisms to ensure that any user data is handled in a manner consistent with applicable laws, social norms, and the expectations and preferences of individual users. For example, the functionality may allow a user to explicitly opt-in (and then, explicitly opt-out) of the functionality's provisions. The functionality may also provide suitable security mechanisms (such as data cleansing mechanisms, encryption mechanisms, password protection mechanisms, etc.) for ensuring the privacy of user data.
[0146] Further, the specification may have presented various concepts in the context of illustrative challenges or problems. Such an interpretation is not intended to imply that others have understood and / or appreciated the challenge or problem in the manner specified herein. Further, such an interpretation is not intended to imply that the subject matter recited in the claims is limited to solving the identified challenges or problems; that is, the subject matter in the claims may be applied in the context of challenges or problems other than those described herein.
[0147] Although the subject matter has been described in terms of structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
Claims
1. One or more computing devices for supporting movement of an object to a target destination, comprising: A hardware logic circuit device, the hardware logic circuit device corresponding to: (a) one or more hardware processors that perform operations by executing machine-readable instructions stored in a memory, or (b) one or more other hardware logic components that use a task-specific set of logic gates to perform the operations, or (a) and (b), the operations comprising: for at least one pair of a source object and a candidate target destination: receiving a signal indicating that a user has selected the source object to be moved, the source object being presented in a source field of a user interface presentation displayed on a display device; identifying a first set of features associated with the source object, at least one feature in the first set of features being generated by a content inspection process that maps the source object into a semantic space representation of a subject associated with the source object; identifying a second set of features associated with the candidate target destination, at least one feature in the second set of features being generated by the content inspection process, the content inspection process mapping the candidate target destination into a semantic space representation of a topic associated with the candidate target destination, at least one other feature of the second set of features is dependent on a history of a target object associated with the candidate target destination, the history comprising information at least about the creation of the target object; using semantic analysis to determine a relationship between the first set of features and the second set of features in a semantic space to provide an output score; displaying, on the user interface presentation, a representation of the candidate target destination, the representation visually enhanced to a degree dependent on the output score; receiving an instruction from the user to move the source object to a representation of a selected candidate target destination; and moving the source object on the user interface presentation to the representation of the selected target destination, which causes the source object to become associated with the selected target destination, The content inspection process is applied to different types of candidate target destinations, including candidate target destinations associated with applications.
2. One or more computing devices according to claim 1, The operations also include: prior to movement of the source object, displaying a set of on-screen target identifiers on the user interface presentation, the on-screen target identifiers corresponding to respective candidate target destinations, and Wherein the displaying of the representation includes changing a rating label of an on-screen target identifier associated with the candidate target destination.
3. One or more computing devices according to claim 1, The displaying of the candidate target destination comprises: displaying a transient target identifier of the candidate target destination on the user interface presentation, Wherein the transient target identifier appears on the user interface presentation in response to the user moving the source object.
4. One or more computing devices according to claim 3, The candidate target destination includes a plurality of target secondary destinations, and wherein the transient target identifier includes a set of sub-target identifiers associated with corresponding target sub-destinations, The displaying of the candidate target destination comprises: The sub-destination identifiers are spatially arranged in an order based on the relevance of the source object to each target sub-destination as assessed by the semantic analysis.
5. The one or more computing devices of claim 3, wherein the operations further comprise: determining that a particular transient target identifier has been previously displayed on the user interface presentation a number of times satisfying a specified threshold; as well as The particular transient target identifier is associated with a particular physical portion of a screen on which the user interface presentation is presented, and display of the particular transient target identifier on the user interface presentation is thereafter omitted.
6. One or more computing devices according to claim 3, wherein the transient identifier shows a representation of a content item associated with the target object, and Wherein the instructions from the user specify instructions to move the source object to a particular location within the representation of the content item.
7. The one or more computing devices of claim 1, wherein the identifying the first feature set, the identifying the second feature set, or both the identifying the first feature set and the identifying the second feature set comprises: At least one characteristic that depends on previous behavior of the plurality of users in moving the source object to the target destination is identified.
8. The one or more computing devices of claim 1, wherein the usage semantic analysis comprises: A machine trained model is used to provide the output score.
9. The one or more computing devices of claim 1, wherein said identifying said first set of features comprises: At least one other characteristic is identified which depends on the history of the source object under consideration, the history of the source object comprising at least information about the creation of the source object.
10. The one or more computing devices of claim 1, wherein the at least one other characteristic comprises information about a person or entity that created the target object.
11. The one or more computing devices of claim 1, wherein the at least one other characteristic comprises information about a time when the target object was created.
12. The one or more computing devices of claim 1, wherein the at least one other characteristic comprises information about at least one source from which the target object was obtained.
13. The one or more computing devices of claim 1, wherein the at least one other characteristic comprises information about previous movement of the target object.
14. The one or more computing devices of claim 1, wherein the at least one other characteristic comprises information about a person with whom the target object has been shared.
15. The one or more computing devices of claim 1, wherein the operations further comprise: An explanation is displayed on the user interface presentation, the explanation identifying why the representation of the candidate target destination has been enhanced.
16. One or more computing devices according to claim 1, wherein the received signal indicates that the user is simultaneously moving a plurality of source objects including the source object, The candidate target destination includes multiple target sub-destinations. wherein using the semantic analysis includes using the semantic analysis to determine a relationship between each source object and each target sub-destination, and The movement includes: Each source object is automatically distributed to a specific target sub-destination based on the semantic analysis.
17. The one or more computing devices of claim 1, wherein the semantic space representation of a topic associated with the source object is a semantic vector generated by a machine-trained model.
18. The one or more computing devices of claim 1, wherein the semantic space representation of the topic associated with the candidate target destination is a semantic vector generated by a machine-trained model.
19. A method for supporting movement of an object to a target destination, comprising: For at least one pair of source object and candidate target destination: receiving a signal indicating that a user has selected the source object to be moved, the source object being presented in a source field of a user interface presentation displayed on a display device; identifying a first set of features associated with the source object, at least one feature in the first set of features being generated by a content inspection process that maps the source object into a semantic space representation of a subject associated with the source object, at least one other feature of the first set of features depends on a history of the source object, the history comprising information at least about the creation of the source object; identifying a second set of features associated with the candidate target destination, at least one feature in the second set of features being generated by the content inspection process, the content inspection process mapping the candidate target destination into a semantic space representation of a topic associated with the candidate target destination, at least one other feature of the second set of features is dependent on a history of a target object associated with the candidate target destination, the history of the target object comprising information at least about the creation of the target object; Based on the machine-trained model, using semantic analysis to determine a relationship between the first feature set and the second feature set in a semantic space to provide an output score; displaying, on the user interface presentation, a representation of the candidate target destination, the representation visually enhanced to a degree dependent on the output score; receiving an instruction from the user to move the source object to a representation of a selected target destination; as well as moving the source object on the user interface presentation to the representation of the selected target destination, which associates the source object with the selected target destination, The content inspection process is applied to different types of candidate target destinations, including candidate target destinations associated with applications.
20. A computer-readable storage medium for storing computer-readable instructions, the computer-readable instructions, when executed by one or more hardware processors, performing a method comprising: For at least one pair of source object and candidate target destination: receiving a signal indicating that a user has selected the source object to be moved, the source object being presented in a source field of a user interface presentation displayed on a display device; identifying a first set of features associated with the source object; identifying a second set of features associated with the candidate target destination; determining a distance between the first set of features and the second set of features in a semantic space using semantic analysis to provide an output score; displaying a transient destination identifier associated with the candidate target destination on the user interface presentation when the output score of the candidate target destination meets a specified criterion; receiving an instruction from the user to move the source object to a selected transient identifier associated with a selected target destination; placing the source object on the selected transient target identifier; as well as when the user drops the source object over the selected transient target identifier, removing the selected transient target identifier from the user interface presentation, at least one feature identifies information related to the creation of the source object or the candidate target destination, and at least one feature is generated by a content inspection process that maps at least the target candidate destination into a semantic space representation of a topic associated with the target candidate destination, The content inspection process is applied to different types of candidate target destinations, including candidate target destinations associated with applications.
