Projection, control, and management of user device application using connection resource

Abstract

To provide a device, a non-transitory computer-readable medium, and a method that enable interaction between a user device and a connection resource.SOLUTION: There is provided a device (gateway 100) comprising one or more processors, the one or more processors are configured to execute the steps of: establishing a connection from a user device 110 to a connection resource 120; projecting, from the user device 110 to the connection resource 120, content received from a source application; calibrating, at the user device 110, at least one input of the connection resource 120 to at least one input of the user device 110; receiving, at the user device 110, user input information from the connection resource 120; and applying the received user input information to the source application.SELECTED DRAWING: Figure 1

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 62 / 954,374, filed December 27, 2019. U.S. Patent Application Publication No. 2014 / 0156734, published June 5, 2014, is incorporated herein by reference. [Background technology]

[0002] User devices such as smartphones, tablets, etc. are ubiquitous. Similarly, connected resources such as in-vehicle systems are widely available. Thus, there is a need to enable interaction between user devices and connected resources. [Brief explanation of the drawings]

[0003] The novel features of the invention are set forth in the appended claims, while certain embodiments are shown by way of illustration in the following drawings.

[0004] [Figure 1] FIG. 1 illustrates an overview of one or more exemplary embodiments described herein in which applications are projected from a user device to connected resources via a gateway.

[0005] [Figure 2] FIG. 1 illustrates an overview of one or more exemplary embodiments described herein in which user interface events are received at a connected resource and applied to a user device via a gateway.

[0006] [Figure 3] FIG. 1 provides an overview of one or more example embodiments described herein in which applications are analyzed, recognized, and filtered using a gateway.

[0007] [Figure 4]1 illustrates an example of an image analysis method according to one or more embodiments described herein, in which adjacent image sections are compared to identify matching blocks.

[0008] [Figure 5] FIG. 1 illustrates an example environment in which one or more embodiments described herein may be implemented.

[0009] [Figure 6] FIG. 1 illustrates another example of an environment in which one or more embodiments described herein may be implemented.

[0010] [Figure 7] FIG. 1 illustrates a flowchart of an example process for projecting application content from a user device to a connected resource.

[0011] [Figure 8] FIG. 1 illustrates a flowchart of an example process for identifying applications executed by a user device.

[0012] [Figure 9] FIG. 1 illustrates a flowchart of an exemplary process for receiving a command at a connection resource and applying the received command at a user device.

[0013] [Figure 10] FIG. 1 illustrates a flowchart of an example process for filtering application content based on the operational state of a connection resource.

[0014] [Figure 11] FIG. 1 illustrates a flowchart of an exemplary process for applying machine learning to generate or update an application recognition model.

[0015] [Figure 12]1 is a schematic block diagram of one or more exemplary devices that may be used to implement various embodiments. Detailed Description of the Invention

[0016] The following detailed description sets forth generally possible ways of carrying out exemplary embodiments. This description is not to be taken in a limiting sense, but is made merely for the purpose of illustrating the general principles of some embodiments, the scope of the present invention being best defined by the appended claims.

[0017] Various features are described below, each of which can be used independently of one another or in combination with other features. Generally, some embodiments generally provide a gateway that projects user device application content (e.g., streaming video) to a connected resource. The user device can be an electronic device such as a smartphone, tablet, wearable device, etc. The connected resource can be an electronic device, system, or component such as a vehicle head unit. The application content can be associated with any application executable by the user device, such as a media streaming service, game, etc.

[0018] User input received at the connected resource can be applied at the user device via the gateway. Such user input can include, for example, input received via a touchscreen or other user interface (UI) element of the connected resource. The user input can be applied at the user device by converting the received user input data as needed and emulating the received input on the user device (and user device applications).

[0019] Applications can be automatically recognized by the gateway so that application content can be filtered or otherwise processed or managed as necessary. For example, video streaming applications can be identified so that the provision of video content can be restricted while the vehicle is operating. Various recognition algorithms, such as color bin matching, can be used to recognize such applications.

[0020] In some embodiments, machine learning may be used to train models related to application or content recognition, rules related to operational constraints, and / or other elements available to perform various processes or algorithms related to application content projection, control, management, analysis, etc.

[0021] 1 provides an overview of one or more exemplary embodiments described herein, in which a gateway 100 is used to project a source application from a user device 110 to a connected resource 120. As shown, the user device 110 can project the source application to the connected resource 120 through the gateway 100. Additionally, the gateway 100 can receive UI events from the connected resource 120 and apply the UI events in the source application running on the user device 110.

[0022] Gateway 100 may be constructed, at least in part, by user devices 110, connectivity resources 120, and / or other suitable resources (e.g., a cloud-based server or another network-accessible resource). User devices 110 may be smartphones, tablets, wearable devices, and / or other suitable devices or components capable of executing instructions and / or communicating with other devices over one or more wired or wireless channels. Connectivity resources 120 may be vehicle head units, entertainment systems or devices, and / or other suitable resources capable of providing content and / or processing UI data. Connectivity resources 120 may typically comprise a display screen.

[0023] In this example, user device 110 may run a navigation application. The navigation application may be mirrored on connected resources 120 via gateway 100. User device 110 may communicate with connected resources 120 over a local wireless channel, such as Bluetooth or Wi-Fi. Gateway 100 may receive application content from the navigation application and provide the application content to connected resources 120 in an appropriate format, such as by converting it to bitmap images or vector data.

[0024] The application content in this example may include image data such as map data, route indicators, etc. Additionally, the application content may include various UI elements, such as buttons, which may be provided as image or other graphical data. Additionally, the connection resource 120 may include, provide, or otherwise perform various UI functions associated with the source application. For example, the display may re-center on a particular point or area based on a single tap identified at the location of the specified point or area. As another example, the display may zoom in or out by varying the distance between two touch points (i.e., "pinch" zoom). As another example, a double tap may be associated with selecting a destination for navigation.

[0025] In this example, a tap-and-drag event can be detected by connectivity resource 120, and relevant data can be sent to user device 110 via gateway 110 for analysis and possible action by the source application. The tap-and-drag event can update the displayed map area so that the selected point moves to a new location. The source application can update its output display based on the received UI event data and provide the updated display (reflecting the new location of the selected point) to connectivity resource 120 via gateway 100.

[0026] 2 provides an overview of one or more exemplary embodiments described herein, in which user interface events are received at a connected resource 120 and applied to a user device 110 via a gateway 100. The user device 110 and the connected resource 120 may be communicatively connected over a secure channel, such as a Bluetooth or Wi-Fi link. Communication between the resources may include and / or utilize various messaging protocols. The communication channel may utilize encryption and / or other security features.

[0027] In this example, the user device 110 includes a display screen 205, such as a touchscreen display. The user device 110 may include various other UI elements, such as buttons, a keypad, a mouse, a joystick, etc. The display screen 205 may be associated with one or more target action areas 210. In this example, the target action areas 210 may be associated with all or substantially all of the surface of the display screen 205. The target action areas 210 may include areas on the display screen 205 (and / or other UI elements) associated with user input actions (or "events"). The target action areas 210 may utilize a coordinate system associated with the display screen 205.

[0028] The connection resource 120 may include a display screen 215, such as a touchscreen display. The connection resource 120 may include various other UI elements, such as buttons, a keypad, a joystick, a mouse, and the like.

[0029] The gateway 100 can replace various UI features associated with the user device 110 to provide similar functionality using UI features associated with the connection resources 120. In this example, the display screen 215 includes a projected client area 220, one or more source action areas 225, a projected target area 230, one or more input ignore areas 235, and one or more gateway input areas 240. Other embodiments may include various types of areas (and / or other UI features) in different configurations than those shown. For example, each area may have various shapes other than a rectangle (e.g., circular, oval, polygonal, etc.). As another example, various single areas may be split into multiple areas or multiple areas may be combined into a single area.

[0030] The projected client area 220 may define one or more areas where events, such as "point" events (e.g., touches, clicks, etc.), need to be sent to the connected user device 110. The projected client area 220 may be represented as a rectangle and may be specified in the coordinate system of the display screen 215. Each of the other areas associated with the connection resource 0120 may be described in and with respect to the coordinate system of the projected client area 220. The simplest shape of an area is a rectangle, although a variety of other shapes may be available and / or supported.

[0031] Each source action area 225 can be an area associated with the projected client area 220 that defines whether the event data should be identified as an external UI event or a gateway user input event (or ignored). In some embodiments, any event outside the source action area 225 and inside the projected client area 220 can be classified (and / or applied) as a gateway UI event. The source action area 225 can include a main action area and / or an ordered list of additional area items that can define one or more sets of excluded and / or included sub-areas.

[0032] The projection target area 230 may correspond to the target action area 210 or other area associated with the display screen 205 that is projected onto the connected resource 120. The projection target area 230 may be described in and with respect to the coordinate system of the projection client area 220. The image captured by the user device 110 may be scaled to fit the dimensions of the projection target area 230.

[0033] Each input ignore region 235 can be an region where UI events may be ignored. Input received through an input ignore region 235 should not be used to identify external input events or gateway input events. UI events associated with an input ignore region 235 can be associated with notifications in some embodiments.

[0034] Each gateway input area 240 may be associated with an event that may be identified as a gateway user input event or as a user input event related to the management of the gateway 100 and / or any associated resources.

[0035] In various embodiments, various different types of regions may be included based on the role of each region (and associated events). For example, some embodiments may include one or more external input regions, in which case any events associated with these regions may be identified as "external user input" events.

[0036] If regions overlap, each touch event may be evaluated to determine which region the touch should be associated with. For example, regions may be matched based on order, with the most recently touched region taking precedence over previously touched regions.

[0037] As shown, a UI event 245 (e.g., associated with a touch or "click" event) may be received at the projection target area 230 and emulated as if a click had been applied at a corresponding click point 250 in the target action area 210. The emulated UI event 250 may be recognized by and applied by (or to) a source application executing on the user device 110. For example, when using a navigation application, a click at location 250 (e.g., by placing the click location at the center of the displayed area) may cause the displayed map area to be updated.

[0038] In this example, gateway 100 may include, store, or utilize various sets of display attributes 255 and / or input ranges 260. Such display attributes 255 may include, for example, the size of the display screen of the user device and / or connection resource. Input ranges 260 may include various definitions relating to different shapes, sizes, and / or coordinates of regions. For example, input ranges 260 may specify the locations and sizes of regions 220-240 and region 210.

[0039] The gateway 100 may utilize various other data elements, such as lists of input event types, parameters or attributes associated with the events, thresholds or other criteria for identifying events, and / or other relevant data.

[0040] In some embodiments, user input for the connection resource 120 can be initialized after establishing a connection between the user device 110 and the connection resource 120 via the gateway 100. The connection resource 120 can send an input capabilities message indicating supported input capabilities and associated functions. The gateway 100 can respond with a list of the user device 110's available input capabilities in the input capabilities message. The gateway 100 can send an input settings message to the connection resource 120 (and / or associated client application) specifying various input areas (or other capabilities) and associated usage profiles. The gateway 100 can also specify the projection target area 230. The connection resource 120 (and / or associated client application) can store the provided input areas. If any external user input areas are defined or recognized, the connection resource 120 can send such definitions to its device control module. The connection resource 120 can respond to the gateway 100 with an input settings message, which may include a device identifier (ID) for each of the defined input areas.

[0041] The gateway 100 can reconfigure the input settings at any time by resending the input setting message. After configuration, the gateway 100 can enable input processing for one or more input areas. The gateway 100 can send a control input message specifying the area IDs of the input areas to enable and the area IDs of the input areas to disable. Multiple messages can be sent at any time to enable or disable one or more input areas.

[0042] 3 provides an overview of one or more example embodiments described herein in which applications are analyzed, recognized, and filtered using a gateway 100. As shown, a user device 110 may be associated with a navigation application 310, one or more device sensor interfaces 320, and a multimedia application 330.

[0043] The device sensors 320 can include various position sensors (e.g., accelerometers, gyroscopes, global positioning system (GPS) components, etc.) and / or other sensors capable of determining operating conditions. For example, such device sensors 320 can be used to determine when a vehicle is stationary or moving. The user device 110 can also receive sensor data from the connectivity resources 120 or other external resources. For example, the user device 110 can receive vehicle speed sensor data, parking brake status, and / or other sensor data via the connectivity resources 120.

[0044] The gateway 100 can receive device sensor data (and / or other suitable data) and determine operational states associated with the connectivity resources 120. The operational states can be associated with various types of connectivity systems or resources. For example, a display integrated into a vehicle head unit installed in a passenger vehicle can be associated with states such as "stationary," "driving," etc. As another example, a display integrated into an in-flight entertainment system can be associated with states such as "taxi," "takeoff," "cruise," "in-flight announcements," etc.

[0045] As shown, if gateway 100 is unable to receive application information from a user device interface (not shown), gateway 100 can utilize various recognition models 340 (also referred to herein as “matching” models), tolerances 350, and / or other suitable data elements. Such recognition models 340 can include image data and / or other data that can be utilized to identify applications running on user device 110. For example, recognition model 340 can include a captured image of an application launch screen or other suitable captured application content (e.g., video content, audio content, graphical content, etc.). Recognition model 340 can include a set of evaluation criteria (e.g., a matching threshold, a minimum matching score, or other metric, etc.).

[0046] The permissions 350 may include instructions for source application elements to project to the connectivity resource 120. For example, a video streaming application (or content element) may be disabled when the vehicle is in motion to reduce driver distraction. Application permissions 350 may be associated with an application, a device type, an operating state, and / or other relevant attributes. For example, the permissions 350 for a video streaming application may instruct a connectivity resource 120 incorporated in a front seat vehicle head unit that video streaming should be disabled while the vehicle is in motion. Video streaming may not be disabled for a connectivity resource 120 incorporated in a rear seat (or passenger seat) vehicle head unit, regardless of whether the vehicle is in motion.

[0047] In a first example, the navigation application 310 may be associated with a full coverage projection area 360 in which all available content is delivered to the display screen 215 regardless of operating state or other attributes.

[0048] In a second example, a multimedia application 330 may be associated with a partially applied projection area 370 in which some content related to the multimedia application 330 is disabled or blocked via a blocked area 380. In this example, the multimedia application 330 may provide audio and video content related to podcasts. In such a case, the partially applied projection area 370 may include providing audio content (and / or other non-video content) regardless of the operating state. Some of the application content may be associated with streaming video presented in the blocked area 380, unless constrained by operating state-based permissions (e.g., blocking video content while moving).

[0049] The unavailability area 380 (and / or other types of content filtering) can be applied based on predefined application elements (e.g., a video display window associated with the application) and / or automatically detected application elements. For example, the projected content can be analyzed to identify regions of one or more frames associated with the video content (e.g., detecting motion by comparing differences between frames). Such content analysis can include analyzing various distracting (or undesirable) application content. For example, dazzling or distracting colors can be dimmed. As another example, loud noises or noises that fit various profiles (e.g., car horns, sirens, screeching tires, etc.) can be muted or presented at a reduced volume.

[0050] 4 illustrates an example of an image analysis technique of one or more embodiments described herein that analyzes adjacent image sections or regions to identify matching blocks. The gateway 100 may select from a variety of image analysis algorithms based on a variety of suitable criteria. For example, the algorithm may be selected to best utilize the capabilities of the user device 110 while meeting a specified accuracy threshold. The selection of the algorithm is described in more detail with reference to process 700 below.

[0051] The captured image 410 may be received from the active application via capturing a projected screen and / or other information related to the active application. A library of similar images associated with various user device applications may be maintained by the gateway 100 and / or other suitable resources. Image analysis and matching model generation are described in more detail with reference to process 1100 below.

[0052] In some embodiments, the gateway 100 can use color profile matching to analyze images and match a captured image (or portion thereof) with a previously captured application screen (or portion thereof). Many or most applications can use a consistent color scheme for the various screens associated with or generated by the application. Therefore, color profile matching can provide effective and efficient results in identifying applications.

[0053] During training of the matching model, a color profile may be created for each of the source application screens (i.e., "source screens"). Similarly, a color profile may be determined for a newly captured image 410 to determine whether the color profile of the captured image 410 matches the color profile of any of the source screens. A matching score, or other determined metric, may be compared to a matching threshold to determine whether the captured image 410 matches the captured application screen. Such a color profile matching algorithm may execute relatively quickly while requiring only limited processor and memory usage.

[0054] A color profile can be determined for the current image 410. The color profile can be determined by evaluating sections of the image (e.g., individual pixels, groups of pixels, or "blobs" of pixels). The color profile can include any number of color "bins" and a number of sections (e.g., the number of pixels) associated with each bin. Each pixel (or other image section) can be evaluated, and the pixel color can be converted to a color bin. The number of bits for each color channel can be limited (e.g., using a color bin channel bits parameter) to more efficiently analyze and store the profile data. The color channels for each pixel (or other image section) can be combined into a single value (e.g., using conventional bitwise operations), such that each pixel or other image section is associated with a single color bin value that can serve as a "key" in a color profile "dictionary." If the key is already in the dictionary, the count associated with the key can be incremented. If the key is not in the dictionary, a new entry can be added to the dictionary and the associated count can be set to 1. If the associated count is below a threshold, the key can be removed from the dictionary.

[0055] The resulting color profile can be compared to various stored color profiles in various suitable ways. For example, the color bins associated with the captured image 410 can be compared to the color bins associated with a matching model. The number of redundant or additional color bins associated with the captured image 410 relative to the number of color bins associated with the model can be determined. Similarly, the number of matching color bins between the captured image 410 and the model can be determined.

[0056] A matching score can be calculated based on the number of matching color bins and the number of excess color bins. In some embodiments, the matching score can be a probability represented by a number ranging from 0 to 1. The number of matching color bins can be compared to a minimum matching threshold number. If the number of matching color bins is less than the minimum matching threshold number, the matching score can be set to zero. If the number of excess color bins is greater than the maximum matching threshold number, the matching score can be set to zero. If the number of excess color bins is less than or equal to the minimum excess matching threshold number, the matching score can be set to 1. If the number of matching color bins is greater than the minimum matching threshold number, the number of excess color bins is less than the maximum matching threshold number, and the number of excess color bins is greater than the minimum excess matching threshold number, the matching score can be calculated by dividing the number of matching color bins by the total number of image color bins. In some embodiments, the matching score can include color bin weighting, where each color bin is associated with a count or other weighting factor. If the calculated matching score exceeds the matching threshold, a matching application may be identified by the gateway 100 and associated with the calculated score.

[0057] Such pixel-type (or other region or section type) color matching algorithms work well for application screens with solid color widgets, such as buttons, background rectangles, and circles. However, some application screens may present photographic images or other graphics that do not have many continuous areas of matching color, for example, due to slight variations in light or shade gradients. For example, a navigation application may display a "street view" of a destination. As another example, a restaurant search application may display thumbnail photos of restaurants in a results list. In such cases, counting individual colors would result in a photographic (or other high-resolution) image producing a large number of colors that are not useful for the matching calculation. To overcome such limitations, the gateway 100 may utilize or implement a runtime filter while counting colors to count only colors in areas with multiple subelements (also called "blobs") that match the color.

[0058] In the example of FIG. 4, a "blob" analysis is used by dividing the image 410 into sections 420 (e.g., pixels, groups of pixels, shape regions, etc.) rather than analyzing each section independently. In this example, each section 420 may correspond to one pixel. For such analysis, the color bins of neighboring pixels may be evaluated. If some matching criteria (e.g., a minimum number of neighboring matching elements) are met, the color bin associated with the pixel or section under evaluation may be added to a dictionary (or the count of matching color bins may be incremented). If the matching criteria are not met, the pixel or other section may be skipped (e.g., the color bin cannot be added to the dictionary and the count cannot be incremented).

[0059] In this example, each section 420 can be associated with a single pixel (center pixel "A"). As shown, in the zoom 430, multiple neighboring elements (in this example, pixels "B" through "I") can be evaluated to determine whether the pixel under evaluation (pixel A) is included in a color blob. Each pixel in the captured image 410 can be evaluated in a similar manner.

[0060] Each pixel can be evaluated against its neighboring pixels, and if there is a minimum number of neighboring pixels that match the central pixel (e.g., determined by having the same color bin), the central pixel is identified as part of the blob and added to the color profile dictionary; otherwise, the central pixel is skipped (e.g., the associated color bin is not added to the color profile dictionary or the count associated with the color bin is not incremented).

[0061] The simplest neighboring pixel evaluation window is 3x3, as shown. In some embodiments, identifying a blob requires at least two neighboring pixels to have the same color bin as the central pixel. For example, in this example, shaded pixels A, B, and C may have matching color bins, and pixel A may therefore be identified as part of a blob and added to the color profile dictionary. In some embodiments, a match may be evaluated based on neighboring pixels. For example, in this example, neighboring pixels B and C, which are adjacent to each other, match central pixel A, so a match can be identified; however, if neighboring pixels B and G, which are not adjacent to each other, match central pixel A, pixel A cannot be identified as part of a blob.

[0062] In various embodiments, different regions, shapes, matching criteria, etc. can be used to define or evaluate color blobs. For example, using a smaller evaluation window allows for faster image processing, but also captures smaller (and therefore more) blobs. In the example 3x3 evaluation window, the smallest blob may contain three pixels, as described above. A larger evaluation window can be used to filter out smaller blobs. For example, in a 5x5 evaluation window, the smallest blob may be a 9-pixel (3x3) square. The size of the evaluation window may be specified in the matching model or may be based on some relevant criteria (e.g., image size, device processing power, etc.).

[0063] Matching model data related to blob matching may include elements such as a color profile, a color bin counting method, a color threshold, a number of color bin channel bits, a minimum threshold number of excess color bins, a maximum threshold number of excess color bins, a minimum matching score threshold, a blob window size or region definition, a blob matching count, and / or other related parameters.

[0064] The color profile can include information such as color bin identifiers and color bin counts, as described above. The color bin counting method can indicate the type of color bin counting used, such as all pixels or blobs only. The color threshold can specify the minimum number of pixels that must match a given color bin to be included in the matching score calculation. Such an approach can exclude colors that have few pixels and are not important factors in classifying the image. The number of color bin channel bits (or other limiting factor) can specify the number of bits to use from each color channel of the image for use in color bins. For example, in the common "RGB888" (8 bits per channel, red, green, and blue) format, an algorithm can reduce color variation by 50% by using 7 bits instead of 8 bits for each channel.

[0065] A minimum threshold number of excess color bins can specify the number of excess color bins above which the images are considered a perfect match (e.g., by assigning a matching score of 1, or 100 percent). A maximum threshold number of excess color bins can specify the number of excess color bins above which the images are considered a mismatch (e.g., by assigning a matching score of 0). A minimum threshold matching score can specify the minimum matching score above which the matching score is considered to indicate a match (e.g., in some embodiments, the minimum threshold matching score can be 8 / 10, or 80%). A blob window size can specify the number of pixels (or other sections) to be evaluated when using a blob-based algorithm (e.g., 9 pixels, 25 pixels, etc.). A blob window can be further associated with a data element such as a shape-defining element (e.g., indicating a square, circle, etc.). A blob matching count can specify the minimum number of matching elements to be considered a match. For example, in the 9-pixel region described above, at least two other pixels must match the center pixel, so the blob matching count can indicate a total of three pixels. Some embodiments may include other parameters related to blob matching, such as the association between matching pixels (e.g., in the example above, two adjacent pixels must match the central pixel).

[0066] Another example of an algorithm that can be used to match a screen capture image to an application is searching for specific image elements in a predetermined area of ​​the captured image (also known as "image template matching"). Such image template matching is often effective when applied to applications that include a unique logo or original artwork icon on the screen. Other image elements, such as standard or common icons (e.g., magnification classes, settings icons, etc.), are distinguishable between applications and are therefore useful for determining a matching score. The matching model used for image template matching can include one or more image artifacts to search for. The image template matching algorithm can scan the input image for any image artifacts and, if an image artifact is detected, can return a label or identifier associated with the matching model associated with the image artifact. Such scanning can be performed by evaluating sections of the captured image 410, such as pixel group 420, to determine whether any of the scanned sections match the image artifact with at least a minimum matching threshold number.

[0067] The image template matching algorithm can obtain template images that match the screen density of the user device 110 (and / or based on some other suitable criteria). For each template image, the image template matching algorithm can compare the template image to the current input image using various difference-based calculations, search for the most likely location of the current input image for the given template, and generate a matching score. The search region can be reduced using a region of interest (ROI) based on the matching model data, template data, captured image data, and / or other relevant factors. If the calculated score meets or exceeds a specified threshold, the image template is considered a match, and an identifier for the matching application can be returned.

[0068] Matching model data associated with image template matching may include a template image, a template mask, a comparison type, and / or various thresholds. A template image may include a list of related images for comparison with an input image. An image may include (or be associated with) information such as screen density (e.g., dots per inch, or "dpi") so that the image can be properly scaled for comparison. Each template image may include an alpha channel that defines areas to be excluded from evaluation. A template mask may include a list of images that define a bitmap mask to be applied to the template image. Such a bitmap mask may indicate which portions of the template image to use for the matching comparison. A comparison type may indicate the type of template matching operation to use. For example, various matrix transpose operations may be utilized, such as SQDIFF, SQDIFF_NORMED, or CCORR. A threshold may include a value, such as a threshold or score, above which the template matching operation must consider a match.

[0069] Image template matching generally requires manual creation of a matching model. For example, an application screen can be evaluated to identify common templates on the screen. A template image can be extracted from the original screen and set as a matching model. In some embodiments, the generation of such a model can be automated by obtaining a distributed source application (e.g., an Android Package Kit (APK) for Android, an iOS Application Archive (IPA) for iOS, etc.) and extracting icon resources from the source application. Images of screens captured from the application (also referred to as "training" images) can then be scanned for the appearance of any of the extracted icons. The scanning can utilize scoring and / or matching algorithms similar to those described above and below. The most commonly identified icons can be automatically added to the matching model.

[0070] The generation of a matching model and its updating are described in more detail below with reference to process 1100.

[0071] 5 illustrates an example of an environment 500 in which one or more embodiments described herein may be implemented. As shown, the environment 500 may include a user device 110 and a connection resource 120. The environment 500 may include various other devices (e.g., cloud-based devices), network communication channels, and / or other resources.

[0072] The user device 110 may be a consumer electronic device capable of executing instructions and / or processing data, such as a smartphone (e.g., iOS, Android, etc.), a smartwatch, smart glasses, other smart devices, a tablet, etc. The user device 110 may at least partially comprise and / or execute a gateway 100. The gateway 100 may capture and manage screen, audio, and / or other content from source applications 545 and may project the content to the connected resources 120. The gateway 100 may at least partially direct the operation of the source applications 545 (e.g., based on UI data received from the connected resources 120). Each source application 545-550 may be a third-party application that the user wants to use with the connected resources 120.

[0073] The connectivity resource 120 may be any electronic system, device, or element communicatively coupled to the user device 110 and capable of utilizing or interacting with an application (or at least some functionality provided thereby) executing on the user device 110. For example, the connectivity resource 120 may be an in-vehicle system (e.g., a head unit, infotainment system, rear-seat display, instrument cluster, smart mirror, etc.), a medical system, a smart TV, a smart speaker, fitness equipment, a smart home device (e.g., an IoT device), smart glasses, a smart appliance, etc. The connectivity resource 120 may execute a dedicated process or application, sometimes referred to as a “client” application, etc. In some embodiments, the connectivity resource 120 may comprise or provide various interfaces or connectivity functionality that enable the user device 110 to send and receive data to and from the connectivity resource 120. Such interfaces or functionality may, in some embodiments, be included as operating system (OS) components or other non-dedicated resources. For example, the connectivity resource 120 may provide an application programming interface or similar functionality that enables video content to be provided to the connectivity resource 120.

[0074] As shown, gateway 100 can include an application capture element 505, an application control element 510, an application manager 515, a projection engine 520, a recognition engine 525, and a repository 530. Gateway 100 can access various user device features, such as an application library 540, which includes various source applications 545-550 executable by user device 110. In this example, source application 550 includes an optional development kit 555. In some cases, gateway 100 can access an OS interface 560, which can provide information such as active application identifiers. Connectivity resource 120 can include a projection client 570 and / or an optional user device interface 575.

[0075] The application capture element 505 can capture the screen or other output of the user device 110 when executing the source application 545. The application capture element 505 can send the captured data to the projection engine 520 or the recognition engine 525 for further processing.

[0076] The application control element 510 can control or interact with the source application 545. The application control element 510 can launch applications provided by the application manager 515, simulate user events, perform screen transitions, cause specific functions in the applications to be performed, and / or interact with the source application 545.

[0077] The application manager 515 can communicate with external resources, such as cloud-based servers, and manage elements related to application discovery and / or operation. The application manager 515 can locally store and / or manage an application catalog 590 (or portions thereof) and can provide application information to other modules. The application manager can implement various application discovery algorithms.

[0078] The projection engine 520 can perform screen projection to the connection resource 120. The projection engine 520 can include or utilize various screen projection engines such as WebLink, CarPlay, Android Auto, SDL, etc.

[0079] The recognition engine 525 may utilize image recognition and / or other identification algorithms to detect the active source application 545 and / or the active screen to project. For more restrictive user device systems, such as iOS, the OS interface 560 may not be available or accessible (e.g., a system API for detecting the active application may not be provided), and the source application 545 may not include a development kit or other resources that may enable such interaction. In such cases, image recognition may be performed by the gateway 100.

[0080] The recognition engine 525 can load available matching models 580 associated with an application catalog 590 from the repository 530. The recognition engine 525 can classify each received image into an application ID (and / or screen ID) and return the classification information to the application manager 515. Such classification can utilize various image analysis algorithms, as described above and below. If the image is not recognized and indicates an unknown application, the recognition engine 525 can report such an unrecognized image status to the application manager 515.

[0081] The recognition engine 525 may execute one or more of the image classification algorithms described herein based on various suitable criteria (e.g., capabilities of the user device 110, capabilities of the connection resource 120, user preferences, application configuration settings, etc.) The matching model 585 may indicate which algorithm(s), parameter values, or other settings, etc., to use.

[0082] The recognition engine 525 may use the hardware-accelerated machine learning (ML) capabilities of the user device 110 whenever possible. For example, in the case of iOS, Core ML may be used or implemented. In the case of Android, the built-in ML Kit may provide such capabilities. Popular third-party libraries, such as the open-source computer vision library (OpenCV), may also be used for image classification and / or analysis tasks.

[0083] As shown, the application catalog 590 can include or be associated with matching models 580 for various supported applications. In some embodiments, general models can be stored within or associated with the application catalog 590 (e.g., using a data-only application record). The recognition engine 525 can load available matching models 580 when the application catalog is initially downloaded (or built) or updated based on interactions with a server or similar resource, such as a gateway manager, described below. The matching models 580 can be processed by generating an empty list of rules (a "rule list"). For each application in the application catalog 590, the recognition engine 525 can determine whether there is a matching model 580 associated with the application entry. Such a determination can be made in various suitable ways, such as by analyzing the application entry (and / or associated data elements) to identify a list of associated matching models 580.

[0084] Once the relevant model is identified, the recognition engine 525 can obtain the root rule object from the model and add the rule to the rule list at the specified position. For example, if the new rule does not have a priority flag or property set, the rule can be added to the end of the list. If the new rule has a priority property or flag set, the recognition engine 525 can search the rule list from the beginning and insert the new rule before the first rule with a lower priority value than the current rule (or the first rule with no priority value). Once all applications have been processed, the rule list can contain a list of all root rule objects from all matching models 580 associated with the application catalog 590, sorted by priority (and the order in which they are listed in the application catalog). The rule list can be processed sequentially during or during the execution of the various recognition or analysis algorithms described herein.

[0085] Once the model is loaded and the rule list is generated, the recognition engine 525 can process the current captured screen and return the detected application ID, screen ID, and / or other appropriate information. In some embodiments, the recognition engine 525 can process all captured screens. However, such an approach can result in a high system resource load on the user device 110. Therefore, in some embodiments, the recognition engine 525 can use various gating functions to determine (or indicate) whether a particular screen image should be processed.

[0086] The gating function may include, for example, time-based gating, where the decision to analyze the current screen is based on a timer. For example, one screen may be analyzed every second. The duration of the gating interval may be adapted based on parameters such as processing load and / or battery consumption on the user device 110. For example, if the processing load is high, the duration of the gating interval may be increased to reduce the processing load.

[0087] Another example of a gating function may be on-change gating, where the decision to analyze the current screen is based on a determined difference between a previously detected screen image and the current screen image. If the determined difference exceeds a set threshold, the current screen image may be analyzed. Such a difference comparison may be performed in a variety of suitable ways, including direct image comparison and / or analysis of the video encoder signal.

[0088] Direct image comparison may include comparison methods such as color bin-based analysis (such as those described with respect to color matching), pixel-by-pixel comparison (e.g., where the number of changed pixels exceeds a threshold number or threshold ratio), perceptual hashing algorithms (e.g., algorithms that use hashes to generate snippets or fingerprints of various forms of media), threshold-based scene detection (e.g., current screen intensity and / or brightness may be compared to a set threshold, and analysis may be triggered if the intensity and / or brightness exceeds the threshold), and / or other suitable direct image comparison algorithms.

[0089] Comparing the video encoder signals can include using the projection engine 520 (and / or an associated video encoder) to perform scene detection based on the captured image being sent to the projection client 570. For example, in the H264 format, various types of frames can be used. Key frames (or "I" frames) can contain the entire scene, while "P" and "B" frames can contain only changes. If there is no change in the scene, the P and B frames will be smaller, and such information can be used by the recognition engine 525 to determine whether to process the current image. In some cases, the video encoder can implement internal scene detection logic (e.g., for better frame encoding) that can provide a separate signal to the recognition engine 525 when a scene change is detected.

[0090] In some embodiments, a hybrid image recognition approach may be utilized, for example, a change in the screen may be detected and screen recognition may be performed at certain intervals regardless of whether the determined difference exceeds a threshold or indicates a change.

[0091] If the recognition engine 525 determines that a screen image should be analyzed, it can process the rule by, for each rule object in the rule list, applying any image operations described in the rule and saving the results. If the rule object has child rules, each child rule is processed recursively and saved or associated with the current screen image. If the rule object has no child rules, the recognition engine 525 can determine the rule type and run the algorithm associated with the rule (and pass any algorithm-specific data).

[0092] If the algorithm indicates that there is a match, the recognition engine 525 can return the associated label of the matching rule, extract the application ID (and screen ID, if any), and send the application ID (and screen ID, if any) to the application manager 515. In some embodiments, the recognition engine 525 can be configured to select a match based on a rule that analyzes the image using all available rules and returns the highest matching confidence score (calculated in various suitable ways).

[0093] If the algorithm indicates there is no match (or there is a match with a different application), the recognition engine 525 can notify the application manager 515. The application manager 515 can be configured to implement a hysteresis function and not consider the current application unknown based on one non-detection (or a limited number of non-detections). Such an approach can help eliminate errors in recognition and avoid poor user experience, such as flickering when a particular screen image is temporarily not recognized or is incorrectly recognized.

[0094] The application capture element 505 allows the active source application screen (and / or other application output) to be captured and received for analysis by the recognition engine 525. When capturing via an external display channel, the captured screen image can be received from the connection resource 120 and passed from the projection engine 520 to the recognition engine 525.

[0095] The recognition engine 525 may be configured with matching models 580 (also referred to as "recognition" models), which may be downloaded from a server along with an application catalog 590 and / or other appropriate information. The recognition engine 525 may evaluate the models and report to the application manager 515 the application ID, the screen ID of the detected application (or "null" if the current application is unknown), and / or other relevant information.

[0096] Repository 530 may comprise memory or storage that may include matching models 580, launcher screens 585, optional application catalog 590, and / or other suitable elements. Each launcher screen 585 may be, contain, or execute an application screen that can be projected onto connection resource 570. Launcher screen 585 may include available applications (represented by application catalog 590) and may enable end users to launch or interact with the available applications. Application catalog 590 may be a document or other data structure managed by a back-end server or other suitable resource and may contain information such as a list of applications that should be accessible to end users via the screen projection system, associated constraints (if any), screen recognition models (or references thereto), etc. Some embodiments may comprise an application management back-end device (not shown) that may manage application catalog 590, matching models 580, and / or other elements used by gateway 100.

[0097] Each matching model 580 may be represented by a file that stores a hierarchical object structure that describes how to classify an image. The hierarchical structure may be constructed using a standard binary or text file format, such as Protocol Buffers, JSON, Extensible Markup Language (XML), or YAML, or may be constructed as a proprietary serialized object structure. Each matching model 580 file may be compressed (e.g., using GZip) and / or encrypted. Such files may be digitally signed so that the recognition engine 525 can verify the authenticity and integrity of each file when loaded.

[0098] One matching model 580 can be associated with one or more applications and can be used to identify the one or more applications. For each application, the matching model 580 can include and / or be used to recognize a screen or state. Some matching models 580 can be used to recognize screen states across multiple applications (e.g., detecting an on-screen keyboard regardless of the active source application).

[0099] Each matching model 580 may include a version and an optional digital signature that may use standard cryptographic APIs provided by the user device's OS. A matching model 580 may include a rule element, which may be the main hierarchical object that describes the algorithm to run and the parameters to use. A rule element may contain one or more child rules.

[0100] A rule element can contain various sub-elements, such as a type, which can define the type of rule. Depending on the type, various other properties can vary. In some embodiments, group rules can be used. A group rule can perform a specific common action based on the input image and then pass the results to various child rules. A final rule can provide instructions for the execution of one of the supported image classification algorithms.

[0101] Each rule may include a name element that may be used for debugging purposes. Each rule may include a priority element that defines the priority of that rule relative to other rules or rule elements. The priority can be used by the recognition engine 525 to determine the order in which to evaluate various rules. If a priority is not specified, the order in the model file and / or the order in which the models are loaded can be used to determine the priority.

[0102] Each rule may contain a child element that contains a list of child rules, if any. Child rules may be evaluated in the order listed unless otherwise associated with a defined priority.

[0103] Each rule can include an image operation element that can specify an operation to be performed on an input image before evaluating the image with respect to the rule and / or the associated evaluation algorithm. Group rules can be associated with specific image operations that can be applied to all child rules to speed up processing. Various image operations can be supported, such as scaling (e.g., an image can be reduced or enlarged to a predetermined pixel size or by a relative percentage), cropping (e.g., this operation can be used to crop system bars), color conversion (e.g., converting to grayscale, converting to another color space, removing bits from color values, etc.), filtering (e.g., using operations such as dilation, erosion, etc.), and / or other suitable operations.

[0104] Each rule may include a region of interest (ROI) element that may specify one or more regions of interest (ROIs) for the input image that may be used by the recognition algorithm. The ROIs may be specified by rectangles, polygons, masks, and / or other suitable methods.

[0105] Each rule can include a label. For rules that define a classification operation, the label field can include the result that the recognition engine 525 should return if the rule is matched. Additionally, the algorithm-specific data can include label data. The label can be expressed as a string that includes information such as the application ID and / or screen ID.

[0106] Depending on the type of rule, each rule may contain algorithm-specific data elements that may store parameters that are passed to the corresponding image classification algorithm.

[0107] A development kit 555 (also referred to as a "notification" software development kit (SDK)) can enable notification messages between the gateway 100 and the source application 545. A source application that integrates with the development kit 555 can provide an improved user experience.

[0108] The projection client 570 can receive and play the projection information stream sent by the projection engine 520. The projection client 570 can display the results on an appropriate display or screen of the connected resource 120. The projection client 570 can capture any user input (button, touch, voice, etc.) and send the captured user input to the gateway 100.

[0109] The user device interface 575 can enable control of the user device 110 outside of the communication path from the projection engine 520 to the projection client 570. For example, the user device interface 575 can comprise or use a human input device (HID), and the application control element 510 can instruct the user device interface 575 to send HID commands to the user device 110.

[0110] The user device 110 and the connectivity resource 120 (and / or other system or environmental elements described herein) can communicate via various paths or channels. For example, the main data channel between the user device 110 and the connectivity resource 120 can utilize a connection method such as Wi-Fi, Bluetooth, Universal Serial Bus (USB), etc., and can use a communication protocol supported by typical user devices 110, such as Android Open Accessory (AOA), External Accessory Protocol (EAP), Transport Control Protocol over Internet Protocol (TCP / IP), etc. Some embodiments can include communication channels such as an external control channel, an external display channel, an audio channel, etc.

[0111] The external control channel can include, for example, HID input, location device input, etc. transmitted via USB or Bluetooth. The external display channel can utilize or provide projection utilities such as AirPlay, MiraCast, HDMI, MHL, ChromeCast, etc., and data can be transmitted via wireless or wired channels. The audio channel can include or provide audio input and / or audio output data via channels such as Bluetooth Advanced Audio Distribution Profile (AADP or "A2DP"), USB Audio, USB Microphone, AirPlay, WiSA (Wireless Speaker and Audio), Digital Living Network Alliance (DLNA), High-Definition Multimedia Interface (HDMI), etc.

[0112] The user device 110 and the connection resource 120 may communicate over the main data channel (and / or other available channels) as described above using the Data Channel Communications Protocol (DCCP) of some embodiments. Messages sent between the gateway 110 and the projection client 570 (and / or other system elements) may utilize or implement such DCCP. Messages may be stored and transmitted using binary, text, and / or other suitable representations. For example, the DCCP of some embodiments may utilize serialized "C" structures, string formats such as JavaScript Object Notation (JSON), protocol buffers, etc. The use of protocol buffers may be preferred because binary formats are scalable and efficient in terms of bandwidth and processing.

[0113] Although various messages or other elements of the DCCP may be described by reference to particular sets, those skilled in the art will recognize that similar, analogous, and / or complementary messages or elements may be implemented using various other sets.

[0114] Regardless of the underlying message format, the messages (and / or associated protocols) may be extensible so that new messages are added to the communication protocol. Older versions of the gateway 100 or projection client 570 may ignore such new messages. The messages and / or protocols may be backward compatible so that new components can interact with older components by sending messages that are compatible with such older or outdated components. The messages and / or protocols may be forward compatible so that, for example, new fields added to a message are ignored by older components and only recognized fields are utilized.

[0115] Some such messages follow a request and response format or paradigm, such that a request sent by one element elicits a response from another element. Such messages may share a common request identifier or other suitable attribute that allows a request to be associated with a corresponding response. Each of the messages described above and below may be divided into multiple smaller messages. Similarly, each message may be combined with other messages into larger messages.

[0116] In some cases, multiple data paths may be available between the user device 110 and the connectivity resource 120. For example, a Bluetooth connection may be used first, followed by the establishment of a Wi-Fi connection and / or an additional physical connection such as via a USB cable. DCCP can use session information to support such multi-channel configurations.

[0117] For example, DCCP may include a "session" message that is used to establish an initial connection over a first channel (e.g., Bluetooth). This session message allows an endpoint to be able to establish successive sessions over one or more connection channels. The message may be used to determine whether a newly established connection (or "channel") is to the same device as an existing or last known connection. Such an approach allows peers to perform seamless switching between connection channels. The session message may be used to establish a new session and disconnect all other communication channels for mismatched devices.

[0118] Each session message may include a session token for the component sending the message and various session handshake flags. Such session handshake flags may include an announce flag indicating whether the message is an announcement (meaning a new session is not required), a new session flag indicating whether the message requests the establishment of a new session, a reply flag indicating whether the message is a request or reply message, and / or other suitable flags. Session request messages may be sent by the projection client 570, and session reply messages may be sent by the gateway 100.

[0119] As part of the initial setup, session tokens may be exchanged and stored by the gateway 100 and / or the projection client 570. Once the second channel is established, the gateway 100 and the projection client 570 may attempt to continue the existing session. If successful, the second channel may be used in parallel with or in place of the first channel.

[0120] Such an approach allows for seamless transitions between channels and can accommodate changes in connectivity (e.g., due to user actions or selections, determinations based on signal strength or other metrics, etc.). Parallel channels can be used for different purposes. For example, a low-throughput Bluetooth channel can be configured to use a low bit rate associated with a low-quality video streaming format. When a high-throughput channel becomes available, the video streaming can switch to the high-throughput channel and its associated high-quality data video streaming format.

[0121] To achieve such multi-channel session-based communication, the gateway 100 and the projection client 570 may utilize multiple sets of session tokens. Such session tokens may include client session tokens generated by the projection client 570 and sent to the gateway 100, and gateway session tokens generated by the gateway 100 and sent to the projection client 570.

[0122] The session token may be generated using a random number generator and attributes (e.g., date, time, etc.), connection channel identification information (e.g., IP address, Media Access Control (MAC) address, etc.), and / or other suitable information.

[0123] The session token may be exchanged in a variety of suitable ways. For example, when a projection client 570 connects to the gateway 100, it may send a session message that includes a previously stored (if available) or newly generated client session token, a raised announcement flag, and / or other suitable content.

[0124] The gateway 100 may receive a session message from the projection client 570 and compare the received client session token with any other active or previously received client session token. If the received client session token matches an active or previously received client session token, the gateway 100 may send a session response message that includes the gateway session token, the generated announcement, and / or a response flag, and / or other appropriate content. The gateway 100 may generate and store a gateway session token if the corresponding stored session token is empty.

[0125] If the received client session token does not match an active session token and thus indicates a new connection, gateway 100 can determine whether there was a previously active connection (e.g., whether a previously received session token matches the received client session token). If a previously active or currently active connection is identified, gateway 100 can terminate the previous connection and switch to the newly available connection, and such switching can occur based on some specified criteria (e.g., gateway configuration, user confirmation, etc.). If the newly available connection is rejected (e.g., if the user selects the option to continue with the previous connection), gateway 100 can send a new session message with an empty session token, a generated response flag (e.g., indicating that the new connection was rejected), and / or other content. If the newly available connection is accepted, gateway 100 can clear any stored active client session tokens, generate a new gateway session token, and send a session response message with the new gateway session token, the generated new session and response flags, and / or other content.

[0126] The projection client 570 can receive the session response message. If the new session flag is set, the projection client 570 can disconnect all other channels or devices until only the current channel is open, save the received session token as a new gateway session token, clear any stored client session tokens, generate and save a new client session token, and, if the response flag is not set, send a new session message with the new session flag set and pass the newly generated session token. If the new session flag is not set and the announce flag is set, the projection client 570 can check the received gateway session token and compare it with the stored gateway session token. If the tokens do not match, the projection client 570 can proceed as if the new session flag was set. If the tokens match, no additional action may be taken because the new channel is for an existing device.

[0127] If gateway 100 receives a session message with the new session flag set and there was a previously active connection, depending on the configuration of gateway 100, gateway 100 can switch to the new connection or ask for confirmation to terminate the previous connection. If the new connection is rejected, gateway 100 can send a new session message with an empty session token and the reply flag set to indicate that the new connection was rejected. If the new connection is accepted, gateway 100 can close any existing connection, save the received token as a client session token, generate and save a new gateway session token, and send the new token via a session message with the new session and reply flag set.

[0128] Other messages are ignored unless the gateway 100 and / or connecting device 120 establishes the session by completing one or more message exchanges as described above.

[0129] As another example, DCCP may include an "identity" message. Such a message may be sent by any element to request the identity of another element. Some of the information sent via the identity message may be sensitive and need to be sent over a secure channel. The identity message may be initiated by anyone at any time once a connection between peers has been established (e.g., by exchanging session tokens, as described above). One side may request the identity of the other by sending an empty identity message or a special parameter or flag indicating that an identity is being requested. The recipient of the identity request may respond with the same identity message type with the identity value entered. The identity request may specify a minimum identity value that must be returned. If the peer device does not respond within a specified time limit (e.g., 5 seconds) from the request, or if the identity information is incomplete or does not match, the requester may terminate the connection.

[0130] The identity message can include an identity value list containing identity IDs and corresponding values. Examples of identity IDs include a system ID, display name, manufacturer, model (any string that describes the device), country code, serial number, OS information, application information, and / or other identifiers. The system ID can be a single string that uniquely identifies the peer device. If the same device is already connected (running the same software), the device will always return the same system ID. The system ID typically includes or can be generated using the model, manufacturer, obfuscated serial number, and / or a random number generated by the device. The display name can be a name of the peer device that can be presented to the user. The string may be in English. In some cases, the device can send the display name in another language via an additional identity ID. The country code can be a string containing one or more comma-separated ISO 3166-1 alpha-2 codes, each representing the country in which the peer device is configured. The serial number can be a unique string that identifies the device. If the message is passed over an insecure connection, the serial number may be a pseudo serial number (e.g., a device-generated or obfuscated serial number). OS information may include the name and version of the operating system. Application information may include information about the main application running on the device, such as the application's name, vendor, and version. Other identifying values ​​may be used as needed.

[0131] Another example of a DCCP message is a "ping" message. Such a message may allow an endpoint to determine whether a peer application is responding. Ping messages can be used to determine whether a connection is still active for transports that do not properly report when the connection is lost. Ping messages may also be sent at regular intervals by the projection client 570 to maintain a connection while an application is running in the background.

[0132] A ping message may include a request ID that identifies the message, in which case the reply ping message must include a matching request ID. A ping message may include various flags, such as a request flag (for request messages) that indicates whether a reply is required, a reply flag that indicates the message is a reply to a previously sent request, or a one-way flag that indicates no reply is required or expected.

[0133] DCCP can include a "sync time" message that is used to synchronize the gateway 100 and the sync client 570. The sync time message can be used to calculate the offset between the local clock and the clock of a remote peer. Various clock synchronization methods can be used. The simplest algorithm is to have a peer send a sync time message with its latest timestamp. The remote peer can respond with its latest timestamp. The sending peer can then calculate the offset by taking the difference between the two timestamps and taking into account the latency of the communication link.

[0134] The sync time message may include a request ID to identify the message, in which case the reply sync time message must include the same ID. The sync time message may include a client timestamp (e.g., a timestamp in milliseconds or microseconds indicating when the message was sent from the client) and a gateway timestamp (e.g., a timestamp in milliseconds or microseconds indicating when the message was sent from the gateway host).

[0135] Another example of a DCCP message is the "connected device status" message used by the projection client 570 to report status and any state changes to the gateway 100. For example, if the connected resource 120 is an in-vehicle device, the connected resource 120 can report the latest driving conditions to reduce driver distraction. The message can be sent by the client 570 without being requested.

[0136] The connected device status message may include a constraint level indicating the current constraint level reported by the device, a display status indicating whether the connected device is displaying any content received from gateway 100, and / or any other suitable status fields or other content.

[0137] 6 illustrates another example environment 600 in which one or more embodiments described herein may be implemented. As shown, environment 600 may include the user device 110 described above and a gateway manager 610 with a catalog interface 620, a repository 630, a control interface 640, and a model trainer 650. Environment 600 may further include a user device 660.

[0138] The gateway manager 610 may comprise one or more electronic devices, such as a server or servers, and may be accessible via a public cloud, private infrastructure, and / or other suitable resources.

[0139] The catalog interface 620 may include an API or other suitable resource that may enable the gateway 100 to request the application catalog 590 and / or other suitable information. The API may utilize a Representational State Transfer (REST) ​​architecture using Hypertext Transfer Protocol (HTTP) requests or other web services APIs such as Google Remote Procedure Call (gRPC), Simple Object Access Protocol (SOAP), Java Remote Method Invocation (RMI), etc. The API may be executed over a secure connection, such as Transport Layer Security (TLS). The API may support authentication (e.g., how the gateway 100 authenticates with the gateway manager 610) and application catalog requests (returning responses matching the connection resource ID).

[0140] The particular application catalog 590 returned may depend on the connectivity resource 120. A user device 110 may connect to different connectivity resources 120 with different capabilities. Similarly, different applications may be available for different geographic regions, or business considerations may dictate which applications should be included or enabled. For example, a provider of a connectivity resource 120 may license a set of applications that are enabled by default. Device information included in the identity message sent by the connectivity resource 120 can be used by the gateway 100 when requesting or providing the application catalog 590. The application catalog 590 can be extracted or constructed at run time using data received from the repository 630.

[0141] The repository 630 may comprise a database management system that stores various application catalog records that allow the catalog interface 620 to quickly extract or build the desired application catalog 590. The database used by the repository 630 may be a relational database (e.g., Structured Query Language (SQL) Server, MySQL, Postgres, etc.) or a non-relational database (e.g., DynamoDB, MongoDB, SimpleDB, etc.).

[0142] The application catalog 590 may be a data structure that describes the source applications to be managed by the gateway 100. The application catalog may be provided in a variety of formats. For example, upon request from the gateway manager 610 via the catalog interface 620, the catalog may be returned in a portable format such as JSON, protobufs, etc. When processed by the gateway 100, the application catalog 590 may be represented as a hierarchy of object references. Regardless of the format for exchange, storage, or processing, the application catalog 590 may include various attributes or parameters. Example parameters may include a version and a set of connection resource identifiers to which the application catalog 590 applies.

[0143] The application catalog 590 may include an application list 680 of application records. Such records may be associated with various fields, including example fields described below. The application ID field may include a unique identifier for the application. The ID may be any string, although the use of a reversed Uniform Resource Locator (URL) notation, such as "com.yelp," is recommended to avoid name collisions. The projection type field may describe the type of projection supported by the application (e.g., captured application, SDK-integrated projection application, web application, etc.). Some embodiments may support a special type of application record designated "data only." Such a record may be used to store common information, such as matching models for system screens such as keyboards, and may not be associated with a specific application.

[0144] The application category field may indicate the type of application (e.g., the primary function of the application). Categories may include multiple levels (e.g., primary, secondary, etc.). Categories may be represented by a string with subcategories separated by slashes (similar to a directory path), with the more general category first, followed by the more specific category. If an application supports multiple categories, commas may be used to separate the categories. For example, for a navigation application, the category may be "Mapping / Navigation," for a video player application, the category may be "Media / Video," for a music player, "Media / Music," and for an application that supports video and music, it may be represented as "Media / Music,Video."

[0145] The install location field may provide a location (e.g., a URL) where the associated application can be installed. The install location field for different supported user device platforms may contain different subfields. For example, for iOS, the install location field may point to the location of the application on the App Store.

[0146] The display information field may contain information about how the application should be presented to the end user. The display information field may include elements such as a title (localized for supported languages), an icon (localized for various supported resolutions, for example), and a description (localized for supported languages). The display information field may include display options such as whether the application icon should be initially displayed on the launcher screen, whether the application should be displayed or hidden on the gateway 100, etc.

[0147] The launch information field may contain information about how the gateway 100 should launch the associated application. Supported user device platforms may have different subfields. For example, for iOS, the launch information field may contain a URL scheme for launching the application, and for Android, the launch information field may contain a launch intent or package name.

[0148] The authentication information field may contain information that allows the gateway 100 to verify and authenticate the correct application installed on the user device 110. There may be different authentication records for different supported platforms as well as different versions of the application. There may be different ways for the gateway 100 to obtain the signature or certificate of an application. For example, in Android, such a resource may be obtained using the PackageManager.getPackageInfo API and the GET_SIGNING_CERTIFICATES flag. As another example, an application may be verified via a checksum; the gateway 100 may calculate a checksum of a predefined binary of the target application and compare this checksum with the value from the authentication information field. In some user device systems, the gateway 100 may have limited access to third-party application information; in that case, the record in the authentication information field for that platform may be empty, and the gateway 100 will know that a given application cannot be authenticated.

[0149] The recognition model field 685 allows for the storage of (and / or reference to) application and screen recognition models for various platforms and supported application versions.

[0150] The application purchase information field can store information for enabling applications and / or features via in-application purchases. If an existing in-app purchase framework is used, such as StoreKit or Google Play billing, the application purchase information field can include a product ID and allow pricing information to be managed through a corresponding backend system. In some embodiments, the application purchase information field can store the product price (e.g., for various supported regions) and any information required to complete the purchase.

[0151] The application mode constraints field 690 may be an optional record that describes whether any constraints should be applied to the projection of the current application depending on the operational state of the connection resource 120 .

[0152] The signature field can be a digital signature of the application catalog that can be used to verify the integrity of the catalog information. The file containing the application catalog can be signed with a private key securely stored in the gateway manager 610 and verified by the gateway 100 via the public key. The signature field allows the catalog to be saved on the user device 110, and when reading the catalog later, the gateway 100 can verify that the catalog has not been altered. If the catalog has been altered, the gateway 100 can reject the catalog and request a new one from the gateway manager 610.

[0153] Application catalog 590 may include various other fields, elements, attributes, parameters, or other data as desired.

[0154] The control interface 640 may include or provide an application management API or other suitable resource that enables application catalog records to be created, modified, deleted, and / or otherwise managed. Such application management API may utilize a REST architecture with HTTP requests or other web services APIs, such as gRPC, SOAP, Java RMI, etc. The application management API may run over a secure connection, such as TLS. The application management API may enable authentication, management of connected devices, creation or deletion of application catalogs, management of application catalog entries, and / or other interactions with the application catalog 590.

[0155] The user device 660 may be a device such as the user device 110 that is associated with an administrator rather than an end user. The user device 660 may include or run a web portal (via a browser) or management application 670 that provides access to the control interface 640. The management application or web portal 670 may be restricted to access via a virtual private network (VPN) for added security.

[0156] User devices 660 and / or management applications or browsers 670 can authenticate with the gateway manager 610 and be granted various access rights. Based on the authentication, some features of the control interface 640 can be enabled or disabled.

[0157] The management application 670 may manage connection resource IDs and allow connection resource IDs to be created, deleted, modified, etc. The management application 670 may create and / or delete application catalogs and / or manage application catalog entries by adding, deleting, and / or modifying information associated with the application catalog entries.

[0158] The model trainer 650 can generate and / or train the recognition model 665 and / or other models used or executed by the gateway manager 610 or the gateway 100. Model training is described in more detail below with reference to process 1100.

[0159] 7 shows an example process 700 for projecting application content from a user device 110 to a connected resource 120. Process 700 may enable a user to extend or enhance the functionality of the connected resource 120 using the connectivity and / or processing power provided by the user device 110. This process may be performed when the source application is launched, at regular intervals, and / or based on some execution criteria. In some embodiments, process 700 may be performed by gateway 100. Components such as projection client 570, catalog interface 620, and / or control interface 640 may perform processes complementary to process 700.

[0160] As shown, process 700 may include generating (710) an application catalog, such as application catalog 590, or similar. The application catalog may be received by gateway 100 via catalog interface 620 from a resource, such as gateway manager 610. Gateway 100 may build the application catalog based on data received from a resource, such as gateway manager 610, via catalog interface 620. If available, gateway 100 may retrieve a previously generated or received application catalog from a resource, such as repository 530. Any newly received or newly generated application catalog may be stored in gateway 100 (e.g., using repository 530) for future use.

[0161] The projection client 570 can send an “application catalog” request message to request the latest application catalog or to register for an application catalog update. The application catalog request message can include elements such as a request ID. The gateway 100 can return the same request ID along with the application catalog response message so that the projection client 570 can match the response with the request. If the gateway 100 sends an update message, the request ID can be set to zero. The application catalog request message can include various application catalog flags that indicate how the application catalog should be handled. The projection client 570 can set the requested flags. The gateway 100 can return the latest gateway flag value. The application catalog flags can include an “auto update” flag that indicates whether the host should automatically send an “application catalog” message every time there is a change in the application catalog. The auto update flag, when sent by the projection client 570, can request that the gateway 100 perform an automatic update. To stop the automatic update, the projection client 570 can set an “auto update stop” flag. When sent by the gateway 100, the auto update stop flag can include the latest auto update status of the gateway 100. The application catalog flags may include an "all catalog" flag that indicates whether the gateway 100 should return the entire application catalog. The all catalog flag may be returned by the gateway 100. Upon receiving the all catalog flag, the projection client 570 must update the entire internal catalog (adding and removing applications). If the all catalog flag is set by the projection client 570, this flag may be ignored.

[0162] The application catalog request message may include one or more application records. Each application record may include a property ID and an associated value pair. When requesting an application catalog, the projection client 570 may enter the IDs of the desired properties and possible request and / or filter values ​​in one application record. Table 1 below provides example property IDs and associated values. In response to the application catalog request message, the gateway 100 may return a list of application records with the requested properties and associated values ​​for each application.

[0163] As mentioned above, application properties can be specified using predefined name / value pairs included in application catalog messages (and / or other application catalog resources). Names can be predefined and assigned values ​​as shown in Table 1 below. Table 1 JPEG2025186218000002.jpg220166JPEG2025186218000003.jpg219166

[0164] The application catalog can be provided to other resources, such as the projection client 570, via application catalog messages. Such messages may be originated by the projection client 570 and received by the gateway 100. The application catalog messages can follow a request and response format. In some cases, the gateway 100 can send automatic updates to the projection client 570. Various application catalog message flags can be used to define or modify messaging procedures (e.g., an auto-update flag or an auto-update stop flag can be set to manage auto-update messages).

[0165] The projection client 570 can request data related to one or more applications (up to the entire application catalog) from the application catalog associated with the gateway 100 in an application catalog message. If available to the projection client 570, the application can be requested using the application ID in the application catalog request message. If no application ID is specified, all available applications are returned and the all catalog flag can be set. The projection client 570 can indicate one or more specific application properties to be returned in the response to the application catalog request message.

[0166] The projection client 570 can request specific application attributes (e.g., image width, image height, display name language, etc.) in the application catalog message. If no application attributes are specified, the gateway 100 can return all available application properties.

[0167] Application catalog messages may typically follow a request and response format. However, the projection client 570 may request to be notified whenever there is a change in the application catalog. For example, an application status may change or a new application may be added to the application catalog. Such notification may be supported via an auto-update flag and an auto-update stop flag associated with the application catalog message. If the auto-update flag is set in the application catalog request message from the projection client 570, the gateway 100 may send an application catalog message whenever there is a change in the application catalog. As part of such a request message, the projection client 570 may specify application properties to be included in each update by populating an application record. If no application record is populated, all properties may be returned for each update. The projection client 570 may stop automatic updates by sending an application catalog message with the auto-update stop flag set.

[0168] Process 700 may include identifying and authenticating (720) the source application. Such identification and authentication may be performed in a variety of ways, depending on which resource launches the application and / or other relevant information. For example, an application may be launched on user device 110 and a request to project the application may be received at gateway 100. As another example, projection client 570 may send a request to launch the application.

[0169] Third-party source applications can be launched through the gateway 100. The launch request can be received through a resource such as a launcher screen or a message received from the connection resource 120. Launchable source applications can be stored in an application catalog, and launch information can be provided in a launch information record. Application launch can be performed using an API or other resources provided by the user device 110 and / or other system elements. For example, in iOS, the gateway 100 can launch an application using a URL schema. In Android, the gateway 100 can launch an application using a launch intent with the desired application package name. Application launch can be performed by a resource such as the application control module 510.

[0170] In some situations, it may be useful for the projection client 570 to be aware of available and active applications. For example, the projection client 570 may select a particular application to launch. Such awareness may enable a user experience as if the projected application were originally running on the connection resource 120.

[0171] For example, a connectivity resource 120 associated with an automobile system may include a hardware button on an in-vehicle head unit that launches a navigation application. A projection client 570 running on the connectivity resource 120 may query the gateway 100 for available applications from the application catalog 590 and request that the gateway 100 launch the navigation application on the user device 110.

[0172] The projection client 570 can receive a list of available applications and their status using the application catalog message, as described above. The projection client 570 can use the list information to display a set of application icons or notify the user about the applications available through the gateway 100.

[0173] The DCCP may include a “Current Application” message that the gateway 100 can use to notify the projection client 570 whenever the active application changes. The active application may change based on a user switch or a switch directed by the projection client 570. The projection client 570 may use the Current Application message to request the launch of a specific application. The Current Application message may include an Application ID element that uniquely identifies the application. The application ID may be stored in the application catalog 590. The Current Application message may include an Application Status element that defines the status of the application (e.g., “Launch”: requests launch of the application; “Activate”: indicates that the application has been activated; “Launch”: indicates that the system is in the process of launching the specified application; and / or “Launch Failed”: indicates that the requested application failed to launch). If the launch is successful, an “Activate” status may be returned in the response to the Current Application message. The Current Application message may include an Application Screen ID element that identifies the ID of the active screen of the current source application (if found).

[0174] The projection client 570 can use a current application message to request the launch of a given application supported by the application catalog. This message is processed by the gateway 100 and forwarded to a resource such as the application manager 515, which can resolve the application ID to an entry in the application catalog. The application manager 515 can provide the application entry information to the application control module 510, which can attempt to launch (or switch to) the requested application on the user device 110. If the requested application is not installed, the gateway 100 can present an appropriate message and / or instruct the user to install the application. The gateway 100 can return a status flag via the current application response message.

[0175] Whenever the currently projected active application changes, the gateway 100 can send a Current Application message with the appropriate status. If supported, the gateway 100 can also send notifications whenever there is a change in the active mode and / or screen. The projection client 570 can use the notification messages to update the status of UI elements associated with the active application.

[0176] When an application is launched, the gateway 100 may need to restrict the applications that are projected onto the connected resources 120. For example, in the case of an in-vehicle system, there may be driving distraction regulations that prohibit certain applications from being displayed on the in-vehicle display while the vehicle is moving. Therefore, the gateway 100 needs to be able to detect the identity of the active application being projected onto the connected device. Depending on the capabilities of the user device 110, the access provided to the gateway 100, and / or other relevant factors, various detection algorithms (or a combination thereof) can be used to identify the active application.

[0177] If the user device 110 provides a system API, such as OS interface 560, or another such resource to the gateway 100, it can report active applications through that interface. The gateway 100 can use an application catalog to match the information reported by the system API with the application ID associated with an entry in the application catalog. For Android, the accessibility services API can be used to discover the package name of the current application. The gateway 100 can then use the application catalog to match the package name with a supported application entry from the catalog.

[0178] Because some user device operating systems may not provide an API or other such resources for detecting active applications, a lightweight notification SDK, such as development kit 555, can be integrated with the source application to provide notifications to gateway 100. Notification development kit 555 can communicate with application control module 510. As a result, application control module 510 can notify application manager 515 whenever a captured application changes state (e.g., moves to the background or foreground, is started, closed, etc.).

[0179] The notification development kit 555 can communicate with the application control module 510 in a variety of ways, such as HTTP messages sent over a socket connection, binary or simple JSON socket messages, inter-process messaging APIs available on the user device 110, and / or other suitable methods.

[0180] The notification development kit 555 can support messages such as authentication messages. A set of such messages can be used to authenticate a source application (e.g., source application 550) to the gateway 100, and vice versa. The two parties can exchange public certificates, which can be signed by the gateway manager 610. Peers can use system APIs (if available) to verify package signatures. Once the two parties are authenticated, the peers can exchange session keys, which can be used for additional messaging or other communications.

[0181] The notification development kit 555 can support messages such as application status change messages. Such messages can be sent by the source application whenever an application status changes (e.g., moves to the foreground, moves to the background, is opened, is closed, etc.). Similar messages can be sent by the application control module 510 when projection is started and / or stopped for the current source application. Such messages can be used by the source application to update the UI associated with the source application.

[0182] The notification development kit 555 can support messages such as screen status change messages. A set of such messages can be sent by the source application whenever the current screen or status changes. For example, a keyboard may appear when a user activates an edit field. Screen status change messages can be used to handle certain restrictions based on application mode.

[0183] Process 700 may include receiving (730) source application information. Such information may be received from a resource, such as repository 530, based on the identified application ID and / or screen ID. Such source information may include, for example, authorization privileges associated with the source application, user information (e.g., username and password), configuration information (e.g., user preferences or settings, manufacturer default values, etc.), and / or other suitable information (e.g., output data format, list of UI features, etc.).

[0184] As shown, process 700 may include receiving (740) connectivity resource information. In some embodiments, such information may be requested and received from connectivity resource 120. The connectivity resource information may be extracted from application catalog 590 and / or related data elements.

[0185] In some embodiments, sensor data (and / or other types of data) may be received from (or through) connectivity resources 120. Such sensor data may include, for example, data received and / or acquired from a GPS component, an accelerometer, a gyroscope, a microphone, a temperature sensor, or other environmental sensors. Such data may be sent over any available channel between gateway 100 and connectivity resources 120.

[0186] Such sensor (and / or other) data can be configured in a variety of suitable ways. For example, the projection client 570 can send an “input capabilities” message to the gateway 100 indicating any supported sensor or other data sources. The gateway 100 can configure the desired input device by sending an “input settings” message, which may specify the device type, protocol, and / or other relevant information. The projection client 570 can respond with an input settings response message, which may include or indicate the device ID for the accepted or applied input data source. The gateway 100 can request streaming of input data from a given input device by sending a “control input” message, which indicates the ID of the desired device and the frequency of data sampling and / or data transmission. The gateway 100 can receive data from the projection client 570 (and / or other components of the connectivity resource 120) and provide (or introduce) the data to the user device OS or source application, as appropriate.

[0187] Sensor (and / or other) data may be sent via an external data input or other connection, if available. Many user devices 110 and / or connectivity resources 120 are capable of supporting the transfer of external sensor data via standard interfaces. For example, iOS provides GPS data via the iAP accessory protocol. Android allows location determination (e.g., via Bluetooth or a serial data bus) by standard National Marine Electronics Association (NMEA)-compatible devices (if mock location is enabled). A variety of other external sensor data inputs can be supported by the gateway 100.

[0188] The projection client 570 can send such data over the main data channel using a "send input event" message (and / or other appropriate messages). Sensor (and / or other) data can be collected by the projection client 570 and sent to the gateway 100 via a set of "send input event" messages. Multiple data sources may send data simultaneously. Multiple data records (from the same or different sources) can be combined into one message. The gateway 100 can introduce or provide the received data to the source application 545 or the OS interface 560. Depending on the type of data, the gateway 100 can simulate or emulate the data as input via the OS interface 560 (e.g., using various system APIs). For example, for Android, GPS location information can be simulated via a "mock location provider" with the "ACCESS_MOCK_location" permission. For other input data types, the gateway 100 can call various APIs of the source application 545, if available.

[0189] Process 700 may include projecting (750) the source application to the connected resource 120. The gateway 100 may efficiently and with low latency capture the current screen of the active source application 545 on the connected user device 110. Various approaches may be used to capture the screen (and / or other output data, including content and / or commands) and project the captured data to the connected resource 120 via the projection client 570 and / or other suitable resources.

[0190] Various display technologies, such as screen mirroring, can be used. Some user devices 110 and associated operating systems may support such external display capabilities. For example, AirPlay, MiraCast, ChromeCast, High-Definition Multimedia Interface (HDMI), Mobile High-Definition Link (MHL), and / or other suitable capabilities may be used to project a screen image. In this mode, the gateway 100 may not capture the active screen. Instead, the projection client 570 may use one of the external display modes supported by the user device 110 and receive the current screen image, for example, via an external display channel of the user device interface 575. The projection client 570 may then display the received content on the connected resource 120.

[0191] Such a "standard" display mode may not provide the functionality necessary for additional processing of captured screen data or other content. For example, application identification or functionality limitations may require that captured data be provided to various components of gateway 100, such as recognition engine 525 or application control module 510. Thus, in some embodiments, projection client 570 can send content, such as images, received via external channels to projection engine 520 and / or other components for analysis. In this way, a more powerful and capable user device 110 can perform any additional processing, allowing connection resources 120 to operate with simpler requirements and reduced hardware capabilities.

[0192] Such an "enhanced" standard display mode may be initiated by sending a "display capabilities" message from the projection client 570 to the gateway 100 after establishing a connection between the gateway 100 and the projection client 570. In some embodiments, such display capabilities information may be obtained (and / or otherwise obtained or determined) from a resource such as the repository 530.

[0193] The gateway 100 can send a "display request" message to the projection client 570 requesting the use of a supported external display channel. The projection client 570 can attempt to open the external display channel to the user device 110 (e.g., via the user device interface 575) and can respond with a "display response" message indicating the connection status.

[0194] The projection client 570 can send a "display request" message to the gateway 100 requesting that the gateway 100 assist in initiating outside display mode on the user device 110. In some cases, the user device 110 may provide an API (and / or other appropriate resources) for initiating outside display mode. For example, Chromecast can be initiated via the "MediaRouteSelector" function. For devices that do not provide such resources, the gateway 100 can provide instructions to the user and / or launch a system settings menu to guide the user through the setup and connection steps. The gateway 100 can respond with a "display response" message.

[0195] When the projection client 570 establishes a connection to an external display, the projection client 570 can notify the projection engine 520 via an "external display status change" message. The projection engine 520 can send a request to the projection client 570 to send a screen image, for example, via a "capture display request" message. The projection engine 520 can request via the "capture display request" message to pause or resume streaming of the captured image and / or to direct streaming of data over an external channel.

[0196] Projection engine 520 may append to (or otherwise combine with) the stream of video (and / or other data) coming over the external display channel an additional overlay video stream or image to be displayed by projection client 570. The additional video or image information may be associated with, for example, position, size, and transparency information that dictates how and where projection client 570 should display the overlay.

[0197] In some embodiments, such overlay information can be used for application or content blocking. The gateway 100 can perform application blocking of content sent over the external display channel. Such application blocking can include a determination (e.g., based on active application identities and any associated rules from the application catalog) in which the gateway 100 can request (e.g., via the projection engine 520) that screen projection be stopped or resumed. Based on the particular external display technology used, the projection client 570 can stop projecting from the user device 110 or stop displaying received data (but without interrupting streaming from the user device to prevent user interaction when projection is later resumed). The projection client 570 can notify the gateway 100 of a change in projection status via an “external display status changing” message. Similarly, the projection engine 520 can stream an overlay image to the projection client 570 to limit certain features (or content) of the projection application or completely replace streaming with information from the user device 110 via “display area request” and “display area data” messages.

[0198] In some embodiments, a screen from an active source application can be captured using functionality of the user device OS, and blocking of the application or content can be performed from the gateway 100.

[0199] For example, an iPhone may include a "ReplayKit" feature. The gateway 100 may register a screen transmission extension that can capture the phone screen (if allowed by the user). The application capture component 505 may receive the captured screen image and send it to the projection engine 520, which may encode and send it to the projection client 570. The projection engine 520 may send the captured image to the recognition engine 525 and / or the application control module 510 for application and / or screen recognition, function block, and control. The gateway 100 may run in the background to capture the screen and process, encode, and send the captured screen to the projection client 570.

[0200] For Android phones, the phone's screen can be captured using the "MediaProjection" API or similar resource. The gateway 100 can request the necessary permission from the end user and capture the phone's screen in the background. The application capture component 505 can receive the captured screen image and send the captured image to the projection engine 520, which can encode the captured image and send it to the projection client 570. The projection engine 520 can send the captured image to the recognition engine 525 and / or application control module 510 to perform application and / or screen recognition, function blocks, and control.

[0201] For greater flexibility and future expandability, the gateway 100 can use an adaptable logical model to represent the screen projection area (i.e., the "projection" model), which can enable multiple displays or multiple areas within a display.

[0202] The projection model may include a "display" element. Such an element may represent an individual screen controlled by the source application 545. The connectivity resource 120 may have multiple physically separated displays. For example, in a vehicle, there may be multiple displays such as a center display (e.g., head unit), an instrument cluster display, a head-up display, a rear or passenger display, etc. There may also be multiple logical displays, such as two separate regions on the same physical display screen. Each display may be assigned a unique identifier.

[0203] The projection model can include a "display area" element. Such an element can represent an area within a display element where data from the gateway 100 should be presented. There can be multiple display areas within a single display element. Display areas are typically rectangular in shape, but any shape can be supported. Display areas can overlap, and the order in which they are displayed can be controlled via a z-level property that specifies the stacking order of the elements. The gateway 100 can configure display areas using a "display area request" message and stream the actual data using a "display area data" message. Each display area can be assigned a unique identifier (called a display area ID). Display area IDs can be unique regardless of the display to which they are assigned. In this way, a single display area ID can identify where each data frame from the display area data message should be presented.

[0204] The projection model may include an "encoding format" element. Data to be displayed in a given display area may be encoded in a format supportable by the projection client 570. Multiple encoding formats may be supported. The format selection may be based on a balance of image quality, speed, and bandwidth requirements. For example, the display data may be provided as bitmap data (e.g., H264 video, Moving Picture Experts (MPEG) format or similar formats such as MPEG-2, Joint Photographic Experts Group (JPEG) format, Motion JPEG, H265 video, etc.) or as vector data. There may be multiple display areas on a particular display that use different encoding formats. For example, one display area may display a captured screen from a source application, while another display area may display an overlay (e.g., with a text message and a semi-transparent rectangle) to block certain areas of the captured image.

[0205] For data provided in bitmap format, several standards offer high compression rates while maintaining quality, and many user devices 110 include hardware acceleration support for such formats. For user devices 110 that do not support such formats, a simpler bitmap representation of each captured frame can be used. To reduce the bandwidth requirements for each frame, the color depth of each pixel can be reduced (e.g., to RGB565) or converted to a different color space (e.g., YUV422) to reduce the bits required for each color. Each frame can be compressed using a fast compression algorithm. Furthermore, instead of sending a complete image frame, the gateway 100 can encode only the difference from the previous frame and then compress the difference data (e.g., using run-length encoding or another compression technique).

[0206] To support flexible display area configuration, the projection client 570 can support a display area that renders vector data sent by the gateway 100. For certain types of images or other graphical content (e.g., simple solid color blocks, text, etc.), vector data is a much more efficient way of representing frame data than any bitmap format. The data can be represented in standard vector formats such as Scalable Vector Graphics (SVG) or Dual Express Graphics (DXG). In some embodiments, the data can be represented in a proprietary format using protocol buffers to store records. Each record can describe one of the supported graphical primitives and provide drawing parameters. The records can be further compressed to reduce bandwidth requirements.

[0207] To establish a screen projection session, after establishing a connection between the gateway 100 and the projection client 570, the projection client 570 can send a "display capability" message to the gateway 100. The gateway 100 may initialize the displays to which the projection data will be sent by sending a series of "display request" messages. If the requested displays (and / or associated data formats) are supported, the projection client 570 can send a "display response" message containing a successful result code and / or other indication of success. For each display, one or more display areas can be initialized by setting the display area dimensions, type, and / or other attributes and sending a "display area request" message from the gateway 100 to the projection client 570. Once the requested display areas are successfully created, the projection client 570 can send back a "display area response" message containing a successful result code.

[0208] To establish data streaming, the gateway 100 can initiate screen capture using a module such as the application capture module 505. The captured image can be passed to the projection engine 520. The projection engine 520 can pass the captured image screen to the application control module 510. The projection engine 520 can compose a final image frame (which may include one or more overlays) for each display area. The projection engine 520 can send the display area frame to the projection client 570 via a "display area data" message.

[0209] The gateway 100 can select the optimal screen capture mode based on the currently available connection methods, the capabilities of the connection resource 120, and / or other relevant factors. For example, while a user is projecting content, the user may disconnect or connect a cable between the user device 110 and the connection resource 120. As a result, the gateway 100 can update or change the screen capture method. For example, the connection resource 120 may initially connect to the user device 110 via a wired HDMI connection. However, when the cable is disconnected, the gateway 100 can switch to gateway capture mode and send content via a Wi-Fi connection without disrupting the user experience.

[0210] When capturing an application screen using one of the techniques described herein, the gateway 100 may need to operate in the background because the user may want to interact with one of the source applications in the application library 540 rather than with an application associated with the gateway 100. In some embodiments, the gateway 100 can run a background companion application that can run in the background. iOS may have more restrictive rules regarding which applications are allowed to run in the background. The gateway 100 for iOS may use an External Accessory Protocol (EAP) connection to the connectivity resource 120 to allow the gateway 100 to run in the background (and not be terminated by the OS). Such background companion applications and / or other similar features may be used with any of the projection modes described herein. If the main data channel is not an EAP connection, an alternative low-bandwidth channel (e.g., Bluetooth) may be used to send occasional ping messages to maintain an active connection and keep the gateway 100 running in the background. For Android-based user devices, there is typically no such requirement, as background services may be allowed to run without restrictions.

[0211] Application projection can utilize various display messages that can be used to configure and / or control video streaming (and / or the provision of other content) from user device 110 to connected resource 120. Various examples of such messages are provided below.

[0212] An example of a display message may be a “display capabilities” message that the projection client 570 may send to the gateway 100 to indicate the capabilities of the connection resource 120. The gateway 100 may respond with a display capabilities response message. Each display capabilities message may include elements such as a supported external display technology element indicating supported external display technologies. The technology may be represented by a fixed constant, such as a string or integer flag, that describes the capabilities, such as whether the device supports AirPlay, MiraCast, ChromeCast, HDMI, etc. Each display capabilities message may include a connection status element that indicates whether and / or how the user device 100 is connected to the connection resource 120 (e.g., established wireless connection, HDMI cable connection, etc.). Each display capabilities message may include a display characteristics element that includes or may indicate a list of display characteristics for each supported display. The display characteristics may include, for example, screen resolution, color depth, supported video encoding formats, and / or other related information.

[0213] Another example of a display message may be a “display request” message. Such a message may be sent from the gateway 100 to the projection client 570 to request registration of a display and initiate external display projection of source application content. The same message type may be sent from the projection client 570 to the gateway 100 to request user action. The recipient of the display request message may send a display request response message in response. The display request response message may include elements such as a request ID that must be included in the display request response message to identify the display request message to which a response should be sent. The display request message may include a display ID element that indicates the unique ID of the display for which this request is made. If the display request message is the first message from the gateway 100, the display ID may be generated by the gateway 100. The display request message may include a display description element that may identify the display to which the external display ID is assigned. The display description field may be populated when a new external display ID is assigned or generated. The display request message may include an action element that indicates an action to be performed as part of the request. Possible actions include "connect" (requesting the client to connect a display to the user device 110 and start projecting), "disconnect" (requesting the client to stop projecting the display and disconnect), "user action request" (requesting the recipient to display a prompt message to assist the user in establishing a connection), "display message" (requesting the recipient to display a specific message such as user instructions), etc. The requested user action field can include at least one code indicating the type of action (e.g., "pair devices," "turn on Wi-Fi," "connect USB / HDMI cable," etc.).The display request message may include a message element indicating an optional message to be displayed by the recipient, if indicated by a message display action (and / or other appropriate criteria).

[0214] Another example of a display message may be a "display response" message. Such a message may be sent in response to a display area request message. The display response message may include elements such as a request ID (matching the request ID received in the corresponding display area request message), a result (describing the outcome of the requested action and may include predefined result constants such as "start," "stop," "user action requested," or "failed"), and an error (containing information about any errors associated with the failed request). The error field may be a composite field containing an error code and / or a human-readable error message.

[0215] Another example of a display message may be an “external display status change” message. Such a message may be sent by the projection client 570 or the gateway 100 when the status of the external display changes. The projection client 570 may provide the latest state of the display. The gateway 100 may send the external display status change message to query the projection client 570 for the latest status. The external display status change message may include elements such as the external display ID of the associated display. The external display status change message may include a display description element (an optional field sent by the projection client 570). The display description field may be sent only when the display is first started or when the display status is requested. The display description field may be complex and may include subelements such as the selected mode (e.g., streaming technology), screen resolution, frames per second (FPS), and transport method. The external display status change message may include a status element that may indicate the status of the external display. Such status may be limited to a predefined set of constant values ​​such as “start,” “stop,” “pause,” “resume,” and “query.” The “query” status may be used to request a status update.

[0216] Another example of a display message may be a “display capture” request message. Such a message may be sent by gateway 100 to request that projection client 570 manage the capture of video frames from a given external display. Projection client 570 may respond with a “display capture” response message. Each display capture message may include elements such as a request ID (a unique value that identifies the request), an external display ID (an identifier of the display on which this capture request should be performed), a requested action (e.g., video capture “start,” video capture “stop,” etc.), capture resolution (e.g., the resolution of the image to be captured), encoding type (indicating the video encoding format at which the image is transmitted, such as H.264, compressed image, video JPEG, etc.), frames per second (i.e., “FPS,” the frequency at which the image is transmitted), and bandwidth (the maximum bandwidth to be applied to the resulting stream).

[0217] Another example of a display message may be a "display capture response" message that may be sent in response to a display capture request message. The display capture response message may include elements such as a request ID (matching the ID of the corresponding display capture request), a result (describing the result of the requested action and may include predefined result constants such as "started," "stopped," or "failed"), and an error (containing information about the error for a failed request and may be a composite field containing a human-readable error message as well as an error code).

[0218] Another example of a display message may be an "external display frame capture" message. Such a message may be sent by the projection client 570 each time a frame is captured for an external display configured to capture frames via a display capture request message. The frequency of the external display frame capture message may be set via the display capture request message. The external display frame capture message may include elements such as an external display ID (which identifies the display from which the screen is captured) and frame data (the data for a given screen frame in the format and size specified in the display capture request message).

[0219] Another example of a display message may be a "display area" request message. Such a message may be used by the gateway 100 to create, show, hide, destroy, and / or otherwise modify a display area on a given display. The projection client 570 may respond with a display area response message indicating success or failure. The display area request message may include fields such as a request ID (a unique identifier for the request), a display area ID (a unique identifier for the display area, which must be new and unique for a request to create a new area that matches the ID of a previously created display area), an action (defines the action to perform), and a display area description (for the newly created display area).

[0220] The action element can be selected from elements such as "Create" (requests the recipient to create a new display area, in which case a value must be entered in the display area description field), "Show" (displays the specified display area), "Hide" (hides the specified display area), "Destroy" (destroys a previously created area, which can also be hidden), and "Query" (the application requests the latest state of the specified display area). After a destroy request, the display area ID is no longer valid and can be reused for a new display area.

[0221] The display area description element may define the parameters for a new display area. The display area description field need only be populated during a create action. The display area description element may contain sub-elements such as "display ID" (indicating the ID of the display on which the new display area should be created), "location" (indicating the location and / or size of the display area), "z level" (indicating the z level of the display area), "encoding format" (indicating the intended encoding format for the data), transparency (indicating any transparency information for the area, such as alpha blending), and / or other appropriate sub-elements.

[0222] Another example of a display message may be a "display area" response message. Such a message may be returned by the projection client 570 in response to a display area request message. The display area response message may include a selection of discrete values, such as a request ID (matching the request ID of the corresponding display area request), a status (indicating the most recent status of the requested display area, which may be "visible," "hidden," "discarded," "failed" (if there was an error trying to create the display area), and an "error" (containing information about the error that caused the failed request, which may be a compound field containing a human-readable error message as well as an error code).

[0223] Another example of a display message may be a "display area data" message. Such a message may be sent by the gateway 100 for each frame of data for a given display area. The display area data message may include elements such as a display area ID (an identifier for the display area for which data is provided) and frame data (data for a single frame to be displayed). The frame data may be provided in an encoding format specified in the display area request message.

[0224] The process may include receiving (760) UI event information at gateway 100. Such UI event information may be received from connectivity resource 120. The UI event information may include information related to various types of UI elements, provided in various suitable formats. For example, the UI elements may include buttons associated with individual values ​​or functions. As another example, the UI elements may include a touchscreen capable of capturing one or more touch points and / or touch point movements.

[0225] As shown, process 700 may include applying (770) the UI event information to a source application. As described herein, various input event data processing and / or handling algorithms may be utilized to receive and / or identify such UI event information. The UI event information may be applied to the source application via a set of messages sent using DCCP. Gateway 100 may analyze the received UI event information and apply the information to the source application via a set of messages or other data sent over an available communication channel.

[0226] DCCP can include a set of various messages that can be used to configure the input capabilities of the connection resource 120 and, optionally, to control the user device 110 (and associated source applications). For example, an "input capabilities" message can be sent from the projection client 570 to the gateway 100 to describe the capabilities of the connection resource 120 in terms of supported input functions. The gateway 100 can respond with its own input capabilities message.

[0227] An input capability message may include elements such as a request ID that uniquely identifies the request. The corresponding response must contain the same request ID. An input capability message may include an input device element that provides a list of input devices. For each input device in the list of input devices, it may include or provide a "type" that defines the type of input control (e.g., touchscreen, keyboard, HW button, joystick, etc.) and an "interface" that defines the communication interface (e.g., HID, iPod Out, etc.). The interface may describe the external user input control channel (e.g., HID over USB) and / or the data sent over the main data channel (e.g., TCP / IP over Wi-Fi). An input capability message may include a data device element that provides a list of data devices. For each data device in the list of data devices, information may be included such as a "Type" that defines the type of data device (e.g., GPS, location provider, temperature sensor, accelerometer, gyroscope, etc.), an "Interface" that specifies the connection method (e.g., data message, HID, BT NMEA, iPod location, etc.), and / or a "Supported Data Inputs" element that specifies what type (and in what units) of data is provided. Standards such as the Vehicle Signal Specification (VSS) for Geneva In-Vehicle Infotainment (Genivi) may be utilized.

[0228] As another example, a "set input" message can be used to configure one or more input devices and / or input regions. The set input message can be initially sent by the gateway 100. The same message can be returned in response by the projection client 570.

[0229] The input setting message may include a request ID element that uniquely identifies the request. The corresponding response must contain the same request ID. The input setting message may include a source action area element that defines a source action area. The source action area element may include a list of input area records, each of which may include an “input area ID” field containing a unique identifier for the input area, a “shape” field indicating the shape of the area, a “type” field indicating the type of input area, and an optional “device ID” field containing the device associated with the given input area. The device ID may be sent by the projection client 570 in an input setting response message. The input setting message may include a target action area element that defines a target action area, which may include properties such as shape, orientation, etc. The input setting message may include a keyboard device element that provides a list of keyboard input devices. Each device in the list may include a device ID and type. The device ID may be assigned by the gateway 100. The keyboard device element may include a “parameters” field containing parameters for the keyboard device, such as which buttons to enable. The input setting message may include a data input device element that provides a list of data input devices. For each data entry device in the list, information such as the device ID (the device ID assigned by the gateway 100) and data entries (providing a list of data entries to be sent by the device) may be indicated or included. For each data entry, the gateway 100 may assign a data entry ID that can later be used to send the input data.

[0230] DCCP may include "control input" messages that may be sent by gateway 100 to start, stop, and / or otherwise control a given input device. The status of the command may be obtained via external input notification messages.

[0231] A control input message may contain a control area element that contains a list of control input areas. Each element in the list of control input areas may contain fields such as "Input Area ID" (the ID of a previously configured input area unless a "calibrate" action is used), "Device ID" (the ID of the device for the calibration action), "Action" (specifies the action to perform, which may be selected from a set of distinct values ​​such as "Start", "Stop", or "Calibrate"), and "Calibration Parameters" (specifies the parameters for the calibration, which may include parameters such as step, scale, docking mode, and docking position such as "nearest" or "top-left corner").

[0232] The control input message may include a data input device element that specifies a list of data input devices to start and / or stop. For each data input device in the list, it may include information such as a device ID (the ID of the previously configured device), an action (the type of action to perform on the device, such as "start" or "stop"), and parameters (providing additional parameters for the device or for each device input, such as data collection frequency, any transformations, etc.).

[0233] The DCCP may include a "send input event" message sent by the gateway 100 requesting that the projection client 570 execute a specified input event with a given device (in this case the device ID must have been previously configured via a control input message). The send input event message may be used to send an event (either an input event or a data event) from the projection client 570 to the gateway 100 over the main data channel.

[0234] A send input event message may include an "input event" element containing a list of events to send, where each entry in the list may include an "event ID" field that uniquely identifies the event, an "event type" field that specifies the type of event to send (e.g., cursor move, mouse up move, mouse down move, etc.), and a "position" field that defines the location of the event (the location may be absolute or relative). A send input event message may include a "data event" element containing a list of data events, where each entry in the list may include a "data input ID" field that uniquely identifies an input device previously assigned by a control input message, a "timestamp" that indicates when the data was collected or received, and a "data" field that contains the actual data payload.

[0235] The DCCP may include an "External Input Notification" message. Such a message may be sent by the projection client 570 when an external input event occurs on the connection resource 120 and is sent via external user input. This message may be received by the gateway 100. The External Input Notification message may include an Event ID element that uniquely identifies the event (if requested in the Send Input Event message). The External Input Message may include an Event Type element that indicates the event type (e.g., "Input Start," "Input Stop," "Up," "Down," "Single Touch," "Multi Touch," etc.). The External Input Notification message may include a Region ID element that includes a unique identifier of the region where the event occurred, a Device ID element that includes a unique identifier of the input device that generated the event, and a Timestamp element (specific to the projection client 570) that indicates when the event occurred.

[0236] User input (and associated UI event information) can be processed using a variety of modes, including an external user input mode and a gateway user input mode. The gateway 100 can select the optimal user input processing mode based on the currently available connection channels, the UI capabilities of the connection resources 120, and / or other relevant factors. The user input processing can attempt to control the source application 545 based on the UI information received from the connection resources 120 as if the end consumer were interacting directly with the user device 110.

[0237] For input controls that rely on pointing devices such as a mouse, a touchscreen (single or multi-touch), a stylus, a digitizer, and / or other similar types of devices, the gateway 100 may use or implement flexible configuration of associated input areas on the connection resource 120 and on the user device 110. An example of such a flexible configuration area was described above with reference to FIG.

[0238] Returning to process 700, one way to control the source application 545 is to use any standard (external) available control path. Such an approach is sometimes referred to as “external user input control.” One or more external control channels can be included in (or provided by) the user device interface 575. One such channel is the HID standard, first introduced for USB and now supported by various communication paths, such as Bluetooth, serial, and ZigBee. HID channels can support a variety of different device types. A user input device (e.g., connected resource 120) can present an HID descriptor to the host. The HID descriptor can include an array of bytes that describes the data packet structure used by the device and indicates the number of packets the device supports, the size of the packets, the purpose of each byte and bit within the packet, and / or other relevant information. The user input device can send actual data in subsequent HID report records that match the previously transmitted HID descriptor.

[0239] The connection resource 120 may be able to mimic various input device types to the user device 110 (and / or associated source applications) via device control modules that may be included in (or executable by) the user device interface 575.

[0240] An example of a type of input device is an "absolute pointing device." Such devices can be included in the HID category used to describe single or multiple point pointing devices, such as digitizers, styluses, single-touch pads, or multi-touch pads. The main feature of such devices is that the connection resource 120 can send absolute coordinates for single and / or multiple touch points, thus simulating touch events on, for example, the main screen of the user device 110.

[0241] In some embodiments, the absolute pointing device can be initialized by generating or otherwise preparing an HID descriptor in the user device interface 575. The descriptor can define the extent or bounds of HID coordinates that can be sent via HID reports or HID messages. The user device interface 575 can send the HID descriptor to the user device 110 via an external control channel (e.g., USB) and / or other available communication channels.

[0242] Absolute pointing device input data can be generated when the projection client 570 receives pointing device input event data (e.g., touchscreen, mouse, joystick, external touchpad, and / or other such input event data). The projection client 570 can evaluate the received coordinates and determine, for example, whether the coordinates are located within any of the input regions defined in the input setting message. When there are multiple touch points (associated with the same event) located within different regions, the event data can be processed by different handlers based on the location of the regions. For example, if a touch point starts in a first input region and moves to a second input region, the information can be ignored. If the touch point starts in a first input region and remains within the first input region, the event information can be processed by an event handler associated with the first input region.

[0243] If the coordinates fall within the external input area, the event information can be forwarded to the user device interface 575, which can convert the input coordinates (for each associated touch point) and send an HID report (which may include multiple reports or entries per touch point, depending on the capabilities of the user device 110) over an external control channel or other available communication path, and send an "External Input Notification" message to the gateway 100. Converting the input coordinates can include scaling and translating the received input coordinates from projected client area 220 coordinates to target action area 210 coordinates, followed by scaling and translating to HID coordinates (taking into account the screen orientation reported in the input setting message).

[0244] Another example of an input device type is a "relative pointing device." Such devices can be included in the HID category used to describe single pointing devices such as mice, trackpads, and trackballs. The main feature of these types of devices is that the resource 120 can send relative coordinates for cursor movement on the primary screen of the user device 110. Associated "click" type events (one, two, or more buttons in various configurations) can be sent as part of the HID report.

[0245] If the connected resource 120 uses a touchscreen as an input device and needs to control the user device 110 through a relative pointing device interface, absolute coordinates must be translated into relative cursor movements and clicks. To perform such translations, the connected resource 120 needs to know the characteristics of the relative pointing device, such as steps, current position, direction, etc. This information can be discovered through a calibration process. In some embodiments, the gateway 100 can utilize cursor docking to ensure error recovery.

[0246] Through the calibration process, the connection resource 120 can discover various characteristics of the pointing device supported by the user device 110, such as the cursor step (which indicates the number of steps the cursor moves on the screen to move one step via the HID). There can be separate step values ​​for each of the x-axis and y-axis. In some embodiments, paired step values ​​can be specified based on the speed of movement (if there is an acceleration curve supported by the user device 110). The calibration process can further discover characteristics such as the current position of the cursor and the docking step (the number of relative steps to send via the HID to dock the cursor). A different step value can be specified for each docking corner.

[0247] Calibration can typically be performed for each newly connected user device 110 and / or when the user device 110's OS or settings are updated. To determine whether calibration is necessary, the gateway 100 can determine whether the latest calibration parameters are valid. Such a calibration check can include displaying a full screen (e.g., a calibration check screen) from the gateway 100 to the user device 110, from which all input event capture can begin. The gateway 100 can activate the necessary pointing device by sending a control input message containing the latest calibration parameters to the projection client 570. The projection client 570 can save the device parameters and pass the message to the user device interface 575, which can activate the HID device. The projection client 570 can detect the cursor position by sending a send input event message providing a zero (e.g., 0,0) movement offset and a button down status. The user device interface 575 can send one or more HID reports indicating zero movement and button up / down status. The emulated click may be received at the gateway 100 via a calibration check screen. The gateway 100 may store the initial position associated with the emulated click.

[0248] The gateway 100 can send a series of pointing device moment and click events via one or more send input event messages. The gateway 100 can send absolute coordinates that fit within the dimensions of the calibration check screen. The user device interface 575 can send one or more HID reports via an HID channel (e.g., an external control channel) based on the received send input event messages. The absolute coordinates can be converted into a set of relative events (based on the latest calibration parameters). The gateway 100 can receive and capture clicks from the calibration check screen and associate the clicks with input events. The gateway 100 can calculate an error score based on the difference between the received cursor position and the expected position based on the latest calibration parameters. Based on the error score, the gateway 100 can determine whether the calibration is valid (e.g., whether the error score is below a specified threshold). If calibration is not required, the projection client can begin sending UI event data.

[0249] The calibration process can be a collaboration between the gateway 100 and the projection client 570. To initiate the calibration process, the gateway 100 can send a "control input" message to the projection client 570 that sets the input region associated with the current HID device to a "calibrating" state. The user device interface 575 can send an HID descriptor of the relative pointing device to the user device 110 over an external control channel or other suitable channel. The user device interface 575 can send an external input notification message to the gateway 100 that includes the input region ID and a status indicating that the calibration process has begun.

[0250] The gateway 100 can display a full screen (e.g., a calibration screen) on the user device 110 and start capturing all received events from the calibration screen. For operating systems such as iOS and Android, there are APIs that allow third-party applications to transition to full-screen mode. There may be system areas where the gateway 100 cannot receive HID input. The gateway 100 can detect such areas and manage the calibration process accordingly.

[0251] The gateway 100 can send calibration events via a Send Input Event message. For example, the gateway 100 can send various calibration sequences such as [8, -8], [-8, 8], [16, -16], [-16, 16], [24, -24], [-24, 24], [32, -32], [-32, 32], etc., where each pair represents a relative movement along the x and y axes (e.g., [<offset x>, <offset y>]), followed by a click event that allows detection by the gateway 100. After each event, the projection client 570 can respond with an External Input Notification message.

[0252] The gateway 100 can capture and record events from the calibration screen during the calibration sequence. The gateway 100 can filter out any unexpected events. For example, an end user may accidentally manipulate the user device screen and generate input events. Unexpected or erroneous events can be filtered out based on various association criteria. For example, a specific button press can be used to indicate that the data is related to a calibration click (e.g., a middle mouse button click, or a sequence of clicks such as left click-right click-left click). As another example, a special up-down sequence of clicks can be used to indicate whether a click is related to calibration. Such a sequence can be very difficult to reproduce accidentally (e.g., there could be a down / up / down / up sequence for each point checked). As yet another example, the captured events can be aligned with notifications received from the projection client 570. The gateway 100 can record the received screen point (in user device screen coordinates) and associate the received screen point with the emulated event. The gateway 100 can store a timestamp indicating when the event was received.

[0253] The gateway 100 can calculate calibration characteristics based on the recorded points. Steps for each of the coordinate axes can be calculated by determining the relative movement of the screen pointer (from the screen point) to the movement generated by the input device using the provided input points. If there are different steps based on the speed of the event, they are recorded as separate steps for that speed. The latest cursor position can be extracted from the last recorded screen point. Docking steps can be calculated based on the estimated steps and the size of the user device screen.

[0254] The gateway 100 can send the calibration characteristics to the projection client 570 via a control input message containing the device ID of the calibrated device and the calibration parameters. The projection client 570 can apply the calibration parameters and save the parameters for future sessions. The gateway 100 can perform a calibration check and, if the calibration check is not successful, can retry the calibration, and the number of retries can be configurable.

[0255] One challenge with relative pointing devices is that they are designed to control the movement of a cursor visible to the end user. As a result, the target device does not report its actual position through the HID interface. The projection client 570 must track the cursor position. An expected cursor position can be calculated based on the provided input HID report and previously calculated calibration parameters. However, errors may occur and the actual cursor position may be incorrect. For example, the movement steps calculated during calibration may have small errors. Even relatively small errors can accumulate over several movements, resulting in a significant difference between the estimated cursor position and the actual position.

[0256] To overcome such errors, the gateway 100 can implement an error correction mechanism using cursor docking. Docking can involve moving the cursor to a known position so that the projection client 570 can ascertain the cursor position and perform accurate relative movements from there. Docking relies on the fact that excessive pointer movement to one of the corners of the screen causes the cursor to "anchor" at that corner, resulting in a known position.

[0257] The cursor may be docked at any of the four corners of the screen. To dock the cursor, the projection client 570 can send a set of docking commands, such as relative movement events equal to or greater than the dimensions of the user device's screen. To account for errors in the calibration data, the calculated screen dimensions can be multiplied by 2 (or some other appropriate factor) to ensure corner docking of the cursor.

[0258] In some cases, high resolution screens can slow performance by sending too many points. In such cases, the multiplication factor can be adjusted to reduce excessive movement (e.g., changing the factor from 2 to 1 and 2 / 10). Docking can be performed by the projection client 570 as part of the normal event sending sequence.

[0259] Various docking modes are available. Such docking modes can include a "pre-click" mode in which the projection client 570 can initially dock the cursor and then move the cursor from a known docked position to the desired position. Such an approach ensures the most accurate final position. The drawback is that the end user may see the cursor move back to the docked position.

[0260] Docking modes can include an "after-click" mode in which docking occurs after a click event is received. Such an approach reduces the visual movement of the cursor before the click. If there are other ways of cursor movement that cannot be detected by the projection client 570 (e.g., the user may interact with the user device 110), the estimated cursor position for the next event may be inaccurate.

[0261] Docking modes can include an "after inactivity" mode that can perform docking after a specified period of inactivity. Such an approach reduces the visual movement of the cursor to and from the docking point. As with after-click mode, if there are other methods of cursor movement that cannot be detected by the projection client 570, the estimated cursor position for the next event may be inaccurate.

[0262] The projection client 570 can select the docking mode and target location based on information received via control input messages from the gateway 100. Docking can be performed with respect to a specific corner or the nearest docking location.

[0263] Once the pointing device has been calibrated and / or verified to operate correctly, UI event information, such as touch event information received from the connection resource 120, can be emulated by the gateway 100 to the user device 110 as relative pointing device event information. In use, the projection client 570 can receive or identify pointing device input events (e.g., touchscreen, mouse, joystick, external touchpad, and / or other such input).

[0264] The projection client 570 can determine whether the received coordinates fall within any of the input regions received via the input setting message. If there are multiple touch points that fall within different regions, the points can be processed by different handlers based on the region. If a touch point starts in a first region and moves to a second region, the associated event information can be ignored. If a touch point starts in a particular region and remains in that region, the associated event information can be processed by the corresponding handler for that particular region.

[0265] If the received input coordinates are within the external input area, the event information can be forwarded to a device control module of the user device interface 575. For each matching touch point, the device control module can convert the received coordinates by scaling and translating from projected client area coordinates to target action area coordinates, and then to HID coordinates (taking into account the orientation reported in the input setting message).

[0266] The device control module can perform docking based on the contents of the input settings. The device control module can calculate the necessary relative movement events based on the latest input position and the cursor position on the target screen. Calibration parameters received via the input setting message can be used to calculate the relative movement. The device control module can send HID reports (each touch point can be associated with multiple reports) via, for example, an external control channel. The device control module can send external input notification messages to the gateway 100.

[0267] In addition to or instead of a touchscreen or other touch-based input, HID elements can include other input types, such as keyboards, game controllers, etc. Such elements can be or comprise physical and / or virtual components. For example, some connected devices 120 can include a physical keyboard or keypad, while other such devices can present a keyboard or similar resource via a touchscreen display.

[0268] If a keyboard HID is available, the projection client 570 can report support for such a keyboard HID through an input capabilities message. As a result, the gateway 100 can request activation of the keyboard to receive data for input fields or to simulate specific actions on the source application 545. Before activating the keyboard, the gateway 100 can configure the keyboard device by sending an input configuration message.

[0269] The keyboard HID can be activated, for example, when data is needed for an input field in the source application 545. For example, the user may be prompted to enter information such as a search term, a username, or a password. The gateway 100 can automatically detect that a text input field has application focus (e.g., based on information, if any, received from the source application and / or the OS interface) and instruct the projection client 570 to activate the keyboard HID by sending a control input command specifying the device ID of a previously configured keyboard device. As a result, the projection client 570 can use an external keyboard connected to the connection resource 120 or display an on-screen virtual keyboard. Any button press information received from the keyboard can be sent to the user device 110 via the external control channel.

[0270] The keyboard HID can be used to simulate or emulate various actions in the active source application 545. For example, the keyboard HID can simulate a "back" button, media play / pause, next / previous, etc. The gateway 100 can initiate such emulation by initiating the keyboard HID via a control input message and specifying only supported key codes (e.g., those associated with consumer control devices such as a mouse, keyboard, etc.).

[0271] If a specific command needs to be simulated based on received UI event information, the gateway 100 can send an input event message specifying the virtual key code of the command to be simulated. As a result of the message, the device control module can send the necessary HID report along with the key code. The device control module can translate the key code into a value that the active HID device understands or recognizes.

[0272] The HID protocol is highly configurable and can be used to provide many different types of input. For example, consumer control input devices such as game controllers can be treated similarly to the keyboards, absolute pointing devices, and relative pointing devices mentioned above.

[0273] Any new input device can be easily added to the system via the input capability message, in which case the projection client 570 reports available HID capabilities, and the gateway 100 can configure such devices via the input configuration message and then start using a specific device via the control input message. User interactions with the connection resource 120 can be sent as HID events, or the gateway 100 can instruct the projection client 570 to simulate a specific HID event via the send input event message.

[0274] In addition to HID, the user device 110 may support other proprietary or open external input methods, such as iPod control, control via an audio interface, etc. Operations similar to those described above may be used for such interfaces.

[0275] In addition to or instead of external user input, user input can be provided to the source application 570 through the gateway 100. Depending on the capabilities, version, and constraints of the user device 110, various APIs can be provided to third-party applications that enable simulation of input events. For example, if the gateway 100 is provided with the necessary permissions, various third-party Android applications can be used to simulate keyboard and touch events through accessibility services (e.g., using the accessibility interaction client class). If the gateway 100 has system permissions (e.g., root access), the gateway 100 can emulate input events by interacting with the OS's input subsystem.

[0276] To send input events from the gateway 100, the projection client 570 can receive or identify pointing device input events (e.g., touchscreen, mouse, joystick, external touchpad, and / or other similar inputs). The projection client 570 can determine whether the received coordinates are located within any of the input areas defined via the input setting message. If there are multiple touch points that fall within different areas, the different points can be processed by different handlers associated with the different areas. If a touch point starts in a first input area and moves to a new input area, the input event information can be ignored. If a touch point starts in a particular area and remains within that area, the touch point can be processed by the handler associated with the particular area.

[0277] If the coordinates are located within the gateway input area, the event data can be sent in a send input event message to the gateway 100. In response to the send input event message, the gateway 100 can send the event information to the source application 545 using one of the supported channels and / or algorithms.

[0278] 8 shows an example process 800 for identifying a source application executed by user device 110. Identification of the source application was described above with reference to FIG. 3. Process 800 may be performed each time a new application is launched (e.g., based on generation or receipt of a most recent application message indicating the launch of the source application, a change in screen output, and / or other relevant criteria). In some embodiments, process 800 may be performed by gateway 100.

[0279] As mentioned above, some user device OSs may provide interfaces or other resources through which gateway 100 can receive active or current application information. Process 800 may be performed when such interfaces are not available and / or when such interfaces execute recognition algorithms different from and / or in addition to those provided by the OS interfaces. Additionally, process 800 may be used to filter content provided via connection resources 120. For example, content areas displaying video or other distracting elements may be excluded or blocked during certain operating conditions (e.g., conditions associated with a moving vehicle).

[0280] As shown, process 800 can include receiving (810) media output of a source application. Such output can include, for example, a screen image or other set of video content, audio content, and / or other media content. Such content can be received via a component such as application capture element 505.

[0281] Process 800 may include selecting (820) sections of the media output for analysis. Such sections may be selected or designated in a variety of suitable ways. For example, if data is received in a video streaming format, each entire frame may be selected for analysis. As another example, some source applications may include graphical user interface (GUI) elements, such as display windows, that may appear in various ways depending on the format of the received data. In such cases, areas associated with video displays or other active content may be selected for analysis, while areas associated with text, still images, or other similar elements may be ignored. As another example, as discussed above, various areas may be used for GUI elements, logos, or other similar content.

[0282] The process may include analyzing (830) the selected media output section using available recognition models (and / or other suitable methods). The recognition models may define factors such as the location of the output section to be analyzed, color profiles, matching image elements, and / or other suitable information. Additionally, each recognition model may be associated with a particular source application (as indicated by an application ID).

[0283] As mentioned above, various image comparison algorithms can be represented by recognition models that include comparison methods such as color bin-based analysis (as described with respect to color matching), pixel-by-pixel comparison (e.g., when the number of changed pixels exceeds a threshold number or threshold ratio), perceptual hashing algorithms (e.g., algorithms that use hashes to generate snippets or fingerprints of various forms of media), threshold-based scene detection (e.g., current screen intensity and / or brightness can be compared to a specified threshold, and analysis can be triggered if the intensity and / or brightness exceeds the threshold), and / or other suitable direct image comparison algorithms.

[0284] As shown, process 800 may include determining (840) whether the application is recognized. Such determination may be made based on criteria indicated by the associated recognition model (e.g., a minimum matching score or other metric threshold). If the application is recognized, the process may include providing (850) an application ID and / or other appropriate information, where the application ID may be obtained from the matching model used to recognize the source application.

[0285] This process may include providing (860) a default ID (if available) or otherwise indicating that no matching application was recognized. Such default application ID may be associated with an application or type of content. For example, some embodiments may include a default video application associated with a source application that provides full-screen video output. Such default application may be associated with various associated constraints (e.g., no video display while the vehicle is moving) and / or other attributes as described with reference to the application catalog.

[0286] 9 illustrates an example process 900 for receiving commands and / or other UI event information at a connection resource 120 and applying the received commands at a user device 110. Such UI events may be received via an external control channel or as gateway user input, as described herein. The process may be performed when a connection is established between the gateway 100 and a projection client 570. In some embodiments, the process 900 may be performed by the gateway 100.

[0287] As shown, process 900 can include receiving (910) UI event information from connectivity resource 120. Such event information can include different types of event data depending on different UI features or properties that can be associated with different input areas (e.g., whether absolute or relative pointing is used). The event information can include information related to multiple events and / or sub-events (e.g., a click event included in a click-and-drag operation).

[0288] Process 900 may include extracting (920) event attributes from the received UI event information. Such event attributes may include, for example, a set of coordinates, an action indicator (e.g., "down," "up," etc.), and / or other suitable information (e.g., identification of a keyboard button or key).

[0289] The process may include transforming (930) the event attributes based on available user device interfaces and / or other suitable criteria. For example, absolute coordinates may be transformed into relative movement information. As another example, action locations may be mapped from a resource such as the projection target area 230 to the target action area 210 (and vice versa).

[0290] As shown, process 900 may include emulating (940) events at the user device to apply UI events to the source application. Such emulation may utilize various messaging algorithms and / or elements, and / or other suitable methods of simulating or emulating event data, as described herein.

[0291] 10 shows an example process 1000 for filtering application content based on the operational state of connection resources 120. This process may be performed each time content is provided by gateway 100 to projection client 570. In some embodiments, process 1000 may be performed by gateway 100.

[0292] In addition to detecting the active source application, in some embodiments, the current screen or state of the application can be detected. Such detection can be used to restrict certain features or UI elements depending on the operational state of the connected resource 120 (and / or associated elements, such as a vehicle). For example, if the connected resource 120 is an in-car display unit and the vehicle is in motion, driver distraction regulations in many countries prohibit the unit from displaying moving content (e.g., video) or allowing a user to enter text using a keyboard. Process 1000 can detect whether such content is being displayed and block such content or prevent the entire application from being projected. Similar application restrictions may be required or implemented for other industries and use cases.

[0293] As shown, process 1000 may include receiving (1010) sensor data, if available, of the connected resource. Depending on the type of connected resource 120 (e.g., vehicle), various different types of sensor data may be available and / or relevant. For example, movement data may be provided (or calculated) based on information received from a GPS element, accelerometer, gyroscope, etc., that may be provided on or associated with the connected resource 120.

[0294] Process 1000 can include receiving 1020 sensor data of the user device. In addition to and / or instead of the sensor data of the connected resource, in some embodiments, sensor data such as positioning information or movement information can be received from the user device.

[0295] The process may include identifying (1030) operational conditions involving the source application 545, the connectivity resource 120 (and / or associated systems or devices, such as a vehicle), and / or other associated components. The process 1000 may identify various parameters and / or attributes associated with such operational conditions (e.g., the speed of the moving vehicle, the processor load of the connectivity resource 120, the network connectivity of the user device 110, etc.). Such data may be received from various elements, measured based on received data, such as sensor data, and / or otherwise generated or received.

[0296] The gateway 100 can detect the latest state of the source application 545. Such detection can be performed through resources such as the OS interface 560, the notification development kit 555, and / or the recognition engine 525 (e.g., using various image recognition algorithms described herein). The latest screen (or other source application output) can be reported to the application manager 515. The application manager 515 can use application behavior constraint information from the application catalog 590 to determine whether any or all UI elements should be blocked or otherwise filtered.

[0297] Similarly, the gateway 100 can detect the operating state of the connected resource 120, such as whether the resource is stationary or moving. Such determinations can be made by comparing received sensor data of the connected resource and / or user device with various state-awareness models. Such models can indicate, for example, a minimum distance and / or minimum speed required to indicate vehicle movement. The operating state can be associated with a temporal attribute (e.g., daytime versus nighttime operation). For example, displayed content may be dimmed or brightened depending on the time of day or detected light conditions (e.g., using data received from a user device camera or a connected resource camera or other sensor).

[0298] Similarly, the gateway 100 can detect the operating state of an associated user device 110. For example, the gateway 100 can determine what type of connection is available from the user device 110 to a resource such as a media server (e.g., Wi-Fi, cellular, etc.). As another example, the gateway 100 can receive information about the user device, such as processor load, memory usage, and / or other metrics that can be used to detect various operating states.

[0299] As shown, process 1000 can include determining 1040 whether any state-related constraints should be applied. Such a determination can be made based on various identified operating states, constraint states, and / or other operating rules, and / or other suitable criteria.

[0300] To manage different types of connection resources 120 and support different constraint states, a general constraint level value can be reported by the connection resource 120 via a "connected device status" message, which can include a constraint level element. The latest constraint level value can be passed to the application manager 515. Different constraint level values ​​can be provided depending on the type of connection resource 120.

[0301] For example, possible constraint level values ​​for an in-vehicle system may include "none," "small," "large," "maximum," and / or other similar values ​​and / or indicators (e.g., example values ​​may map to values ​​0 through 3). In such an example, a constraint level of none (or zero) may indicate that no constraints need to be applied (e.g., all UI is acceptable) and may generally be applied when the vehicle is not moving and the user is not involved in driving (e.g., when the parking brake is engaged or the automatic transmission is in "park").

[0302] A constraint level of small (or 1) can indicate that small constraints should be applied (e.g., most of the UI can be tolerated), and can typically be applied when the vehicle is moving very slowly (e.g., less than 8 km / h) or when stopped at an intersection (e.g., when no motion is detected and the automatic transmission is in "drive" or "reverse").

[0303] A constraint level of large (or 2) can indicate that large constraints should be applied (e.g., only UI elements related to driver assistance features may be allowed), and may typically be applied when the user is fully involved in driving and the vehicle is moving at a relatively high speed (e.g., greater than 8 km / h).

[0304] A maximum (or 3) constraint level may indicate that maximum constraints should be applied (e.g., no driving distractions are allowed and the UI may be completely blocked), typically when the vehicle is moving at very high speeds and / or is being driven under other dangerous conditions.

[0305] The above example describes various possible constraint level values ​​for an automotive implementation. Various other implementation examples may include various other constraint level values ​​and / or associated properties based on the specific requirements of the particular implementation state. For example, medical and / or industrial implementations may be associated with various other sets of constraint level values.

[0306] In some situations, the connectivity resource 120 may report its operational state. Alternatively, in some embodiments, the gateway 100 may determine the operational state (and / or any associated constraint level) independently of the connectivity resource 120. Such an approach may be useful when the connectivity resource 120 could be tampered with, causing an unsafe condition. For example, in an automotive system, the head unit typically uses wiring from the parking brake to detect whether the vehicle is running. Such wiring can be easily disabled by the end user, removing effective constraints on driver distraction from the system.

[0307] The gateway 100 can independently verify whether the vehicle is moving using the location and / or motion sensors of the user device 110, so that it can report the correct operating state and identify the correct constraint level. The gateway 100 can report the identified constraint level to the application manager 515.

[0308] Based on the current operational state of the connection resource 120 (and / or other components), a determination of whether to restrict the currently projected source application 545 can be made using the application operational constraint records in the application catalog 590.

[0309] A possible configuration for application behavior constraints for a given application in the catalog is shown in Table 2 below. This example uses a hierarchical structure stored in text or binary format. Storage formats can utilize JSON, XML, protocol buffers, etc. An application behavior constraint record can be associated with a list of matching rules that define the constraints and allowances for the application and / or associated widgets. Application behavior constraint records can follow a hierarchical matching structure similar to a cascading style sheet (CSS). Table 2 below shows examples of various high-level properties that can be associated with each rule record. Table 2 JPEG2025186218000004.jpg147166

[0310] Constraint rules can overlap based on widget and screen definitions. The main principle of correspondence is that the most specific rule is selected over more general rules. For example, a widget is more specific than a screen, and a widget ID is more specific than a widget type.

[0311] When the application manager 515 matches rules to the screen being projected, it may first apply the screen rules (if any), then rules that apply to the widget type, and then rules for the specific widget. Thus, for example, by defining a permissive rule for the widget ID of one widget, an application can be configured to block the entire screen and only allow interaction with that widget.

[0312] Constraints can filter or block certain operations (or data) from the UI associated with the projected application. Constraints can be associated with an application (or screen, widget, etc.) via the application catalog 590, as a state constraint element 690, and / or other suitable manner. Constraints can be defined as flags so that multiple constraints can be associated with a single rule. Some examples of constraints that may be supported are shown in Table 3 below. Table 3 JPEG2025186218000005.jpg106166

[0313] If the process 1000 determines that state-related constraints should be applied (1040), the process 1000 can include applying (1050) the state-related constraints identified in the matching rules. Such constraints can be applied in a variety of suitable ways depending on the attributes of the connected resource, the rule content, and / or other relevant factors. For example, a display block constraint can cause streaming data to stop or otherwise not be displayed on the connected resource 120.

[0314] If the process determines that no state-related constraints should be applied (1040), the process may include applying default constraints, or application-specific constraints, or no display or interaction constraints (1060).

[0315] As shown, process 1000 can include analyzing (1070) content received from a source application. Such analysis can include applying various recognition models to the screen content (or portions thereof). In some embodiments, such recognition models can be analyzed and applied to other types of content. As an example, a difference calculation can be used to identify areas of the UI or screen that may contain distracting content, such as video.

[0316] Process 1000 may include determining (1080) whether any constraint criteria are met, such as by evaluating the received content with respect to any available constraint rules. Such determination may be based on an analysis of the content, various identified operational states, constraint states, and / or other operational rules, and / or other suitable criteria. Such an approach may enable automatic detection of content types and application of constraint rules to such content types. Thus, for example, even if a source application does not directly provide video or similar content, similar benefits may be achieved through UI features and / or other elements.

[0317] If the process determines (1080) that the constraint criteria are met, the process may include applying a filter to the received content (1090), blocking such content, and / or otherwise manipulating the content as desired.

[0318] 11 shows an example process 1100 for applying machine learning to generate or update the application recognition models and / or other models, rules, or elements described herein. This process can be performed whenever additional model training data is available. In some embodiments, process 1100 can be performed by gateway manager 610, and specifically, by model trainer 650.

[0319] Machine learning algorithms based on neural networks (NNs) have shown very good results for various computer vision and image classification tasks. In some embodiments, the gateway manager 610 can be used to generate, train, and / or otherwise manipulate recognition models and / or related objects, such as rule objects. Rule objects that are part of a recognition model can include parameters, such as an associated NN model and NN model type, under an algorithm-specific data section. The NN model parameters can store the trained neural network model in a format readable by the machine learning library used by the recognition engine 525. The NN model type parameter can describe the type of neural network model provided in the NN model field.

[0320] There are many off-the-shelf tools and libraries that enable the rapid training and use of neural network-based image classification models. One such library is the Google ML Kit. Such tools can be run on the user device 110 to classify images using custom neural network models provided. Another such library is Apple's Vision framework, available on iOS devices. The Vision framework is optimized for iOS devices and enables the use of custom Core ML models to perform various machine learning tasks. Core ML models, as NN models, can be associated with rules that are part of a larger recognition model. The recognition engine 525 can utilize multiple neural network libraries and make a selection based on rule data, such as the NN model type.

[0321] As shown, process 1100 may include receiving (1110) content for an application. Such content may include video content, screen content, UI elements, photo elements, and / or other content related to the source application. Such content may be received from a resource, such as gateway manager 610, via repository 630.

[0322] The process 1100 may include receiving (1120) an application ID associated with the received content (if available). For example, a screen (or other content) used by the individual gateway 100 to identify the application (or other element) may be sent to the gateway manager 610 along with the ID of any matching application identified by the gateway 100.

[0323] The process may include receiving (1130) a recognition model associated with the ID received in 1120. Such information may be provided by the gateway 100 to the gateway manager 610 for storage and association with the application's content and / or ID, if any.

[0324] As shown, process 1100 can include receiving (1140) recognition feedback that can be used to train the model received in 1130. For example, a user (or other ML recognition model) can identify an application based on the received content.

[0325] Process 1100 can include updating 1150 the recognition model based on the received feedback. For example, if the application is correctly identified by the model, a reward can be applied to the model, while if the application is incorrectly identified, a penalty can be applied to the model.

[0326] Process 1100 and / or similar processes can be used to apply machine learning to the various models, rules, algorithms, etc. described herein.

[0327] A separate model (stored as a separate rule element) can be trained for each supported application. However, such an approach can slow down the classification process in the gateway 100 because the recognition engine 525 must evaluate multiple models. Alternatively, all supported applications can be trained with a single NN model with one rule. Depending on the accuracy of the combined models, a hybrid approach can be used, where the majority of applications are trained with a single NN model, but a subset of available applications and / or screen modes (e.g., keyboard mode) can be trained with separate NN models and rules.

[0328] The decision to use one rule for all applications or some combination of individual rules can be made during the training process using a meta-learning approach that tries various combinations and selects the combination that gives the best overall results (e.g., results that show the highest accuracy and performance).

[0329] The various image recognition algorithms described herein utilize trained models to correctly classify captured screen images, and such training can be performed as an offline process by the gateway manager 610.

[0330] To train a recognition model, model trainer 650 may require a large collection of sample images of screens taken from each of the supported applications. In addition, non-application images (e.g., "negative" examples) are required that can assist in the training and model evaluation tasks. Screens can be captured manually, by an automated process, or by a combination of the two. Captured images can be stored in an image repository (e.g., an image repository included in repository 630).

[0331] During the manual screen capture process, one or more human operators (or "analysts") can run an application on a user device 110 or a user device emulator and collect captured screen images. The analysts can then annotate the captured screen images with the correct labels. During this process, the analysts have access to all possible screens and can utilize various application operating modes. Screen images can be collected for various user device models, screen resolutions, screen orientations, display modes (e.g., "night," "day," etc.), etc.

[0332] Employing manual screen capture can be tedious and time-consuming, especially if the process must be repeated for each new version of the target application. In some embodiments, an automated application capture process can be utilized. The automated process can interact with the user device 110 in a manner similar to the screen capture and user input described herein. The automated application capture process can utilize an application debugging interface (such as ADB for Android or XCTest for iOS) that enables detection of active applications, screens, and / or UI widgets. As a result, the automated application capture process can “walk” through all screens and application modes, capturing sample screenshots during the process. Furthermore, the automated application capture process can know the current application and screen ID, allowing it to properly label images.

[0333] In some embodiments, a human operator can generate an application interaction path by recording a set of input events that can be replayed by an automated application capture process and screenshots collected during the replay process.

[0334] Screen capture images collected manually or by an automated process can be stored in an image repository, which can include a database of training and evaluation images captured for various versions of the application. Such an image repository can be a component of repository 630. The image repository can store label information associated with the captured images. Images can be categorized by application, application version, and / or other suitable attributes.

[0335] The model training process of some embodiments can perform model training. The model training process can be configured as to what type of model to train and what algorithm to use. The model training process can be configured with multiple possible algorithms and input parameters. The model training process can utilize meta-training to determine the optimal combination of algorithms, input parameters, and / or other relevant attributes.

[0336] To generate a recognition model, images can be retrieved from an image repository and various training algorithms can be run. The retrieved images can include training images and evaluation images (which can be manually specified or randomly selected from the training set). The recognition model can be evaluated and scored using a set of evaluation images (which can also be retrieved from the image repository). If the model meets some specified quality criteria (e.g., by exceeding a score threshold) or is an improvement compared to other models, the updated model can be saved to the model repository and tagged with the canonical version.

[0337] The model repository can be a database that stores the latest recognition models generated by the model training process. Recognition models can be versioned and have associated evaluation scores or other performance metrics. If the evaluation score exceeds a specified threshold, the recognition model can be uploaded to the gateway manager 610 and the associated application catalog record 590.

[0338] As an example of training a specific model (and / or rules), color matching rules can be trained using a set of training images by using the model data parameters as input parameters to a training algorithm.

[0339] Input parameters for the training algorithm may include a list of allowed image manipulations and a merging color bin threshold that may define the maximum number of additional color bins that two rules may be allowed to merge. The input parameters may be defined by a human operator and / or learned using standard meta-learning techniques.

[0340] The training process may generate or obtain an empty training model. For each training image, the training process may extract a label associated with the image. Such a label may typically be extracted from the file name or from a separate lookup table. For each training image and for each supported image operation, the training process may apply the image operation to the training image and save the result as the latest training image, compute color bins as described for the color matching classification algorithm, and save the result as a candidate rule.

[0341] For each training image, the training process may evaluate the candidate rules for the current training image and select the rule with the fewest number of color bins.

[0342] For each candidate rule, the training process may search for another rule in the training model with the same label. If no such rule is found, the candidate rule may be added to the training model. If a rule with the same label is found, the candidate rule may be merged with an existing rule by checking whether the candidate rule has fewer excess colors than the number of color bins threshold (or other specified threshold). If the number of candidate rules is less than the threshold, the color bins of the candidate rule may be combined with the color bins of the existing rule. If the candidate rule has more excess colors than the number of color bins to merge, the candidate rule may be added to the training model.

[0343] Those skilled in the art will recognize that processes 700-1100 can be performed in a variety of different ways without departing from the scope of the present invention. For example, elements may be performed in an order different from that shown. As another example, some embodiments may include additional elements or omit various elements listed. Elements or groups of elements may be performed iteratively and / or based on meeting certain activation criteria. Elements that are not dependent on each other may be performed in parallel.

[0344] The processes and modules described above may be implemented, at least in part, as a software process that may be specified as a set of one or more instructions recorded on a non-transitory storage medium. These instructions may be executed by one or more computing elements (e.g., a microprocessor, microcontroller, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), other processor, etc.) that may be included in various suitable devices to perform the actions specified by the instructions.

[0345] As used herein, the terms "computer-readable medium" and "non-transitory storage medium" are intended to be strictly limited to tangible, physical objects that store information in a form that can be read by an electronic device.

[0346] 12 is a schematic block diagram of an exemplary device (or system or devices) 1200 that can be used to implement some embodiments. For example, the systems and / or devices described above with reference to FIGS. 1-6 can be implemented, at least in part, using device 1200. As another example, the processes described with reference to FIGS. 7-11 can be implemented, at least in part, using device 1200.

[0347] Device 1200 may be implemented using various suitable elements and / or sub-devices. For example, device 1200 may be implemented using one or more personal computers (PCs), servers, mobile devices (e.g., smartphones), tablet devices, wearable devices, and / or any other suitable devices. The various devices may operate alone (e.g., device 1200 may be implemented as a single smartphone) or in conjunction (e.g., some components of device 1200 may be provided by a mobile device while other components may be provided by a server).

[0348] As shown, device 1200 can include at least one communication bus 1210, one or more processors 1220, memory 1230, input components 1240, output components 1250, and one or more communication interfaces 1260.

[0349] Bus 1210 may include various communication paths that enable communication between components of device 1200. Processor 1220 may comprise a processor, microprocessor, microcontroller, digital signal processor, logic circuit, and / or other suitable processing component that may be capable of interpreting and executing instructions and / or manipulating data. Memory 1230 may include dynamic and / or non-volatile memory structures and / or devices that store data and / or instructions used by other components of device 1200. Such memory device 1230 may comprise memory space within a single physical memory device or may be distributed across multiple physical memory devices.

[0350] Input components 1240 may include elements that enable a user to communicate information to the computer system and / or manipulate various operations of the system. Input components may include a keyboard, cursor control devices, audio and / or video input devices, touch screens, motion sensors, etc. Output components 1250 may include a display, touch screen, audio elements such as speakers, indicators such as light-emitting diodes (LEDs), printers, tactile or other sensory elements, etc. Some or all of the input and / or output components may be connected to device 1200 wirelessly or optically.

[0351] Device 1200 may include one or more communication interfaces 1260 that can be connected to one or more networks 1270 or other communication paths. For example, device 1200 may be connected to a web server on the Internet so that a web browser executing on device 1200 can interact with the web server when a user interacts with an interface running within the web browser. Device 1200 may be able to access one or more remote storage devices 1280 and one or more external components 1290 via communication interface 1260 and network 1270. Communication interface 1260 may include one or more application programming interfaces (APIs) that can enable device 1200 to access remote systems and / or remote storage, and that can enable remote systems and / or remote storage to device 1200 (or elements thereof).

[0352] Those skilled in the art will recognize that any or all of the components of computer system 1200 can be used in conjunction with some embodiments. Furthermore, those skilled in the art will understand that many other system configurations can be used in conjunction with some embodiments or components of some embodiments.

[0353] Additionally, while the illustrated examples may show many individual modules as separate elements, those skilled in the art will recognize that these modules may be combined into a single functional block or element, and that a single module may be divided into multiple modules.

[0354] Device 1200 may perform various operations in response to processor 1220 executing software instructions stored on a computer-readable medium, such as memory 1230. Such operations may include operation of output component 1250 (e.g., displaying information, haptic feedback, audio output, etc.), operation of communication interface 1260 (e.g., establishing a communication channel with another device or component, sending and / or receiving a set of messages, etc.), and / or operation of other components of device 1200.

[0355] Software instructions can be loaded into memory 1230 from another computer-readable medium or another device. The software instructions stored in memory 1230 can cause processor 1220 to perform the processes described herein. Alternatively, hardwired circuitry and / or special purpose components (e.g., logic circuits, ASICs, FPGAs, etc.) can be used in place of or in combination with software instructions to perform the processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0356] The actual software code or specialized control hardware used to implement the embodiments is not limiting of the embodiments, and thus the function and operation of the embodiments are described without reference to specific software code, and it will be understood that the software and control hardware can be implemented based on the description herein.

[0357] While particular connections or devices are shown, in reality, additional, fewer, or different connections or devices may be used. Furthermore, while various devices and networks are shown separately, in reality, the functionality of multiple devices may be provided by a single device, and the functionality of one device may be provided by multiple devices. Furthermore, multiple instances of illustrated networks may be included in a single network, and a particular network may include multiple networks. While several devices are shown as communicating with a network, some of such devices may be incorporated, in whole or in part, as part of the network.

[0358] Some implementations are described herein in relation to a threshold value. To the extent that the term "greater than" (or similar term) is used herein to describe the relationship of a value to a threshold value, it should be understood that the term "greater than or equal to" (or similar term) may be similarly considered, even if not explicitly stated. Similarly, to the extent that the term "less than" (or similar term) is used herein to describe the relationship of a value to a threshold value, it should be understood that the term "less than or equal to" (or similar term) may be similarly considered, even if not explicitly stated. Furthermore, the term "meeting," when used in reference to a threshold value, may refer to "greater than the threshold value," "greater than or equal to the threshold value," "less than the threshold value," "less than or equal to the threshold value," or other similar terms, depending on the appropriate context.

[0359] No element, act, or instruction used herein should be construed as critical or required unless explicitly stated otherwise. As used herein, an instance using the term "and" does not necessarily exclude the interpretation that the phrase "and / or" is intended in that instance. Similarly, as used herein, an instance using the term "or" does not necessarily exclude the interpretation that the phrase "and / or" is intended in that instance. Also, as used herein, the article "a" is intended to include one or more items and can be used interchangeably with the phrase "one or more." When only one item is intended, the terms "one," "single," "only," or similar language are used. Furthermore, the phrase "based on," unless otherwise specified, is intended to mean "based at least in part on."

[0360] The foregoing relates to specific details of exemplary embodiments, and modifications may be made without departing from the scope of the present invention. Although particular combinations of features are recited in the claims and / or disclosed herein, these combinations are not intended to limit the possible implementations of the present invention. Indeed, many of these features can be combined in ways not specifically recited in the claims and / or disclosed herein. For example, although each dependent claim listed in the claims may depend directly on only one other claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claims.

Claims

1. A device comprising one or more processors, the one or more processors: establishing a connection from the gateway to the connectivity resource; identifying a source application at the gateway; projecting content received from the source application from the gateway to the connected resource; receiving user input information from the connection resource at the gateway; and applying the received user input information to the source application. device.

2. The step of identifying the source application at the gateway includes: receiving at the gateway at least one screen image from the source application; receiving a set of application matching rules, each associated with a particular source application; applying each application matching rule in the set of application matching rules to the at least one screen image to generate a set of matching scores; identifying the source application by selecting the particular source application associated with the application matching rule that produces the highest matching score among the set of matching scores; The device of claim 1 .

3. The device of claim 2 , wherein the application matching rules are received from an external gateway manager service.

4. The one or more processors further determining that parallel control paths from said connection resource are available; sending input configuration parameters from said gateway to said connection resource defining an input mode; configured to run applying the received user input information to the source application, receiving absolute movement event information associated with a user input event; converting the received absolute movement event information into relative movement event information based at least in part on the input mode; receiving the relative movement event information via the parallel control paths; The device of claim 1 .

5. 5. The device of claim 4, wherein applying the received user input information to the source application further comprises receiving, via the parallel control paths, a set of docking commands including relative movement events that exceed a dimension of a user device interface associated with the gateway, and docking a cursor by applying the received set of docking commands to the user device interface.

6. The step of projecting the content received from the source application includes: identifying an operational state associated with the connection resource; identifying a set of state-related constraints associated with the operational state; applying the set of state-related constraints to the content to be projected. The device of claim 1 .

7. 7. The device of claim 6, wherein identifying a motion state associated with the connected resource comprises determining that movement of the connected resource exceeds a specified velocity threshold, and applying the set of state-related constraints to the content to be projected comprises preventing projection of at least one user interface element.

8. 1. A non-transitory computer-readable medium storing a plurality of processor-executable instructions, the plurality of processor-executable instructions comprising: establishing a connection from the gateway to the connectivity resource; identifying a source application at the gateway; projecting content received from the source application from the gateway to the connected resource; receiving user input information from the connection resource at the gateway; applying the received user input information to the source application; Non-transitory computer-readable medium.

9. The step of identifying the source application at the gateway includes: receiving at the gateway at least one screen image from the source application; receiving a set of application matching rules, each associated with a particular source application; applying each application matching rule in the set of application matching rules to the at least one screen image to generate a set of matching scores; identifying the source application by selecting the particular source application associated with the application matching rule that produces the highest matching score among the set of matching scores; The non-transitory computer-readable medium of claim 8.

10. The non-transitory computer-readable medium of claim 9 , wherein the application matching rules are received from an external gateway manager service.

11. determining that parallel control paths from said connection resource are available; sending input configuration parameters from the gateway to the connection resource defining an input mode; applying the received user input information to the source application, receiving absolute movement event information associated with a user input event; converting the received absolute movement event information into relative movement event information based at least in part on the input mode; receiving the relative movement event information via the parallel control paths; The non-transitory computer-readable medium of claim 8.

12. 12. The non-transitory computer-readable medium of claim 11, wherein applying the received user input information to the source application further comprises receiving, via the parallel control paths, a set of docking commands including relative movement events that exceed a dimension of a user device interface associated with the gateway, and docking a cursor by applying the received set of docking commands to the user device interface.

13. The step of projecting the content received from the source application includes: identifying an operational state associated with the connection resource; identifying a set of state-related constraints associated with the operational state; applying the set of state-related constraints to the content to be projected. The non-transitory computer-readable medium of claim 8.

14. 14. The non-transitory computer-readable medium of claim 13, wherein identifying a motion state associated with the connected resource comprises determining that movement of the connected resource exceeds a specified velocity threshold, and applying the set of state-related constraints to the content to be projected comprises preventing projection of at least one user interface element.

15. establishing a connection from the gateway to the connectivity resource; identifying a source application at the gateway; projecting content received from the source application from the gateway to the connected resource; receiving user input information from the connection resource at the gateway; applying the received user input information to the source application; method.

16. In the gateway, the step of identifying the source application comprises: receiving at the gateway at least one screen image from the source application; receiving a set of application matching rules, each associated with a particular source application; applying each application matching rule in the set of application matching rules to the at least one screen image to generate a set of matching scores; identifying the source application by selecting the particular source application associated with the application matching rule that produces the highest matching score among the set of matching scores; 16. The method of claim 15.

17. The method of claim 16 , wherein the application matching rules are received from an external gateway manager service.

18. determining that parallel control paths from said connection resource are available; sending input configuration parameters from the gateway to the connection resource that define an input mode; applying the received user input information to the source application, receiving absolute movement event information associated with a user input event; converting the received absolute movement event information into relative movement event information based at least in part on the input mode; receiving the relative movement event information via the parallel control paths; 16. The method of claim 15.

19. 20. The method of claim 18, wherein applying the received user input information to the source application further comprises receiving, via the parallel control paths, a set of docking commands including relative movement events that exceed a dimension of a user device interface associated with the gateway, and docking a cursor by applying the received set of docking commands to the user device interface.

20. The step of projecting the content received from the source application includes: identifying an operational state associated with the connection resource; identifying a set of state-related constraints associated with the operational state; applying the set of state-related constraints to the content to be projected; The step of identifying an operational state associated with the connection resource comprises: determining that the movement of the connection resource exceeds a specified velocity threshold; applying the set of state-related constraints to the content to be projected includes preventing projection of at least one user interface element.

16. The method of claim 15.

Images