Device, method and medium for projecting, controlling and managing applications

Abstract

A gateway is described that projects user device application content (e.g., streaming video) to connected resources. A user device may be an electronic device such as a smartphone, tablet, wearable device, etc. A connected resource may be a device such as a vehicle head unit. Application content may be associated with any application that can be executed by a user device, such as a media streaming service, a game, etc. User input received at a connected resource may be applied on a user device via the gateway by appropriately converting the received user input data and emulating the received input on the user device. The gateway may identify the application so that the application content may be appropriately filtered or otherwise processed or managed. Machine learning may be used to train models related to application or content identification, rules related to operational restrictions, and / or other such elements.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 62 / 954,374, filed on December 27, 2019, and U.S. Patent Publication No. 2014 / 0156734, published on June 5, 2014, which are incorporated herein by reference. Background Art

[0003] User devices such as smartphones and tablets are ubiquitous. Similarly, connected resources such as in-vehicle systems are also widely available. Therefore, it is necessary to allow interaction between user devices and connected resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The novel features of the present disclosure are set forth in the appended claims.For purposes of illustration, however, several embodiments are shown in the following drawings.

[0005] Figure 1 An example overview of one or more embodiments described herein is shown, in which an application is projected from a user device to a connected resource through a gateway;

[0006] Figure 2 An example overview of one or more embodiments described herein is shown, in which user interface events are received at a connection resource via a gateway and applied to a user device;

[0007] Figure 3 An example overview of one or more embodiments described herein is shown, in which a gateway is used to analyze, identify, and filter applications;

[0008] Figure 4 An example image analysis technique of one or more embodiments described herein is shown, wherein adjacent image portions are compared to identify matching blocks;

[0009] Figure 5 An example environment is shown in which one or more embodiments described herein may be implemented;

[0010] Figure 6 Another example environment in which one or more embodiments described herein may be implemented is shown;

[0011] Figure 7 A flowchart illustrating an exemplary process for projecting application content from a user device to a connected resource is shown;

[0012] Figure 8 A flowchart illustrating an exemplary process for identifying an application executed by a user device;

[0013] Figure 9A flow chart illustrating an exemplary process for receiving commands at a connection resource and applying the received commands at a user device is shown;

[0014] Figure 10 A flow chart illustrating an exemplary process for filtering application content based on an operating mode of a connected resource;

[0015] Figure 11 A flowchart illustrating an exemplary process for applying machine learning to generate or update an application identification model; and

[0016] Figure 12 A schematic block diagram of one or more exemplary devices for implementing various embodiments is shown. DETAILED DESCRIPTION

[0017] The following describes in detail presently contemplated modes of carrying out the exemplary embodiments. This description should not be taken in a limiting sense, but is merely for the purpose of illustrating the general principles of some embodiments, since the scope of the disclosure is best defined by the appended claims.

[0018] Various features are described below, which can be used independently of each other or in combination with other features. In summary, some embodiments generally provide a gateway for projecting user device application content (e.g., streaming video) to a connected resource. The user device can be an electronic device such as a smartphone, tablet, wearable device, etc. The connected resource can be an electronic device, system, or component, such as a vehicle head unit. The application content can be associated with any application that can be executed by the user device, such as a media streaming service, a game, etc.

[0019] User input received at a connected resource may be applied on a user device via the gateway. Such user input may include input received via, for example, a touch screen or other user interface (UI) element of the connected resource. The user input may be applied on the user device by appropriately converting the received user input data and emulating the received input at the user device (and user device application).

[0020] Applications can be automatically identified by the gateway so that application content can be filtered or otherwise processed or managed appropriately. For example, video streaming applications can be identified so that the provision of video content can be restricted during vehicle operation. This application identification can utilize various identification algorithms, such as color bin matching.

[0021] In some embodiments, machine learning may be used to train models related to application or content identification, rules related to operational restrictions, and / or other such elements that may be used to implement various processes or algorithms related to application content projection, control, management, analysis, and the like.

[0022] Figure 1 1 shows an example overview of one or more embodiments described herein, in which a gateway 100 is used to project a source application from a user device 110 to a connection resource 120. As shown, the user device 110 can project the source application to the connection resource 120 through the gateway 100. In addition, the gateway 100 can receive UI events from the connection resource 120 and apply the UI events to the source application running on the user device 110.

[0023] The gateway 100 may be implemented at least in part at a user device 110 or a connection resource 120 and / or other appropriate resources (e.g., a cloud-based server or other network-accessible resource). The user device 110 may be a smartphone, a tablet, a wearable device, and / or other appropriate devices or components capable of executing instructions and / or communicating with other devices via one or more wired or wireless channels. The connection resource 120 may be an in-vehicle head unit, an entertainment system or device, and / or other appropriate resources capable of providing content and / or processing UI data. The connection resource 120 may typically include a display screen.

[0024] In this example, user device 110 may be executing a navigation application. The navigation application may be backed up at connected resource 120 via gateway 100. User device 110 may communicate with connected device 120 via a local wireless channel, such as Bluetooth or Wi-Fi. Gateway 100 may receive application content from the navigation application and provide the application content to connected resource 120 in an appropriate format, such as by converting the application content into a bitmap image or vector data.

[0025] The application content in this example may include image data, such as map data, route indicators, and the like. In addition, the application content may include various UI elements, such as buttons, that may be provided along with the image or other graphical data. In addition, the connection resources 120 may include, provide, or otherwise implement various UI features related to the source application. For example, the display may be re-centered to a specified point or area based on a single tap identified at that location. As another example, the display may be enlarged or reduced by changing the distance between two touch points (i.e., "pinch" zoom). As another example, a double-click may be associated with selecting a navigation destination.

[0026] In this example, tap and drag events can be detected at the connected device 120, and the relevant data can be sent to the user device 110 via the gateway 110 for analysis and potential implementation at the source application. The tap and drag events can cause the displayed map area to be updated, thereby moving the selected point to the new location. The source application can update the output display based on the received UI event data and provide the updated display (reflecting the new location of the selected point) to the connected resource 120 via the gateway 100.

[0027] Figure 2 An example overview of one or more embodiments described herein is shown, in which user interface events are received at a connection resource 120 via a gateway 100 and applied to a user device 110. The user device 110 and the connection resource 120 can be communicatively coupled via a secure channel, such as a Bluetooth or Wi-Fi link. Communication between the resources can include and / or utilize various messaging protocols. The communication channel can utilize encryption and / or other security features.

[0028] In this example, user device 110 includes a display screen 205, such as a touch screen display. User device 205 may include various other UI elements, such as buttons, a keypad, a mouse, a joystick, and the like. Display screen 205 may be associated with one or more target action areas 210. In this example, target action areas 210 may be associated with all or substantially all of the surface of display screen 205. Target action areas 210 may include areas (and / or other UI elements) on display screen 205 that are associated with user input actions (or "events"). Target action areas 210 may utilize a coordinate system associated with display screen 205.

[0029] The connection resources 120 may include a display screen 215, such as a touch screen display. The connection resources 120 may include various other UI elements, such as buttons, a keypad, a joystick, a mouse, and the like.

[0030] The gateway 100 can convert various UI features associated with the user device 110 so that similar functionality can be provided using UI features associated with the connection resource 120. In this example, the display screen 215 includes a projection client area 220, one or more source action areas 225, a projection target area 230, one or more ignored input areas 235, and one or more gateway input areas 240. Different embodiments may include different types of areas (and / or other UI features) in various different arrangements different from those shown. For example, the shape of the area may be different from a rectangle (e.g., a circle, an ellipse, a polygon, etc.). As another example, various single areas may be divided into multiple areas, or multiple areas may be combined into a single area.

[0031] The projected client area 220 may define one or more regions where events, such as "point" events (e.g., touch, click, etc.), should be sent to the connected user device 110. The projected client area 220 may be represented as a rectangle and specified in the coordinate system of the display screen 215. Each other region associated with the connected resource 0120 may be described in the coordinate system of the projected client area 220 and may be described relative to the coordinate system of the projected client area 220. The simplest shape for a region is a rectangle, but various other shapes may be used and / or supported.

[0032] Each source action region 225 may be a region associated with the projected client region 220 that defines whether event data should be identified as an external UI event or a gateway user input event (or ignored). In some embodiments, any event outside the source action region 225 and inside the projected client region 220 may be classified (and / or applied) as a gateway UI event. The source action region 225 may include a primary action region and / or may define an ordered list of additional region items that exclude and / or include a set of sub-regions.

[0033] The projected target area 230 may correspond to the target action area 210, or other area associated with the display screen 205 projected onto the connected resource 120. The projected target area 230 may be described in and relative to the coordinate system of the projected client area 220. The image captured by the user device 110 may be stretched or shrunk to fit the size of the projected target area 230.

[0034] Each ignore input region 235 may be an area where a UI event may be ignored. Input received via an ignore input region 235 may not be used to identify an external or gateway input event. In some embodiments, a UI event associated with an ignore input region 235 may be associated with a notification.

[0035] Each gateway input area 240 may be associated with an event that may be identified as a gateway user input event or a user input event associated with management of the gateway 100 and / or any related resources.

[0036] Different embodiments may include various different types of areas based on the role of each area (and associated events). For example, some embodiments may include one or more external input areas, where any events associated with these areas may be identified as "external user input" events.

[0037] If the regions overlap, each touch event can be evaluated to determine which region the touch event should be associated with. For example, region matching can be based on order, where the most recently touched region may take precedence over previously touched regions.

[0038] As shown, a UI event 245 (e.g., associated with a touch or "click" event) may be received at the projected target area 230 and simulated as if a click were applied at a corresponding click point 250 of the target action area 210. The simulated UI event 250 may be recognized and applied by (or in accordance with) a source application running on the user device 110. For example, a click at location 250 may cause a displayed map area to be updated (e.g., by placing the click location in the center of the display area) when using a navigation application.

[0039] In this example, the gateway 100 may include, store, or utilize various display attribute sets 255 and / or input ranges 260. Such display attributes 255 may include, for example, the size of a user device and / or a connected resource display screen. The input ranges 260 may include various definitions associated with various region shapes, sizes, and / or coordinates. For example, the input ranges 260 may specify the location and size of regions 0 220-240 and region 210.

[0040] The gateway 100 may utilize various other data elements, such as a list of input event types, parameters or attributes associated with the events, thresholds or other criteria for identifying events, and / or other relevant data.

[0041] In some embodiments, user input to the connection resource 120 may be initiated after a connection is established between the user device 110 and the connection resource 120 via the gateway 100. The connection resource 120 may send an input capabilities message indicating supported input features and associated capabilities. The gateway 100 may respond with a list of available input features for the user device 110 in the input capabilities message. The gateway 100 may send a configuration input message to the connection resource 120 (and / or associated client application) that defines various input areas (or other features) and associated usage profiles. The gateway 100 may also define a projection target area 230. The connection resource 120 (and / or associated client application) may store the provided input areas. If any external user input areas have been defined or identified, the connection resource 120 may send such definitions to the device control module of the connection resource 120. The connection resource 120 may respond to the gateway 100 with a configuration input message that may include a device identifier (ID) for each defined input area.

[0042] The gateway 100 can reconfigure the input configuration at any time by resending the Configure Input message. After configuration, the gateway 100 can enable input operations for one or more input zones. The gateway 100 can send a Control Input message that specifies the zone IDs of the input zones to be enabled and the zone IDs of the input zones to be disabled. Multiple messages can be sent at any time to enable and / or disable one or more input zones.

[0043] Figure 3 An example overview of one or more embodiments described herein is shown in which applications are analyzed, identified, and filtered using gateway 100. As shown, user device 110 may be associated with navigation application 310, one or more device sensor interfaces 320, and multimedia application 330.

[0044] Device sensors 320 may include various position sensors (e.g., accelerometers, gyroscopes, global positioning system (GPS) components, etc.) and / or other sensors capable of determining operating modes. For example, such device sensors 320 may be used to determine when a vehicle is stationary or moving. User device 110 may also receive sensor data from connection resource 120 or another external resource. For example, user device 110 may receive vehicle speed sensor data, parking brake status, and / or other such sensor data via connection resource 120.

[0045] The gateway 100 may receive device sensor data (and / or other appropriate data) and determine an operating mode associated with the connected resource 120. Operating modes may be associated with various types of connected systems or resources. For example, a display associated with a vehicle head unit installed in a passenger vehicle may be associated with modes such as "stationary" or "in motion." As another example, a display associated with an in-flight entertainment system may be associated with modes such as "slow taxi," "takeoff," "cruise," or "announcement."

[0046] As shown, if the gateway 100 is unable to receive application information from a user device interface (not shown), the gateway 100 may utilize various recognition models 340 (also referred to herein as "matching" models), permissions 350, and / or other appropriate data elements. Such recognition models 340 may include image data and / or other data that can be used to identify applications running on the user device 110. For example, the recognition model 340 may include an image captured from an application startup screen or other appropriately captured application content (e.g., video content, audio content, graphical content, etc.). The recognition model 340 may include a set of evaluation criteria (e.g., a matching threshold, a minimum matching score, or other metrics, etc.).

[0047] Permissions 350 may include instructions for which source application elements are projected to connected resources 120. For example, a video streaming application (or content element) may be disabled while the vehicle is in motion to limit driver distraction. Application permissions 350 may be associated with an application, device type, operating mode, and / or other relevant attributes. For example, permissions 350 for a video streaming application may indicate that video streaming should be disabled for connected resources 120 associated with the front seat vehicle head unit when the vehicle is in motion. Video streaming may not be disabled for connected resources 120 associated with the rear (or passenger) seat vehicle head unit, regardless of whether the vehicle is in motion.

[0048] In a first example, navigation application 310 may be associated with projection 360 of a full application, where all available content is sent to display screen 215 , regardless of operating mode or other attributes.

[0049] In a second example, a multimedia application 330 may be associated with a partial application projection 370, where some content associated with the multimedia application 330 is disabled or blocked via a non-permitted area 380. In this example, the multimedia application 330 may provide audio and video content related to a podcast. In this case, the partial application projection 370 may include providing audio content (and / or other non-video content) regardless of the operating mode. If not restricted by the operating mode permissions (e.g., blocking video content during exercise), a portion of the application content may be associated with a streaming video to be presented in the non-permitted area 380.

[0050] The non-permitted regions 380 (and / or other types of content filtering) can be applied based on predefined application elements (e.g., a video display window associated with an application) and / or automatically detected application elements. For example, the projected content can be analyzed to identify regions of one or more frames associated with the video content (e.g., by comparing differences between frames to detect motion). Such content analysis can include analysis of various distracting (or unwelcome) application content. For example, bright and distracting colors can be dimmed. As another example, loud noises or noises that match various profiles (e.g., car horns, sirens, screeching tires, etc.) can be muted or provided at a reduced volume.

[0051] Figure 4An example image analysis technique for one or more embodiments described herein is shown, in which adjacent image portions or regions are analyzed to identify matching blocks. Gateway 100 can select from a variety of image analysis algorithms based on various appropriate criteria. For example, an algorithm can be selected to best utilize the capabilities of user device 110 while meeting a specified accuracy threshold. Algorithm selection is described in more detail below with reference to process 700.

[0052] The captured image 410 may be received from the active application by capturing the projected screen and / or other information associated with the application. A library of similar images associated with various user device applications may be maintained by the gateway 100 and / or other appropriate resources. Image analysis and generation of matching models will be described in more detail with reference to the following process 1100.

[0053] In some embodiments, the gateway 100 can use color profile matching to analyze an image and match a captured image (or portion thereof) to a previously captured application screen (or portion thereof). Many or most applications can use a consistent color scheme for various screens associated with or generated by the application. Thus, color profile matching can provide effective and efficient results in identifying applications.

[0054] During matching model training, a color profile can be created for each screen of the source application (or "source screen"). Similarly, a color profile can be calculated for the currently captured image 410 to determine whether the color profile of the captured image 410 matches the color profile of any source screen. A match score or other calculated metric can be compared to a match threshold to determine whether the captured image 410 matches the captured application screen. Such a color profile matching algorithm can require limited processor and memory usage while executing relatively quickly.

[0055] A color profile can be calculated for the current image 410. The color profile can be calculated by evaluating image portions (e.g., individual pixels, groups of pixels, or "blobs"), etc. The color profile can include any number of color "bins" and a portion count (e.g., number of pixels) associated with each bin. Each pixel (or other image portion) can be evaluated, and the pixel color can be converted to a color bin. The number of bits per color channel can be limited (e.g., using a color bin channel bits parameter) to more efficiently analyze and store the profile data. The color channels of each pixel (or other image portion) can be combined into a single value (e.g., using conventional bitwise operations) so that, for example, each pixel or other image portion is associated with a single color bin value, which can serve as a "key" in a color profile "dictionary." If the dictionary already contains the key, the count associated with the key can be incremented. If the dictionary does not contain the key, a new entry can be added to the dictionary and the associated count set to one. If the associated count is less than a threshold, the key can be deleted from the dictionary.

[0056] The resulting color profile can be compared to various stored color profiles in various suitable ways. For example, the color bins associated with the captured image 410 can be compared to the color bins associated with the matching model. The number of additional or additional color bins associated with the captured image 410 relative to the number of color bins associated with the model can be calculated. Similarly, the number of matching color bins between the captured image 410 and the model can be calculated.

[0057] A match score can be calculated based on the number of matching color bins and the number of additional color bins. In some embodiments, the match score can be a probability expressed as a number ranging from zero to one. The number of matching color bins can be compared to a minimum match threshold number. If the number of matching color bins is less than the minimum match threshold, the match score can be set to zero. If the number of additional color bins is greater than the maximum match threshold, the match score can be set to zero. If the number of additional color bins is less than or equal to the minimum additional match threshold, the match score can be set to one. If the number of matching color bins is greater than the minimum match threshold, the number of additional color bins is less than the maximum match threshold, and the number of additional color bins is greater than the minimum additional match threshold, the match score can be calculated by dividing the number of matching color bins by the total number of image color bins. In some embodiments, the match score can include color bin weighting, where each color bin is associated with a count or other weighting factor. If the calculated match score exceeds the match threshold, the matching application can be identified by the gateway 100 and associated with the calculated score.

[0058] This pixel-type (or other area or portion-type) color matching algorithm is suitable for application screens with solid color widgets (such as buttons, background rectangles, circles, etc.). However, some application screens may display photographic images or other graphic features that are unlikely to have continuous areas of matching colors due to, for example, slightly different gradients of light or shadow. For example, a navigation application may display a "street view" of a destination. As another example, a restaurant search application may display thumbnails of restaurants in a list of results. In this case, if a single color is calculated, the photographic (or other high-density) image will produce a large number of colors that are useless for matching calculations. To overcome such limitations, the gateway 100 can utilize or implement a runtime filter when calculating colors to only count the colors of areas with multiple sub-elements (also called "spots") of matching colors.

[0059] exist Figure 4 In the example of , rather than analyzing each portion independently, a "blob" analysis is utilized by dividing the image 410 into portions 420 (e.g., pixels, sets of pixels, shape domains, etc.). In this example, each portion 420 can represent a pixel. For such an analysis, color bins of adjacent pixels can be evaluated. If certain matching criteria are met (e.g., a minimum number of adjacent matching elements), the color bin associated with the pixel or portion being evaluated can be added to a dictionary (or a count of matching color bins can be increased). If the matching criteria are not met, the pixel or other portion may be skipped (e.g., the color bin may not be added to the dictionary and the count may not be increased).

[0060] In this example, each portion 420 can be associated with a single pixel (center pixel "A"). As shown in the expanded view 430, multiple neighboring elements (pixels "B"-"I" in this example) can be evaluated to determine whether the pixel under evaluation (pixel A) is included in the color blob. Each pixel in the captured image 410 can be evaluated in a similar manner.

[0061] Each pixel can be evaluated relative to its neighboring pixels, and if there is a minimum number of matching neighboring pixels with the center pixel (e.g., determined by having the same color bin), the center pixel is identified as part of the blob and added to the color profile dictionary. Otherwise, the center pixel is skipped (e.g., by not adding the associated color bin to the color profile dictionary or not incrementing a count associated with the color bin).

[0062] The simplest neighbor pixel evaluation window is three by three, as shown in the figure. In some embodiments, at least two neighboring pixels must have the same color bin as the center pixel for the blob to be identified. For example, in this example, shadow pixels A, B, and C may have matching color bins, so pixel A can be identified as part of the blob and added to the color profile dictionary. In some embodiments, a match can be evaluated based on adjacent neighbor pixels. For example, in this example, a match can be identified because adjacent neighbor pixels B and C match center pixel A, while if non-adjacent neighbor pixels B and G match center pixel A, pixel A may not be identified as part of the blob.

[0063] Different embodiments may utilize different domains, shapes, matching criteria, etc. to define or evaluate color blobs. For example, using a smaller evaluation window may process the image faster, but may also capture smaller (and therefore more) blobs. In a three by three example evaluation window, the smallest blob may comprise three pixels, as described above. To filter out smaller blobs, a larger evaluation window may be used. For example, in a five by five evaluation window, the smallest blob may be a nine-pixel (three by three) square. The size of the evaluation window may be specified in the matching model or may be based on some relevant criteria (e.g., image size, device processing power, etc.).

[0064] Matching model data related to blob matching may include elements such as color profile, color bin counting method, color threshold, number of color bin channel bits, additional color bin minimum threshold, additional color bin maximum threshold, match score minimum threshold, blob window size or area definition, blob match count and / or other related parameters.

[0065] A color profile can include information such as color bin identifiers, counts, etc., as described above. The color bin counting method can indicate the type of color bin counting to be used, such as all pixels, spots only, etc. The color threshold can specify the minimum number of pixels that should match a given color bin in order to be included in the match score calculation. This approach can filter out colors that have only a few pixels and are not an important factor in classifying the image. The number of color bin channel bits (or other limiting factor) can specify the number of bits used in each color channel of the image that should be used in the color bin. For example, for the typical "RGB888" (red channel, green channel, and blue channel, 8 bits per channel) format, the algorithm can reduce color variation by 50% by using 7 bits for each channel instead of 8 bits.

[0066] The additional color bin minimum threshold can specify the number of additional color bins below which an image is considered a perfect match (e.g., by assigning a match score of one or one hundred percent). The additional color bin maximum threshold can specify the number of additional color bins above which an image is considered a mismatch (e.g., by assigning a match score of zero). The match score minimum threshold can specify the minimum match score above which a match score is considered to indicate a match (e.g., the match score minimum threshold can be eight tenths or eighty percent in some embodiments). When using a blob-based algorithm (e.g., 9 pixels, 25 pixels, etc.), the blob window size can specify the number of pixels (or other fraction) to be evaluated. The blob window can also be associated with a data element such as a shape-defining element (e.g., indicating a square, a circle, etc.). The blob match count can specify the minimum number of matching elements to be considered a match. For example, in the nine-pixel region described above, at least two other pixels must match the center pixel, so the blob match count might represent three total pixels. Some embodiments may include other parameters associated with blob matching, such as the correlation between matching pixels (e.g., in the example above, two adjacent pixels must match the center pixel).

[0067] Another example algorithm that can be used to match a screen capture image to an application is to search for specific image components in a given area of ​​the captured image (also known as "image template matching"). This type of image template matching may be effective when applied to applications that include unique logos or original artwork icons on their screens, as many do. Other image elements, such as standard or generic icons (e.g., magnifying categories, settings icons, etc.) may be different between applications and therefore useful for calculating a matching score. The matching model used for image template matching may include one or more image artifacts to search for. The image template matching algorithm can scan the input image for any image artifacts and, if an image artifact is detected, return a label or identifier associated with the matching model associated with the image artifact. Such scanning can be performed by evaluating portions of the captured image 410 (e.g., pixel groups 420) and determining whether any scanned portion matches an image artifact to at least a minimum matching threshold.

[0068] The image template matching algorithm can retrieve a template image that matches the screen density of the user device 110 (and / or based on some other appropriate criteria). For each template image, the image template matching algorithm can use various difference-based calculations to compare the template image with the current input image, and can search for the most likely location of the current input image for a given template and generate a matching score. Based on the matched model data, template data, captured image data, and / or other relevant factors, a region of interest (ROI) can be used to narrow the search area. If the calculated score meets or exceeds a specified threshold, the image templates are considered to match, and an identifier of the matching application is returned.

[0069] Matching model data related to image template matching may include template images, template masks, comparison types, and / or various thresholds. Template images may include a list of associated images for comparison with the input image. The images may include (or be associated with) information such as screen density (e.g., dots per inch or "resolution") so that the images can be scaled correctly for comparison. Each template image may include an alpha channel that defines areas that should be excluded from the evaluation. Template masks may include a list of images that define bitmap masks to be applied to the template images. Such bitmap masks may indicate which parts of the template image are used for matching comparison. The comparison type may indicate the type of template matching operation to be used. For example, various matrix transposition operations may be used, such as SQDIFF, SQDIFF_NORMED, CCORR, etc. Thresholds may include values ​​such as thresholds or scores, above which the template matching operation should be considered a match.

[0070] Image template matching typically requires manual creation of a matching model. For example, an application screen can be evaluated to identify common templates on the screen. Template images can be extracted from the original screen and configured as a matching model. In some embodiments, such a model can be automatically generated by obtaining the source application in a distributed form (e.g., Android Package Kit (APK) for Android, iOS Application Archive (IPA) for iOS, etc.) and extracting icon resources from the source application. Screen images captured from the application (also called "training" images) can then be scanned to see what any extracted icons look like. The scan can use scoring and / or matching algorithms similar to those described above and below. The most commonly recognized icons can be automatically added to the matching model.

[0071] The generation and updating of the matching model will be described in more detail with reference to the following process 1100 .

[0072] Figure 5An example environment 500 is shown in which one or more embodiments described herein may be implemented. As shown, the environment 500 may include a user device 110 and a connection resource 120. The environment 500 may include various other devices (e.g., cloud-based devices), network communication channels, and / or other resources.

[0073] The user device 110 may be a consumer electronic device capable of executing instructions and / or otherwise processing data, such as a smartphone (e.g., iOS, Android, etc.), a smartwatch, smart glasses, other smart devices, a tablet, etc. The user device 110 may at least partially include and / or implement the gateway 100. The gateway 100 may capture and manage screen, audio, and / or other content from the source application 545 and may project the content to the connection resource 120. The gateway 100 may at least partially direct the operation of the source application 545 (e.g., based on UI data received from the connection resource 120). Each source application 545-550 may be a third-party application that the user wants to use with the connection resource 120.

[0074] Connectivity resource 120 can be any electronic system, device, or component that can be communicatively coupled to user device 110 and that is capable of utilizing or otherwise interacting with (or at least providing certain functionality to) applications running on user device 110. For example, connectivity resource 120 can be an in-vehicle system (e.g., a head unit, infotainment system, rear-seat display, instrument cluster, smart mirror, etc.), a medical system, a smart TV, a smart speaker, a fitness device, a smart home device (e.g., an Internet of Things (IoT) device), smart glasses, smart appliances, etc. Connectivity resource 120 can execute a dedicated process or application, which can be referred to as a "client" application or the like. In some embodiments, connectivity resource 120 can include or provide various interfaces or connectivity features that can allow user device 110 to send data to and receive data from connectivity resource 120. In some embodiments, such interfaces or features can be implemented as operating system (OS) components or other non-dedicated resources. For example, connectivity resource 120 can provide an application programming interface (API) or similar feature to allow video content to be provided to connectivity resource 120.

[0075] As shown, gateway 100 may include an application capture element 505, an application control element 510, an application manager 515, a projection engine 520, a recognition engine 525, and a repository 530. Gateway 100 may be able to access various user device features, such as an application repository 540, which may include various source applications 545-550 that can be executed by user device 110. In this example, source application 550 includes an optional development toolkit 555. In some cases, gateway 100 may have access to an OS interface 560 that may provide information such as active application identifiers. Connectivity resources 120 may include a projection client 570 and / or an optional user device interface 575.

[0076] When running the source application 545, the application capture element 505 can capture the screen or other output of the user device 110. The application capture element 5050 can send the captured data to the projection engine 520 or the recognition engine 525 for further processing.

[0077] Application control element 510 can control or otherwise interact with source application 545. Application control element 510 can launch an application provided by application manager 515, simulate user events, perform screen transitions, cause execution of specific functions in an application, and / or otherwise interact with source application 545.

[0078] Application manager 515 can communicate with external resources, such as cloud-based servers, to manage elements associated with application detection and / or operation. Application manager 515 can locally store and / or manage application catalog 590 (or portions thereof) and can provide application information to other modules. Application manager can execute various application detection algorithms.

[0079] The projection engine 520 can implement screen projection to the connection resource 120. The projection engine 520 can include or utilize various screen projection engines, such as WebLink, CarPlay, AndroidAuto, SDL, etc.

[0080] The recognition engine 525 can utilize image recognition and / or other recognition algorithms to detect the active source application 545 and / or the active screen to be projected. On more stringent user device systems (e.g., iOS), the OS interface 560 may not be available or accessible (e.g., a system API for detecting active applications may not be provided), and the source application 545 may not include a development kit or other resources that may allow such interaction. In this case, image recognition can be implemented by the gateway 100.

[0081] The recognition engine 525 can load available matching models 580 associated with the application catalog 590 from the repository 530. The recognition engine 525 can classify each received image into an application ID (and / or screen ID) and return the classification information to the application manager 515. This classification can utilize various image analysis algorithms described above and below. If the image cannot be recognized, indicating that the application is unknown, the recognition engine 525 can report the unrecognized image status to the application manager 515.

[0082] The recognition engine 525 can implement one or more of the image classification algorithms described herein based on various appropriate criteria (e.g., capabilities of the user device 110 , capabilities of the connected resources 120 , user preferences, application configuration settings, etc.) The matching model 585 can indicate which algorithm, parameter values ​​or other settings to use, etc.

[0083] The recognition engine 525 can utilize the hardware-accelerated machine learning (ML) capabilities of the user device 110 whenever possible. For example, for iOS, Core ML can be used or implemented. For Android, an on-device ML toolkit may provide such capabilities. Popular third-party libraries, such as the Open Source Computer Vision Library (OpenCV), can be used for certain image classification and / or analysis operations.

[0084] As shown, the application catalog 590 may include or be associated with matching models 580 for various supported applications. In some embodiments, a generic model may be stored (e.g., using a data-only application record) in or associated with the application catalog 590. When the application catalog is initially downloaded (or built) or updated based on interaction with a server or similar resource (e.g., a gateway manager described below), the recognition engine 525 may load available matching models 580. Matching models 580 may be processed by generating an empty list of rules ("rule list"). For each application in the application catalog 590, the recognition engine 525 may determine whether a matching model 580 is associated with the application entry. This determination may be made in various suitable ways, such as by analyzing the application entry (and / or associated data elements) to identify a list of associated matching models 580.

[0085] If an associated model is identified, the recognition engine 525 can retrieve the root rule object from the model and add the rule to the rule list at the specified position. For example, if the new rule does not have a priority flag or attribute set, the rule can be added to the end of the list. If the new rule does have a priority attribute or flag set, the recognition engine 525 can search the rule list from the beginning and insert the new rule before the first rule with a lower priority value than the current rule (or the first rule without a priority value). After all applications have been processed, the rule list can therefore include a list of all root rule objects from all matching models 580 associated with the application directory 590, sorted by priority (and the order in which they are listed in the application directory). The rule list can be processed sequentially during the various recognition or analysis algorithms or operations described herein.

[0086] Once the model is loaded and the rule list is generated, the recognition engine 525 can process the currently captured screen and return the detected application ID, screen ID, and / or other appropriate information. In some embodiments, the recognition engine 525 can process each captured screen. However, this approach may cause an excessive load on the system resources of the user device 110. Therefore, in some embodiments, the recognition engine 525 can use various gating features to determine (or indicate) whether a given screen image should be processed.

[0087] Gating features may include, for example, time-based gating, where the determination of whether to analyze the current screen is based on a timer. For example, one screen may be analyzed per second. The gating interval duration may be adaptive based on parameters such as the processing load and / or battery consumption of user device 110. For example, if the processing load is high, the gating interval duration may be increased to reduce the processing load.

[0088] Another example gating feature can be change gating, where the determination of whether to analyze the current screen is based on a calculated difference between a previously detected screen image and the current screen image. If the calculated difference exceeds a specified threshold, the current screen image can be analyzed. This difference comparison can be performed in various suitable ways, including direct image comparison and / or video encoder signal analysis.

[0089] Direct image comparison may include comparison techniques such as color bin-based analysis (such as color matching as described in reference), pixel-by-pixel comparison (e.g., when the number of changed pixels exceeds a threshold or ratio), perceptual hashing algorithms (e.g., algorithms that use hashes to generate fragments or fingerprints of various media forms), threshold-based scene detection (e.g., the intensity and / or brightness of the current screen can be compared to a specified threshold, and analysis can be triggered when the intensity and / or brightness exceeds the threshold), and / or other appropriate direct image comparison algorithms.

[0090] The video encoder signal comparison may include using the projection engine 520 (and / or associated video encoder) to perform scene detection based on the captured screen sent to the projection client 570. For example, the H264 format may use different types of frames. Key frames (or "I" frames) may contain complete scenes, while "P" and "B" frames may only contain changes. If there is no change in the scene, the P and B frames will be very small, and the recognition engine 525 may use this type of information to determine whether to process the current screen. In some cases, the video encoder may implement internal scene detection logic (e.g., for better frame encoding) that provides a separate signal to the recognition engine 525 when a scene change is detected.

[0091] In some embodiments, a hybrid image recognition approach can be used. For example, screen changes can be detected and screen recognition can be performed at some specified interval, regardless of whether any calculated differences exceed a threshold or otherwise indicate a change.

[0092] If the recognition engine 525 determines that the screen image should be analyzed, the recognition engine 525 can process the rule for each rule object in the rule list by applying any image operations described in the rule and storing the results. If the rule object has sub-rules, each sub-rule can be recursively processed and stored or otherwise associated with the current screen image. If the rule object has no sub-rules, the recognition engine 525 can determine the rule type and execute the algorithm associated with the rule (and pass any algorithm-specific data).

[0093] If the algorithm indicates that there is a match, the recognition engine 525 can return the associated tag of the matching rule, extract the application ID (and screen ID, if any), and send the application ID (and screen ID, if any) to the application manager 515. In some embodiments, the recognition engine 525 can be configured to analyze the image using all available rules and select a match based on the rule that returns the highest match confidence score (e.g., calculated in various suitable ways).

[0094] If the algorithm indicates that there is no match (or there is a match with a different application), the recognition engine 525 can notify the application manager 515. The application manager 515 can be configured to perform a hysteresis function rather than assuming that the current application is unknown based on one non-detection (or a limited number of non-detections). This approach can help resolve recognition errors and avoid poor user experience, such as flickering when a given screen image is temporarily unrecognizable or misidentified.

[0095] The active source application screen (and / or other application output) can be captured by the application capture element 505 and received by the recognition engine 525 for analysis. In the case of capture through an external display channel, the captured screen image can be received from the connection resource 120 and passed from the projection engine 520 to the recognition engine 525.

[0096] The recognition engine 525 can be configured with a matching model 580 (also referred to as a "recognition" model), which can be downloaded from a server along with an application catalog 590 and / or other appropriate information. The recognition engine 525 can evaluate the model and report the application ID, the screen ID of the detected application (or "null" if the current application is unknown), and / or other relevant information to the application manager 515.

[0097] The repository 530 may include a memory or storage that may include a matching model 580, a launcher screen 585, an optional application catalog 590, and / or other appropriate elements. Each launcher screen 585 may be, include, or otherwise implement an application screen that can be projected onto the connection resource 570. The launcher screen 585 may include available applications (as shown in the application catalog 590) and may allow the end user to launch or otherwise interact with the available applications. The application catalog 590 may be a document or other data structure managed by a backend server, or other appropriate resource that includes information such as a list of applications that the end user should have access to through the screen projection system, associated restrictions (if any), a screen recognition model (or a reference thereto), etc. Some embodiments may include an application management backend device (not shown) that can manage the application catalog 590, the matching model 580, and / or other elements used by the gateway 100.

[0098] Each matching model 580 can be represented by a file storing a hierarchical object structure that describes how to classify an image. The hierarchical structure can be implemented using a standard binary or text file format, such as protocol buffers, JSON, extensible markup language (XML), YAML, etc., or as a proprietary serialized object structure. Each matching model 580 file can be compressed (e.g., using GZip) and / or encrypted. Such files can be digitally signed so that the recognition engine 525 can verify the authenticity and integrity of each file when loading.

[0099] A matching model 580 can be associated with one or more applications and used to identify one or more applications. For each application, a matching model 580 can include and / or be used to identify a screen or state. Some matching models 580 can be used to identify screen modes across multiple applications (e.g., detecting an on-screen keyboard regardless of the active source application).

[0100] Each matching model 580 may include a version and an optional digital signature, which may use standard cryptographic APIs provided by the user device OS. A matching model 580 may include a rule element, which may be a primary hierarchical object that describes which algorithm to run and which parameters to use. A rule element may include one or more sub-rules.

[0101] The rule element can include various sub-elements, such as a type that can define the type of rule. Depending on the type, various other properties may vary. Some embodiments may use group rules. Group rules may be able to perform certain common operations based on the input image and then pass the results to various sub-rules. The final rule may provide instructions for executing one of the supported image classification algorithms.

[0102] Each rule may include a name element, which can be used for debugging purposes. Each rule may include a priority element, which can define the priority of the rule relative to other rules or rule elements. The recognition engine 525 can use the priority to determine the order in which the various rules are evaluated. If no priority is specified, the order in the model file and / or the order in which the models are loaded can be used to determine the priority.

[0103] Each rule may include a sub-element that includes a list of sub-rules (if any). Sub-rules may be evaluated in the order listed unless they are associated with a specifically defined priority.

[0104] Each rule may include an image operation element, which may specify which operations should be performed on the input image before the image is evaluated according to the rule and / or associated evaluation algorithm. Group rules may be associated with certain image operations that apply to all sub-rules to speed up processing. Various image operations may be supported, such as scaling (e.g., an image may be scaled down or enlarged to a predefined pixel size or relative percentage), cropping (e.g., system bars may be cropped using this operation), color conversion (e.g., converting to grayscale, converting to a different color space, removing bits from color values, etc.), filtering (e.g., using operations such as dilation, attenuation, etc.), and / or other appropriate operations.

[0105] Each rule may include an Image Region of Interest (ROI) element that specifies one or more ROIs for the input image used by the recognition algorithm. The ROIs may be specified as rectangles, polygons, masks, and / or other suitable means.

[0106] Each rule can include a tag. For rules that define a classification operation, the tag field can include the result that the recognition engine 525 should return if the rule matches. In addition, algorithm-specific data may include tag data. Tags can be represented as strings that include information such as application ID and / or screen ID.

[0107] Each rule may include an algorithm-specific data element that, depending on the type of rule, may store parameters passed to the corresponding image classification algorithm.

[0108] Development kit 555 (also referred to as a "Notification" software development kit (SDK)) can enable notification messaging between gateway 100 and source applications 545. Source applications integrated with development kit 555 can provide an enhanced user experience.

[0109] The projection client 570 can receive and play back the projection information stream sent by the projection engine 520. The projection client 570 can display the results on an appropriate display or screen of the connected resource 120. The projection client 570 can capture any user input (button, touch, voice, etc.) and send the captured user input to the gateway 100.

[0110] The user device interface 575 can allow control of the user device 110 outside of the communication path from the projection engine 520 to the projection client 570. For example, the user device interface 575 can include or utilize a human input device (HID), and the application control element 510 can instruct the user device interface 575 to send HID commands to the user device 110.

[0111] The user device 110 and the connection resources 120 (and / or other systems or environment elements described herein) can communicate across various paths or channels. For example, the primary data channel between the user device 110 and the connection resources 120 can utilize connection technologies such as Wi-Fi, Bluetooth, Universal Serial Bus (USB), and can use communication protocols supported by typical user devices 110, such as Android Open Accessory (AOA), External Accessory Protocol (EAP), Transmission Control Protocol over Internet Protocol (TCP / IP), etc. Some embodiments may include communication channels such as external control channels, external display channels, audio channels, etc.

[0112] The external control channel may include, for example, HID input sent via USB or Bluetooth, pointing device input, etc. The external display channel may utilize or provide projection utilities such as AirPlay, MiraCast, HDMI, MHL, ChromeCast, etc., where data may be sent via wireless or wired channels. The audio channel may include audio input and / or audio output data or provide audio input and / or audio output data via channels such as Bluetooth Advanced Audio Distribution Profile (AADP or "A2DP"), USB audio, USB microphone, AirPlay, WiSA (wireless speakers and audio), Digital Living Network Alliance (DLNA), High Definition Multimedia Interface (HDMI), etc.

[0113] The user device 110 and the connection resource 120 may communicate via the primary data channel (and / or other available channels) using the Data Channel Communication Protocol (DCCP) of some embodiments. Messages sent between the gateway 110 and the projection client 570 (and / or other system elements) may utilize or implement DCCP. Messages may be stored and transmitted using binary, text, and / or other suitable representations. For example, the DCCP of some embodiments may utilize serialized "C" structures, string formats such as JavaScript Object Notation (JSON), protocol buffers, and the like. Use of protocol buffers may be preferred because the binary format is scalable and efficient in terms of bandwidth and processing.

[0114] Although various messages or other elements of the DCCP may be described with reference to particular parties, one of ordinary skill in the art will recognize that various other parties may be used to implement the same, similar, and / or complementary messages or elements.

[0115] Regardless of the underlying message format, the messages (and / or related protocols) can be extensible so that new messages can be added to the communication protocol. Older versions of the gateway 100 or projection client 570 may ignore such new messages. The messages and / or protocols can be backward compatible so that newer components can interact with older components by sending messages that are compatible with such older or legacy components. The messages and / or protocols can be forward compatible, for example, so that new fields added to a message will be ignored by older components and only recognized fields will be used.

[0116] Some such messages may follow a request and response format or paradigm, such that one element sends a request and receives a response from another element. Such messages may share a common request identifier or other appropriate attributes that allow the request to be associated with the corresponding response. Each of the messages described above and below can be divided into multiple smaller messages. Similarly, each message can be combined with other messages to form larger messages.

[0117] In some cases, multiple data paths may be available between user device 110 and connection resource 120. For example, a Bluetooth connection may be used initially, a Wi-Fi connection may be established later, and / or a physical connection such as a USB data cable may be added. DCCP may use session information to support such multi-channel arrangements.

[0118] For example, DCCP may include a "Session" message for establishing an initial connection via a first channel (e.g., Bluetooth). This Session message allows endpoints to establish a continuous session across one or more connection channels. This message can be used to determine whether a newly established connection (or "channel") is to the same device as an existing connection or the last known connection. This approach allows peers to seamlessly switch between connection channels. The Session message is used to establish a new session and disconnect all other communication channels with non-matching devices.

[0119] Each session message may include a session token for the component sending the message and various session handshake flags. Such session handshake flags may include an announcement flag indicating whether the message is an announcement (meaning not requesting a new session), a new session flag indicating whether the message is requesting the establishment of a new session, a response flag indicating whether the message is a request message or a response message, and / or other appropriate flags. Session request messages may be sent by projection client 570, and session response messages may be sent by gateway 100.

[0120] As part of the initial setup, session tokens may be exchanged and stored by the gateway 100 and / or the projection client 570. If a second channel is established, the gateway 100 and projection client 570 may attempt to continue the existing session. If successful, the second channel may be used in parallel with or in place of the first channel.

[0121] This approach allows for seamless transitions between channels to accommodate changes in connectivity (e.g., due to user action or selection, determination based on signal strength or other metrics, etc.). Parallel channels can be used for different purposes. For example, a lower-throughput Bluetooth channel can be configured to use a lower bit rate associated with a lower-quality video stream format. If a higher-throughput channel becomes available, the video stream can be switched to the higher-throughput channel and the associated higher-quality data video stream format.

[0122] In order to implement such session-based multi-channel communication, the gateway 100 and the projection client 570 can utilize a set of session tokens. Such session tokens can include a client session token that can be generated by the projection client 570 and sent to the gateway 100 and a gateway session token that can be generated by the gateway 100 and sent to the projection client 570.

[0123] The session token may be generated using a random number generator and attributes (eg, date, time, etc.), connection channel identification information (eg, IP address, Media Access Control (MAC) address, etc.), and / or other appropriate information.

[0124] The session token can be exchanged in various suitable ways. For example, the projection client 570 can send a session message when connected to the gateway 100, and the session message includes a previously stored (if available) or newly generated client session token, a triggered announcement flag, and / or other suitable content.

[0125] The gateway 100 may receive a session message from the shadowing client 570 and compare the received client session token with any active or previously received client session tokens. If the received client session token matches an active or previously received client session token, the gateway 100 may send a session reply message including the gateway session token, a triggered announcement and / or response flag, and / or other appropriate content. If the corresponding stored session token is empty, the gateway 100 may generate and store a gateway session token.

[0126] If the received client session token does not match the active session token, thereby indicating a new connection, the gateway 100 may determine whether a previously active connection exists (e.g., if the previously received session token matches the received client session token). If a previously active or currently active connection is identified, the gateway 100 may stop the previous connection and switch to the newly available connection, where such switching may be based on some specified criteria (e.g., gateway configuration, user confirmation, etc.). If the newly available connection is rejected (e.g., the user selects the option to continue the previous connection), the gateway 100 may send a new session message with an empty session token, a triggered response flag (e.g., indicating that the new connection is rejected), and / or other content. If the newly available connection is accepted, the gateway 100 may clear any stored active client session tokens, generate a new gateway session token, and send a session response message including the new gateway session token, the triggered new session and response flag, and / or other content.

[0127] The projection client 570 may receive a session response message. If the new session flag is set, the projection client 570 may disconnect all other channels or devices until only the current channel is open, store the received session token as the new gateway session token, clear any stored client session tokens, generate and store a new client session token, and, if the response flag is not set, send a new session message with the new session flag set and the newly generated session token. If the new session flag is not set, but the announce flag is set, the projection client 570 may check the received gateway session token and compare it with the stored gateway session token. If the tokens do not match, the projection client 570 may proceed as if the new session flag was set. If the tokens match, no additional action is taken because the new channel is for an existing device.

[0128] If gateway 100 receives a Session message with a new session flag and a previously active connection exists, gateway 100 may switch to the new connection or request confirmation to end the previous connection, depending on the configuration of gateway 100. If the new connection is rejected, gateway 100 may send a New Session message with an empty session token and a response flag, indicating that the new connection is rejected. If the new connection is accepted, gateway 100 may disconnect any existing connection, store the received token as the client session token, generate and store a new gateway session token, and send the new token via a Session message with the new session and response flags.

[0129] Unless the gateway 100 and / or connected device 120 establishes a session through one or more message exchanges completed as described above, any further messages will be ignored.

[0130] As another example, the DCCP may include an "Identity" message. Such a message may be sent by any element to request the identity of another element. Some of the information sent via an Identity message may be sensitive and need to be sent over a secure channel. Once a connection between peers has been established (e.g., by exchanging session tokens as described above), either party may initiate an Identity message at any time. A party may request the identity of the other party by sending an empty Identity message or a special parameter or flag indicating that an identity is being requested. The recipient of an Identity request may respond with the same Identity message type with a populated Identity value. An Identity request may specify a minimum required Identity value that should be returned. If the peer device does not respond within some specified time limit of the request (e.g., five seconds), the Identity information is incomplete or does not match, and the requester may be disconnected.

[0131] The identity message may include a list of identity values, which consists of identity IDs and corresponding values. Examples of identity IDs include system ID, display name, manufacturer, model (any string describing the device), country code, serial number, OS information, application information, and / or other identifiers. The system ID can be a single string that uniquely identifies the peer device. If the same device (running the same software) is connected, it will always return the same system ID. The system ID may typically include or be generated using the model number, manufacturer, obfuscated serial number, and / or a random number generated by the device. The display name can be the name of the peer device that can be presented to the user. This string may be in English. In some cases, the device may send the display name in other languages ​​using other identity IDs. Each country code can be a string consisting of one or more comma-separated ISO 3166-1alpha-2 codes representing one or more countries for which the peer device is configured. The serial number can be a string that uniquely identifies the device. If the message is transmitted over an insecure connection, the serial number may be a pseudo-serial number (e.g., a number generated on the device or an obfuscated serial number). OS information may include the name and version of the operating system. The application information can include information about the main application running on the device, such as the application's name, vendor, version, etc. Other identification values ​​can be used as needed.

[0132] Another example of a DCCP message is a "ping" message. Such a message can allow an endpoint to determine whether a peer application is responsive. Ping messages can be used to determine whether a connection is still active, so that transmissions that were not properly reported when the connection was lost can be detected. Ping messages can also be sent periodically from the projection client 570 to keep the connection alive while the application is running in the background.

[0133] A ping message may include a request ID identifying the message, where a response ping message should include a matching request ID. A ping message may include various flags, such as a request flag indicating whether a response is required (for a request message), a response flag indicating that the message is a response to a previously sent request, and a one-way flag indicating that no response is required or expected.

[0134] DCCP can include "Sync Time" messages for synchronization gateway 100 and shadowing client 570. Sync Time messages can be used to calculate the offset between the local clock and the remote peer's clock. Different clock synchronization strategies can be used. The simplest algorithm has a peer send a Sync Time message with the peer's current timestamp. The remote peer can respond with the remote peer's current timestamp. The sending peer can then calculate the offset by subtracting the two timestamps and accounting for the delay of the communication link.

[0135] The sync time message may include a request ID identifying the message, wherein a response sync time message should include the same ID. The sync time message may include a client timestamp (e.g., a timestamp in milliseconds or microseconds indicating when the message was sent from the client) and a gateway timestamp (e.g., a timestamp in milliseconds or microseconds indicating when the message was sent from the gateway host).

[0136] Another example DCCP message is the "connected device status" message used by projection client 570 to report status and any mode changes to gateway 100. For example, if connected resource 120 is an in-vehicle device, connected resource 120 can report the current driving state to limit any driver distraction. This message can be sent proactively by client 570.

[0137] The connected device status message may include a restriction level indicating the current restriction level reported by the device, a display status indicating whether the connected device is displaying any content received from gateway 100, and / or other appropriate status fields or other content.

[0138] Figure 6 Another example environment 600 is shown in which one or more embodiments described herein may be implemented. As shown, the environment 600 may include the user device 110 described above and a gateway manager 610, which includes a directory interface 620, a repository 630, a control interface 640, and a model trainer 650. The environment 600 may also include a user device 660.

[0139] Gateway manager 610 may include one or more electronic devices, such as a server or server cluster, and may be accessed via a public cloud, private infrastructure, and / or other suitable resources.

[0140] The directory interface 620 may include an API or other appropriate resources that allow the gateway 100 to request the application directory 590 and / or other appropriate information. The API can utilize a representative state transfer (REST) ​​architecture using Hypertext Transfer Protocol (HTTP) requests or other web service APIs such as Google Remote Procedure Call (gRPC), Simple Object Access Protocol (SOAP), Java Remote Method Invocation (RMI), etc. The API can be executed over a secure connection, such as Transport Layer Security (TLS). The API can support authentication (e.g., a way for the gateway 100 to authenticate to the gateway manager 610) and requests for the application directory (returning a response that matches the connection resource ID).

[0141] The specific application catalog 590 returned may depend on the connection resource 120. User device 110 may connect to different connection resources 120 with different capabilities. Similarly, different applications may be available for different geographic regions, or business considerations may determine which applications should be included or enabled. For example, the provider of the connection resource 120 may license a set of applications to be enabled by default. When requesting or providing the application catalog 590, gateway 100 may use the device information included in the identity message sent by the connection resource 120. The application catalog 590 may be extracted or constructed at runtime using data received from the repository 630.

[0142] The repository 630 may include a database management system (DBMS) that stores various application catalog records, which allows the catalog interface 620 to quickly extract or build the required application catalog 590. The database used by the repository 630 may be a relational database (e.g., Structured Query Language (SQL) Server, MySQL, Postgres, etc.) or a non-relational database (e.g., Dynamo DB, Mongo DB, Simple DB, etc.).

[0143] The application directory 590 can be a data structure that describes which source applications should be managed by the gateway 100. The application directory can be provided in different formats. For example, when requested from the gateway manager 610 via the directory interface 620, the directory can be returned in a portable format such as JSON, prototype buffer, etc. When processed by the gateway 100, the application directory 590 can be represented as a hierarchy of object references. Regardless of the exchange, storage, or processing format, the application directory 590 can include various attributes or parameters. Example parameters can include the version to which the application directory 590 applies and a set of connection resource identifiers.

[0144] The application catalog 590 may include an application list 680 of application records. Such records may be associated with various fields, including examples of such fields described below. An application ID field may include a unique identifier for the application. The ID may be any string, but to avoid name conflicts, it is recommended to use reverse Uniform Resource Locator (URL) notation, such as "com.yelp". A projection type field may describe the types of projections supported by the application (e.g., captured applications, projection applications integrated with an SDK, web applications, etc.). In some embodiments, a special type of application record may be supported that is marked as "data only". Such a record may be used to store common information, such as matching models for system screens (such as keyboards), and may not be associated with a specific application.

[0145] The application category field can indicate the type of application (e.g., the primary function of the application). The category may include multiple levels (e.g., primary, secondary, etc.). Categories can be represented by a string with subcategories separated by slashes (similar to a directory path), where the more general categories come first, followed by the more specific categories, etc. If the application supports multiple categories, the categories can be separated by commas. For example, for a navigation application, the category might be "maps / navigation", while for a video player application, the category might be "media / video", a music player might be "media / music", and an application that supports both video and music might be "media / music, video".

[0146] The Installation Location field can provide the location (e.g., URL) where the associated application can be installed. The Installation Location field may contain different subfields for each supported user device platform. For example, for iOS, the Installation Location field may point to the location of the application on the App Store.

[0147] The display information field may include information about how the application is presented to the end user. The display information field may include elements such as a title (localized for supported languages), an icon (e.g., for different supported resolutions), a description (localized for supported languages), etc. The display information field may include display options, such as whether the application icon should be initially displayed in the launcher screen, whether the application should be shown or hidden from the gateway 100, etc.

[0148] The launch information field may include information about how the gateway 100 can launch the associated application. There may be different subfields for supported user device platforms. For example, for iOS, the launch information field may include the URL pattern for launching the application, while for Android, the launch information field may include the launch intent or package name.

[0149] The authentication information field may include information that allows the gateway 100 to verify and authenticate that the application installed on the user device 110 is the correct application. There may be different authentication records for the various supported platforms and different versions of the application. The gateway 100 may have different ways to obtain application signatures or certificates. For example, on Android, such resources can be obtained using the PackageManager.getPackageInfo API and the GET_SIGNING_CERTIFICATES flag. As another example, an application can be verified by a checksum, where the gateway 100 can calculate a checksum of a predefined binary of the target application and compare it to the value from the authentication information field. On some user device systems, the gateway 100 may be restricted from accessing third-party application information, in which case the record for the authentication information field for that platform may be empty, and the gateway 100 will recognize that a given application cannot be authenticated.

[0150] The recognition model field 685 may store application and screen recognition models (and / or references thereto) for different platforms and supported application versions.

[0151] The application purchase information field can store information to enable applications and / or features through in-app purchases. If an existing in-app purchase framework is used, such as the Store Toolkit or Google Play Billing, the application purchase information field may contain a product ID, with pricing information managed by the corresponding backend system. In some embodiments, the application purchase information field can store the product price (e.g., for different supported domains) and any information required to complete the purchase.

[0152] The application mode restriction field 690 may be an optional record that describes whether any restrictions should be placed on the projection of the current application according to the operation mode of the connection resource 120 .

[0153] The signature field can be a digital signature of the application directory, which can be used to verify the integrity of the directory information. The file containing the application directory can be signed using a private key securely stored on the gateway manager 610 and verified by the gateway 100 using the public key. The signature field allows the directory to be stored at the user device 110, and when the directory is later read, the gateway 100 can verify that the directory has not been modified. If the directory has been modified, the gateway 100 can reject the directory and request a new directory from the gateway manager 610.

[0154] The application catalog 590 may include various other fields, elements, attributes, parameters, or other data as appropriate.

[0155] The control interface 640 may include or provide an application management API or other appropriate resource to allow creation, modification, deletion, and / or other management of application directory records. Such an application management API may utilize a REST architecture with HTTP requests or other web service APIs, such as gRPC, SOAP, Java RMI, etc. The application management API may be executed over a secure connection, such as TLS. The application management API may allow authentication, management of connected devices, creation or deletion of application directories, management of application directory entries, and / or other interactions with the application directory 590.

[0156] User device 660 may be a device associated with an administrator rather than an end user, such as user device 110. User device 660 may include or execute a web portal (via a browser) or management application 670 that provides access to control interface 640. Management application or web portal 670 may be restricted to access through a virtual private network (VPN) for added security.

[0157] The user device 660 and / or the management application or browser 670 can authenticate to the gateway manager 610 and be provided with different access rights. Based on the authentication, some functions of the control interface 640 can be enabled or disabled.

[0158] Management application 670 can manage connection resource identities, allowing creation, deletion, modification of connection resource IDs, etc. Management application 670 can create and / or delete application catalogs and / or manage application catalog entries by adding, deleting, and / or modifying information associated with the application catalog entries.

[0159] The model trainer 650 may generate and / or train the recognition model 665 and / or other models used or implemented by the gateway manager 610 or gateway 100. Model training will be described in detail below with reference to process 1100.

[0160] Figure 7 An example process 700 for projecting application content from a user device 110 to a connected resource 120 is shown. Process 700 can allow a user to utilize the connectivity and / or processing power provided by the user device 110 to extend or enhance the capabilities of the connected resource 120. The process can be performed at regular intervals, upon launching a source application, and / or based on some execution criteria. In some embodiments, process 700 can be performed by the gateway 100. Components such as the projection client 570, the directory interface 620, and / or the control interface 640 can perform processes complementary to process 700.

[0161] As shown, process 700 may include generating (at 710) an application catalog, such as or similar to application catalog 590. The application catalog may be received by gateway 100 from a resource, such as gateway manager 610, via catalog interface 620. Gateway 100 may construct the application catalog based on data received from a resource, such as gateway manager 610, via catalog interface 620. If available, gateway 100 may retrieve a previously generated or received application catalog from a resource, such as repository 530. Any newly received or newly generated application catalog may be stored at gateway 100 (e.g., using repository 530) for future use.

[0162] The shadowing client 570 can send an "Application Catalog" request message to request the latest application catalog or register for application catalog updates. The application catalog request message may include elements such as a request ID. The gateway 100 may return the same request ID with the application catalog response message, allowing the shadowing client 570 to match the request with the response. If the gateway 100 sends an update message, the request ID may be set to zero. The application catalog request message may include various application catalog flags that indicate how the application catalog should be handled. The shadowing client 570 may set the requested flags. The gateway 100 may return the current gateway flag values. The application catalog flags may include an "auto-update" flag, which indicates whether the host should automatically send an "Application Catalog" message each time the application catalog changes. When sent by the shadowing client 570, the auto-update flag requests that the gateway 100 provide automatic updates. To stop automatic updates, the "stop auto-update" flag may be set by the shadowing client 570. When sent by the gateway 100, the stop auto-update flag may include the current automatic update status of the gateway 100. The application catalog flags may include a "full catalog" flag, which indicates whether the gateway 100 should return the complete application catalog. The gateway 100 may return a complete directory flag. If the complete directory flag is received, the shadowing client 570 should update the entire internal directory (add and delete applications). If the shadowing client 570 sets the complete directory flag, this flag may be ignored.

[0163] The application catalog request message may include one or more application records. Each application record may include a pair of attribute IDs and associated values. When requesting the application catalog, the projection client 570 may populate an application record with the IDs of the desired attributes and possible required and / or filter values. Example attribute IDs and associated values ​​are shown in Table 1 below. In response to the application catalog request message, the gateway 100 may return a list of application records containing the requested attributes and associated values ​​for each application.

[0164] As described above, application properties can be specified using predefined name / value pairs contained in application catalog messages (and / or other application catalog resources). Names can be predefined and assigned values, as shown in Table 1 below.

[0165]

[0166]

[0167]

[0168]

[0169] Table 1

[0170] The application catalog can be provided to other resources, such as the shadowing client 570, via application catalog messages. Such messages can be initiated by the shadowing client 570 and received by the gateway 100. Application catalog messages can follow a request and response format. In some cases, the gateway 100 can send automatic updates to the shadowing client 570. Various application catalog message flags can be used to define or modify the message delivery process (e.g., the auto-update or stop auto-update flags can be set to manage automatic update messages).

[0171] The projection client 570 may request data associated with one or more applications (up to the entire application catalog) from the application catalog associated with the gateway 100 in an Application Catalog message. The application may be requested using the application ID in the Application Catalog Request message, if available at the projection client 570. If no application ID is specified, all available applications may be returned and the full catalog flag may be set. The projection client 570 may indicate that one or more specific application attributes be returned in response to the Application Catalog Request message.

[0172] The projection client 570 may request specific application properties in the application catalog message (eg, image width, image height, display name language, etc.) If no application properties are specified, the gateway 100 may return all available application properties.

[0173] Application catalog messages may generally follow a request and response format. However, the projection client 570 may request to be notified when changes occur in the application catalog. For example, the application state may change, or a new application may be added to the application catalog. Such notifications may be supported by an auto-update flag and a stop auto-update flag associated with the application catalog message. If the auto-update flag is set in the application catalog request message from the projection client 570, the gateway 100 may send an application catalog message each time the application catalog changes. As part of such a request message, the projection client 570 may specify which application attributes should be included with each update by populating an application record. If the application record is not populated, all attributes may be returned with each update. The projection client 570 may stop automatic updates by sending an application catalog message with the stop auto-update flag set.

[0174] Process 700 may include identifying and authenticating (at 720) the source application. This identification and authentication may be performed in various ways, depending on which resource launches the application and / or other relevant information. For example, the application may be launched at user device 110, and a request to project the application may be received at gateway 100. As another example, projection client 570 may send a request to launch the application.

[0175] Third-party source applications can be launched through the gateway 100. The launch request can be received through a resource such as a launcher screen or a message received from the connection resource 120. The source applications that can be launched can be stored in the application directory, and the launch information can be provided in the launch information record. Application launch can be performed using APIs or other resources provided by the user device 110 and / or other system elements. For example, for iOS, the gateway 100 can launch an application using a URL pattern. For Android, the gateway 100 can launch an application using a launch intent with the desired application package name. Application launch can be performed by a resource such as the application control module 510.

[0176] In some cases, it may be useful for the projection client 570 to be aware of available and active applications. For example, the projection client 570 can select a specific application to launch. This awareness can allow the user to experience it as if the projected application were running natively on the connection resource 120.

[0177] For example, the connection resource 120 associated with the car system may include a hardware button on the vehicle head unit for launching a navigation application. The projection client 570 running at the connection resource 120 can query the gateway 100 for available applications from the application directory 590 and request the gateway 100 to launch the navigation application at the user device 110.

[0178] The projection client 570 may receive a list of available applications and their status using the application catalog message as described above. The projection client 570 may use the list information to display a collection of application icons or to notify the user about applications available through the gateway 100.

[0179] The DCCP may include a "Current Application" message that the gateway 100 may use to notify the projection client 570 when the active application changes. The active application may change based on a user switch or a directed switch by the projection client 570. The projection client 570 may use the Current Application message to request the launch of a specific application. The Current Application message may include an Application ID element that uniquely identifies the application. The application ID may be stored in the application directory 590. The Current Application message may include an Application State element that defines the state of the application (e.g., "Launch" - requesting the launch of an application, "Active" - ​​indicating that the application is activated, "Launching" - indicating that the system is launching the specified application, and / or "Launch Failed" - indicating that the requested application launch failed). When the launch is successful, an "Active" state may be returned in response to the Current Application message. The Current Application message may include an Application Screen ID element that identifies the ID of the active screen of the current source application (if detected).

[0180] The projection client 570 can use the current application message to request the launch of a given application supported by the application catalog. The message can be processed by the gateway 100 and redirected to a resource such as the application manager 515, which can in turn resolve the application ID to an entry in the application catalog. The application manager 515 can provide the application entry information to the application control module 510, which can attempt to launch (or switch to) the requested application on the user device 110. If the requested application is not installed, the gateway 100 can display an appropriate message and / or instruct the user to install the application. The gateway 100 can return a status indicator via the current application response message.

[0181] Whenever the currently projected active application changes, the gateway 100 can send a current application message with the appropriate status. If supported, the gateway 100 can also send a notification each time the active mode and / or screen changes. The projection client 570 can use the notification message to update the status of the UI elements associated with the active application.

[0182] Once an application is launched, there may be situations where the gateway 100 should limit which applications are projected to the connected resources 120. For example, for in-vehicle systems, there may be distracted driving regulations that prohibit certain applications from being displayed on the in-vehicle display while the vehicle is in motion. Therefore, the gateway 100 must be able to detect the identity of the active application projected to the connected device. Various detection algorithms (or combinations thereof) may be used to identify the active application, depending on the capabilities of the user device 110, the access provided to the gateway 100, and / or other relevant factors.

[0183] If the user device 110 provides a system API to the gateway 100, such as the OS interface 560 or other such resources, active applications can be reported through the interface. The gateway 100 can use the application catalog to match the information reported by the system API with the application ID associated with the entry in the application catalog. For Android, the accessibility service API can be used to detect the package name of the current application. The gateway 100 can then use the application catalog to match the package name with the supported application entries in the catalog.

[0184] Because some user device operating systems may not provide an API or other such resources to detect active applications, a lightweight notification SDK (e.g., development kit 555) can be integrated with the source application to provide notifications to the gateway 100. The notification development kit 555 can communicate with the application control module 510. The application control module 510 can, in turn, notify the application manager 515 when the captured application changes state (e.g., moves to the background or foreground, opens, closes, etc.).

[0185] Notification development kit 555 may communicate with application control module 510 in various ways, such as HTTP messages sent over a socket connection, binary or simple JSON socket messages, an inter-process messaging API available on user device 110, and / or other suitable ways.

[0186] Notification development toolkit 555 may support messages such as authentication messages. This set of messages can be used to authenticate a source application (e.g., source application 550) to gateway 100, and vice versa. Both parties may exchange public certificates, which may be signed by gateway manager 610. The peers may verify the packet signature using system APIs (if available). If both parties are authenticated, they may exchange session keys for additional messaging or other communications.

[0187] The notification development toolkit 555 can support messages such as application state change messages. A source application may send such messages whenever the application state changes (e.g., moved to the foreground, moved to the background, opened, closed, etc.). The application control module 510 can send similar messages when starting and / or stopping shadowing for the current source application. The source application can use such messages to update the UI associated with the source application.

[0188] The Notification SDK 555 can support messages such as screen state change messages. A source application can send this message set whenever the current screen or state changes. For example, if the user activates an edit field, a keyboard can be displayed. Screen state change messages can be used to handle special restrictions based on the application mode.

[0189] Process 700 may include receiving (at 730) source application information. Such information may be received based on the identified application ID and / or screen ID from a resource such as repository 530. Such source information may include, for example, permissions associated with the source application, user information (e.g., username and password), configuration information (e.g., user selections or preferences, manufacturer defaults, etc.), and / or other appropriate information (e.g., output data format, UI function list, etc.).

[0190] As shown, process 700 may include receiving (at 740) connection resource information. In some embodiments, such information may be requested and received from connection resource 120. The connection resource information may be extracted from application catalog 590 and / or related data elements.

[0191] In some embodiments, sensor data (and / or other types of data) may be received from (or via) connectivity resources 120. Such sensor data may include, for example, data received and / or derived from a GPS component, an accelerometer, a gyroscope, a microphone, a temperature sensor, or other environmental sensors. Such data may be sent via any available channel between gateway 100 and connectivity resources 120.

[0192] Such sensor (and / or other) data can be configured in various appropriate ways. For example, the projection client 570 can send an "input capabilities" message to the gateway 100, which indicates any supported sensors or other data sources. The gateway 100 can configure the required input devices by sending a "configure input" message that can specify the device type, protocol and / or other relevant information. The projection client 570 can respond with a configure input reply message, which can include or indicate the device ID of the input data source accepted or applied. The gateway 100 can request an input data stream from a given input device by sending a "control input" message, which indicates the ID of the desired device and the data sampling and / or transmission frequency. The gateway 100 can receive data from the projection client 570 (and / or other components of the connection resource 120) and provide (or inject) the data into the user device OS or source application as appropriate.

[0193] Sensor (and / or other) data can be sent via an external data input or other connection (if available). Many user devices 110 and / or connection resources 120 can support the transmission of external sensor data via standard interfaces. For example, iOS provides GPS data via the iAP attachment protocol. Android allows positioning via standard National Marine Electronics Association (NMEA) compliant devices (e.g., via Bluetooth or serial data bus) (if analog positioning is enabled). The gateway 100 can support a variety of other external sensor data inputs.

[0194] The projection client 570 may send such data on the main data channel using a "Send Input Event" message (and / or other appropriate messages). Sensor (and / or other) data may be collected by the projection client 570 and sent to the gateway 100 via a "Send Input Event" message set. Multiple data sources may send data simultaneously. Multiple data records (from the same or different sources) may be merged into a single message. The gateway 100 may inject or otherwise provide the received data to the source application 545 or the OS interface 560. The gateway 100 may simulate or emulate the data as input via the OS interface 560 (e.g., using various system APIs), depending on the type of data. For example, for Android, a GPS location may be simulated via a "Mock Location Provider" with the "ACCESS_MOCK_LOCATION" permission. For other input data types, the gateway 100 may call various APIs of the source application 545 (if available).

[0195] Process 700 may include projecting (at 750) the source application to the connected resource 120. Gateway 100 may efficiently and with low latency capture the current screen of the active source application 545 on the connected user device 110. Various methods may be used to capture the screen (and / or other output data, including content and / or commands) and project the captured data to the connected resource 120 via projection client 570 and / or other appropriate resources.

[0196] For example, various display technologies, such as screen mirroring, can be used. Some user devices 110 and associated operating systems may support such external display capabilities. For example, AirPlay, MiraCast, ChromeCast, High Definition Multimedia Interface (HDMI), Mobile High Definition (MHL), and / or other appropriate functions may be used to project screen images. In this mode, the gateway 100 may not capture the active screen. Instead, the projection client 570 may use one of the external display modes supported by the user device 110 and receive the current screen image through an external display channel, such as a user device interface 575. The projection client 570 can then display the received content at the connection resource 120.

[0197] This "standard" display mode may not provide the functionality required for additional processing of captured screen data or other content. For example, application identification or functionality restrictions may require that the captured data be provided to various components of the gateway 100, such as the recognition engine 525 or the application control module 510. Therefore, in some embodiments, the projection client 570 can send content (e.g., images received via an external channel) to the projection engine 520 and / or other components for analysis. In this way, the more powerful and capable user device 110 can perform any additional processing, thereby allowing the connected resource 120 to operate with simpler requirements and reduced hardware capabilities.

[0198] This "extended" standard display mode may be initiated by sending a "display capabilities" message from the projection client 570 to the gateway 100 after a connection is established between the gateway 100 and the projection client 570. In some embodiments, this display capabilities information may be retrieved (and / or otherwise retrieved or determined) from a resource such as the repository 530.

[0199] Gateway 100 may send a "display request" message to projection client 570 to request the use of a supported external display channel. Projection client 570 may attempt to open an external display channel to user device 110 (e.g., via user device interface 575) and may respond with a "display response" message indicating the connection status.

[0200] The projection client 570 may send a "Show Request" message to the gateway 100 to request the gateway 100 to assist in starting the external display mode on the user device 110. In some cases, the user device 110 may provide an API (and / or other appropriate resources) to start the external display mode. For example, ChromeCast may be started through the "MediaRouteSelector" function. On devices that do not provide such resources, the gateway 100 may provide instructions to the user and / or launch a system settings menu so that the user can complete the configuration and connection steps. The gateway 100 may respond with a "Show Response" message

[0201] Once the projection client 570 establishes a connection to the external display, the projection client 570 may notify the projection engine 520 via a "Changed in External Display Status" message. The projection engine 520 may send a request to the projection client 570 to send the screen image, for example, via a "Capture Display Request" message. The projection engine 520 may request to pause or resume streaming of the captured image and / or otherwise direct data streaming via an external channel via the "Capture Display Request" message.

[0202] The projection engine 520 can send additional overlay video streams or images on top of (or otherwise combined with) the video (and / or other data) stream via the external display channel to be displayed by the projection client 570. The additional video or image information can be associated with, for example, position, size, and transparency information that indicates how and where the projection client 570 should display the overlay.

[0203] In some embodiments, this overlay information can be used for application or content blocking. Gateway 100 can perform application blocking on content sent via the external display channel. Such application blocking can include determining (e.g., based on the active application identity and any relevant rules from the application catalog) that gateway 100 can request (e.g., via projection engine 520) to stop or resume screen projection. Based on the specific external display technology used, projection client 570 can stop projection from user device 110 or stop displaying received data (but without interrupting streaming from the user device to avoid user interaction later when projection can be resumed). Projection client 570 can notify gateway 100 of changes in projection state via an "external display state changed" message. Similarly, projection engine 520 can stream an overlay image to projection client 570 to restrict certain features (or content) of the projected application, or completely replace streaming with information from user device 110 via "display area request" and "display area data" messages.

[0204] In some embodiments, screens from active source applications may be captured using capabilities of the user device OS and application or content blocking may be implemented from gateway 100 .

[0205] For example, an iPhone may include a "Replay Kit" feature. The gateway 100 can register a screen broadcast extension that (when the user allows it) can capture the phone screen. The application capture component 505 can receive the captured screen image and send the captured screen image to the projection engine 520, which can encode the captured screen image and send it to the projection client 570. The projection engine 520 can send the captured image to the recognition engine 525 and / or the application control module 510 to perform application and / or screen recognition, function blocking and control. The gateway 100 can operate in the background to capture the screen, and process, encode and send the captured screen to the projection client 570.

[0206] For Android phones, the "Media Projection" API or similar resources can be used to capture the phone screen. Gateway 100 can request the necessary permissions from the end user and capture the phone screen in the background. Application capture component 505 can receive the captured screen image and send the captured image to projection engine 520, which can encode the captured image and send it to projection client 570. Projection engine 520 can send the captured image to recognition engine 525 and / or application control module 510 to perform application and / or screen recognition, function blocking, and control.

[0207] For greater flexibility and future scalability, gateway 100 may use an adaptive logic model to represent screen projection areas (or a "projection" model). This projection model may allow the use of multiple displays or multiple areas within a display.

[0208] The projection model may include a "display" element. Such an element may represent a single screen controlled by a source application 545. Connected resources 120 may include multiple physically separate displays. For example, in a vehicle, there may be multiple displays, such as a central display (e.g., a head unit), an instrument panel display, a head-up display, rear or passenger displays, etc. There may be logical displays, such as two separate areas on the same physical display screen. Each display may be assigned a unique identifier.

[0209] The projection model may include a "Display Area" element. Such an element may represent an area within a display element within which data from the gateway 100 should be presented. There may be multiple display areas within a single display element. The shape of a display area may typically be rectangular, but any shape may be supported. Display areas may overlap, and the order in which they are displayed may be controlled by a z-level attribute that specifies the stacking order of the elements. The gateway 100 may configure display areas using "Display Area Request" messages and stream the actual data using "Display Area Data" messages. Each display area may be assigned a unique identifier (called a Display Area ID). A Display Area ID may be unique regardless of the display to which it is assigned. In this way, one Display Area ID is sufficient to identify where each frame of data in a Display Area Data message should be presented.

[0210] The projection model may include an "encoding format" element. The data to be displayed in a given display area may be encoded in a format that the projection client 570 can support. A variety of encoding formats may be supported. Format selection may be based on a balance between image quality, speed, and bandwidth requirements. For example, display data may be provided as bitmap data (e.g., via H264 video, Moving Picture Experts Group (MPEG), or similar formats, such as MPEG-2, Joint Photographic Experts Group (JPEG), Motion JPEG, H265 video, etc.), or may be provided as vector data. On a given display, there may be multiple display areas using different encoding formats. For example, one display area may display a screen captured from a source application, while another display area may display an overlay (e.g., with a text message and a translucent rectangle) to block certain areas of the captured image.

[0211] For data provided in bitmap format, some standards provide high compression rates while maintaining quality, and many user devices 110 include hardware accelerated support for such formats. For user devices 110 that do not support such formats, a simpler bitmap representation of each individually captured frame can be used. To reduce the bandwidth requirements for each frame, the color depth of each pixel can be reduced (e.g., to RGB565) or converted to a different color space (e.g., YUV422) and the number of bits required for each color can be reduced. A fast compression algorithm can be used to compress each frame. In addition, the gateway 100 can encode only the difference from the previous frame and then compress the difference data (e.g., using run-length encoding or other compression techniques) instead of sending a complete image frame.

[0212] To support flexible display area arrangements, the projection client 570 can support display areas that present vector data sent by the gateway 100. For certain types of images or other graphic content (e.g., simple uniform color blocks, text, etc.), vector data is a more efficient way to represent frame data than any bitmap format. The data can be represented in standard vector formats, such as scalable vector graphics (SVG), dual-fast graphics (DXG), etc. In some embodiments, the data can be represented in a proprietary format using protocol buffers to store records. Each record can describe one of the supported graphics primitives and provide drawing parameters. The records can be further compressed to reduce bandwidth requirements.

[0213] To establish a screen projection session, after establishing a connection between the gateway 100 and the projection client 570, the projection client 570 may send a "Display Capability" message to the gateway 100. The gateway 100 may initialize a display, wherein the projection data is sent by sending a "Display Request" message set. If the requested display (and / or associated data format) is supported, the projection client 570 may send a "Display Response" message including a success result code and / or other success indication. For each display, one or more display areas may be initialized by configuring the size, type, and / or other properties of the display area and sending a "Display Area Request" message from the gateway 100 to the projection client 570. If the requested display area is successfully created, the projection client 570 may send back a "Display Area Response" message including a success result code.

[0214] To establish a data stream, the gateway 100 can start screen capture using a module such as the application capture module 505. The captured image can be passed to the projection engine 520. The projection engine 520 can pass the captured image screen to the application control module 510. The projection engine 520 can construct a final image frame (which can include one or more overlays) for each display area. The projection engine 520 can send the display area frame to the projection client 570 via a "display area data" message.

[0215] Gateway 100 can select the optimal screen capture mode based on the currently available connection method, the capabilities of connection resource 120, and / or other relevant factors. For example, while a user is projecting content, the user can disconnect or connect the cable between user device 110 and connection resource 120. As a result, gateway 100 can update or modify the screen capture method. For example, connection resource 120 may initially be connected to user device 110 via a wired HDMI connection. However, if the cable is disconnected, gateway 100 may switch to gateway capture mode and send the content via a Wi-Fi connection without interrupting the user experience.

[0216] When capturing an application screen using one of the techniques described herein, the gateway 100 may need to work in the background because the user may wish to interact with one of the source applications in the application library 540 rather than the application associated with the gateway 100. In some embodiments, the gateway 100 may execute a background companion application that is capable of running in the background. iOS may have more restrictive rules about applications that are allowed to run in the background. The iOS gateway 100 can connect to the connection resource 120 using the External Accessory Protocol (EAP), allowing the gateway 100 to run in the background (and not be terminated by the OS). Such a background companion application and / or other similar features can be used with any of the projection modes described herein. If the primary data channel is not an EAP connection, an alternative low-bandwidth channel (e.g., Bluetooth) can be used to send occasional ping messages to maintain an active connection while the gateway 100 continues to run in the background. For Android-based user devices, there is typically no such requirement as background services can run unrestricted.

[0217] Application shadowing may utilize various display messages that may be used to configure and / or control the flow of video (and / or provision of other content) from user device 110 to connected resources 120. Various examples of such messages are provided below.

[0218] An example display message may be a "display capabilities" message that may be sent by the projection client 570 to the gateway 100 to indicate the capabilities of the connection resource 120. The gateway 100 may respond with a display capabilities response message. Each display capabilities message may include elements such as a supported external display technologies element indicating supported external display technologies. These technologies may be represented as fixed constants that are strings or integer flags that describe the capabilities, such as whether the device supports AirPlay, MiraCast, ChromeCast, HDMI, etc. Each display capabilities message may include a connection status element that indicates whether and / or how the user device 100 is connected to the connection resource 120 (e.g., wireless connection established, HDMI cable plugged in, etc.). Each display capabilities message may include a display characteristics element that may include or indicate a list of display characteristics for each supported display. Display characteristics may include, for example, screen resolution, color depth, supported video encoding formats, and / or other relevant information.

[0219] Another example display message may be a "display request" message. Such a message may be sent by the gateway 100 to the projection client 570 to request display registration and initiate external display projection of source application content. The same message type may be sent by the projection client 570 to the gateway 100 to request a user action. The recipient of a display request message may send a display request response message in response. The display request response message may include elements such as a request ID, which must be included in the display request response message to identify the display request message to which the response is being sent. The display request message may include a display ID element, which indicates the unique ID of the display for which the request is being made. If the display request message is the first message from the gateway 100, the display ID may be generated by the gateway 100. The display request message may include a display description element, which may identify the display to which the external display ID is assigned. The display description field may be populated when a new external display ID is assigned or generated. The display request message may include an action element, which indicates the action to be performed as part of the request. Possible actions include "connect" (requesting the client to connect the display to the user device 110 and start projection), "disconnect" (requesting the client to stop projection and disconnect the display), "request user action" (requesting the recipient to display a prompt message to help the user establish a connection), "show message" (requesting the recipient to display a specific message, such as a user instruction), etc. The requested user action field may include at least one code representing an action (e.g., "pair device," "turn on Wi-Fi," "connect USB / HDMI cable," etc.). If the display message action (and / or other appropriate criteria) indicates so, the display request message may include a message element indicating an optional message to be displayed by the recipient.

[0220] Another example display message may be a "display response" message. Such a message may be sent in response to a display area request message. The display response message may include information such as a request ID (matching the request ID of the corresponding display area request message received), a result (describing the result of the requested action and may include predefined result constants such as "started," "stopped," "user action required," "failed," etc.), and an error (including error information related to the failed request). The error field may be a composite field that includes an error code and / or a human-readable error message.

[0221] Another example display message may be an "external display state changed" message. Such a message may be sent by the projection client 570 or the gateway 100 when the state of an external display has changed. The projection client 570 may provide the current state of the display. The gateway 100 may send an external display state change message to query the projection client 570 for the current state. The external display state change message may include elements such as the external display ID of the associated display. The external display state change message may include a display description element (an optional field sent by the projection client 570). The display description field may only be sent when the display is first started or when display state is requested. The display description field may be a composite value and may include sub-elements such as the selected mode (e.g., streaming technology), screen resolution, frames per second (FPS), transmission method, etc. The external display state change message may include a state element, which may indicate the state of the external display. This state may be limited to a set of predefined constant values, such as "start," "stop," "pause," "resume," "query," etc. The "query" state may be used to request a status update.

[0222] Another example display message may be a "capture display" request message. Such a message may be sent by the gateway 100 to request that the projection client 570 manage the capture of video frames from a given external display. The projection client 570 may respond with a "capture display" response message. Each capture display message may include a request ID (a unique value identifying the request), an external display ID (an identifier of the display for which this capture request should be performed), a requested action (e.g., "start" video capture, "stop" video capture, etc.), a capture resolution (e.g., the resolution at which images should be captured), an encoding type (indicating the video encoding format in which images should be sent, such as H.264, compressed images, motion JPEG, etc.), frames per second (or "FPS," how often images should be sent), and a bandwidth (the maximum bandwidth for the resulting stream).

[0223] Another example display message may be a "capture display response" message, which may be sent in response to a capture display request message. The capture display response message may include a request ID (matching the ID of the corresponding capture display request), a result (describing the result of the requested operation and may include predefined result constants such as "start", "stop", "fail", etc.), and an error (including information about the error that failed the request and may be a composite field containing an error code and a human-readable error message).

[0224] Another example display message may be a "Capture External Display Frame" message. Projection client 570 may send such a message each time a frame is captured for an external display that has been configured to capture frames via a Capture Display Request message. The frequency of the Capture External Display Frame message may be configured via the Capture Display Request message. The Capture External Display Frame message may include elements such as an external display ID (identifying the display from which the screen is being captured) and frame data (data for a given screen frame in the format and size specified in the Capture Display Request message).

[0225] Another example display message may be a "display area" request message. Gateway 100 may use such a message to create, display, hide, destroy, and / or otherwise modify a display area for a given display. Projection client 570 may respond with a display area response message indicating success or failure. A display area request message may include a request ID (a unique identifier for the request), a display area ID (a unique identifier for the display area, which should be new and unique for a request to create a new area to match the ID of a previously created display area), an action (defining the action to be performed), and a display area description (for the newly generated display area).

[0226] Action elements can be selected from elements such as "Create" (requests the recipient to create a new display area, in which case the display area description field should be filled), "Show" (shows the specified display area), "Hide" (hides the specified display area), "Destroy" (destroys a previously created area and optionally hides it), and "Query" (the application requests the latest status of the specified display area). After a destroy request, the display area ID may no longer be valid and can be reused for a new display area.

[0227] The DisplayAreaDescription element can define the parameters of the new display area. The DisplayAreaDescription field should only be populated during a create action. The DisplayAreaDescription element can include sub-elements such as "DisplayID" (indicating the ID of the display for which the new display area should be created), "Position" (indicating the position and / or size of the display area), "ZLevel" (indicating the z-level of the display area), "EncodingFormat" (indicating the expected encoding format for the data), Transparency (indicating any transparency information for the area, such as alpha blending), and / or other appropriate sub-elements.

[0228] Another example display message may be a "display area" response message. Such a message may be returned by the projection client 570 in response to a display area request message. The display area response message may include elements such as a request ID (matching the request ID of the corresponding display area request), a state (indicating the current state of the requested display area, where the state may be selected from discrete values ​​such as "visible", "hidden", "destroyed", "failed" (if an error occurred while trying to create the display area), and an "error" (including information about the error that caused the request to fail and may be a composite field containing an error code and a human-readable error message).

[0229] Another example display message may be a "Display Area Data" message. Gateway 100 may send such a message for each frame of data for a given display area. The Display Area Data message may include element data such as a Display Area ID (an identifier of the display area for which data is provided) and Frame Data (the data for a single frame to be displayed). The Frame Data may be provided in the encoding format specified in the Display Area Request message.

[0230] The process may include receiving (at 760) UI event information at gateway 100. Such UI event information may be received from connection resource 120. The UI event information may include information related to various types of UI elements, provided in various suitable formats. For example, a UI element may include a button associated with a discrete value or function. As another example, a UI element may include a touch screen capable of capturing one or more touch points and / or movement of the touch points.

[0231] As shown, process 700 may include applying (at 770) the UI event information to the source application. Various input event data processing and / or processing algorithms may be utilized to receive and / or identify such UI event information, as described herein. The UI event information may be applied to the source application via a message set sent using DCCP. Gateway 100 may analyze the received UI event information and apply the information to the source application via a message set or other data sent via an available communication channel.

[0232] The DCCP may include a variety of message sets that may be used to configure the input capabilities of the connection resource 120 and, optionally, to control the user device 110 (and associated source applications). For example, the projection client 570 may send an "input capabilities" message to the gateway 100 to describe the capabilities of the connection resource 120 in terms of supported input capabilities. The gateway 100 may respond with its own input capabilities message.

[0233] The Input Capabilities message may include an element such as a request ID that uniquely identifies the request. The corresponding response should contain the same request ID. The Input Capabilities message may include an InputDevices element that provides a list of input devices. For each input device in the InputDevices list, a "Type" element defining the input control type (e.g., touchscreen, keyboard, hardware button, joystick, etc.) and an "Interface" defining the communication interface (e.g., HID, iPod Out, etc.) may be included or provided. The interface may describe the external user input control channel (e.g., HID over USB) and / or the data sent over the primary data channel (e.g., TCP / IP over Wi-Fi). The Input Capabilities message may include a DataDevices element that provides a list of data devices. For each data device in the DataDevices list, information such as a "Type" element defining the data device type (e.g., GPS, location provider, temperature sensor, accelerometer, gyroscope, etc.), an "Interface" element specifying the connection method (e.g., data message, HID, BTNMEA, iPod location, etc.), and / or a "SupportedDataInputs" element specifying which types of data (and in which units) may be provided. A standard such as the Geneva In-Vehicle Infotainment (Genivi) Vehicle Signalling Specification (VSS) may be used.

[0234] As another example, a "Configure Input" message may be used to configure one or more input devices and / or input areas. The Configure Input message may initially be sent by the gateway 100. The same message may be returned by the projection client 570 as a response.

[0235] The configuration input message may include a request ID element that uniquely identifies the request. The corresponding response should include the same request ID. The configuration input message may include a source action area element that defines the source action area. The source action area element may include a list of input area records, each of which may include an "input area ID" field containing a unique identifier for the input area, a "shape" field indicating the area's shape, a "type" field indicating the input area's type, and an optional "device ID" field containing the device associated with the given input area. The device ID may be sent by the projection client 570 in the configuration input response message. The configuration input message may include a target action area element that defines the target action area and may include attributes such as shape and orientation. The configuration input message may include a keyboard device element that provides a list of keyboard input devices. For each device in the list, the device ID and type may be included. The device ID may be assigned by the gateway 100. The keyboard device element may include a "parameters" field that contains parameters for the keyboard device, such as which buttons are enabled. The configuration input message may include a data input device element that provides a list of data input devices. For each data input device in the list, information such as a device ID (a device ID assigned by the gateway 100) and data inputs (a list providing data inputs that should be sent by the device) may be indicated or included. For each data input, the gateway 100 may assign a data input ID that may be used later to send the input data.

[0236] The DCCP may include a "control input" message that may be sent by the gateway 100 to start, stop, and / or otherwise control a given input device. The status of the command may be obtained through an external input notification message.

[0237] A control input message may include a control region element, which includes a list of control input regions. Each element in the control input region list may include fields such as "input region ID" (the ID of a previously configured input region, unless the "calibrate" action is used), "device ID" (the ID of the device used for the calibration action), "action" (specifies the action to be performed, which can be selected from a set of discrete values, such as "start", "stop", or "calibrate"), and "calibration parameters" (specifies the parameters of the calibration, which may include parameters such as step size, scale, docking mode, docking position such as "nearest" or "top left", etc.).

[0238] A control input message may include a Data Input Device element that specifies a list of data input devices to start and / or stop. For each data input device in the list, information such as the device ID (the ID of a previously configured device), action (the type of action to be performed for the device, such as "start" or "stop"), and parameters (providing additional parameters that may be included for the device or each device input, such as the frequency of data collection, any conversions, etc.) may be included.

[0239] The DCCP may include a "Send Input Event" message that may be sent by the gateway 100 to request the shadowing client 570 to perform a specified input event using a given device (where the device ID should have been previously configured via a control input message). The Send Input Event message may be used to send an event (either an input event or a data event) from the shadowing client 570 to the gateway 100 via the primary data channel.

[0240] A send input event message may include an "input event" element including a list of events to be sent, wherein each entry in the list may include an "event ID" field that uniquely identifies the event, an "event type" field that specifies the type of event to be sent (e.g., cursor moved, mouse up, mouse down, etc.), and a "position" field that defines the location of the event (where the position may be absolute or relative). A send input event message may include a "data event" element including a list of data events, wherein each entry in the list may include a "data input ID" field that uniquely identifies the input device to which the control input message was previously assigned, a "timestamp" that indicates when the data was collected or received, and a "data" field that contains the actual data payload.

[0241] The DCCP may include an "external input notification" message. Such a message may be sent by the projection client 570 when an external input event has been executed on the connection resource 120 and sent via external user input. The message may be received by the gateway 100. The external input notification message may include (if requested in the send input event message) an event ID element that uniquely identifies the event. The external input message may include an event type element that indicates the type of event (e.g., "input start," "input stop," "up," "down," "single touch," "multi-touch," etc.). The external input notification message may include an area ID element that includes a unique identifier for the area where the event occurred, a device ID element that includes a unique identifier for the input device that generated the event, and a timestamp element that indicates when the event occurred (specified for the projection client 570).

[0242] User input (and associated UI event information) can be processed using various modes, including external user input mode and gateway user input mode. Gateway 100 can select the best user input processing mode based on the currently available connection channels, the UI capabilities of the connection resource 120, and / or other relevant factors. User input processing can attempt to control the source application 545 based on the UI information received from the connection resource 120, as if the end consumer is interacting directly with the user device 110.

[0243] For input control relying on pointing devices such as a mouse, touch screen (single or multi-touch), stylus, digitizer, and / or other similar types of devices, the gateway 100 may use or implement flexible configuration of associated input areas on the connection resource 120 and the user device 110. Figure 2 An example of such a flexibly configured area is described.

[0244] Returning to process 700, one way to control the source application 545 is to use any standard (external) available control path. This approach can be referred to as "external user input control". One or more external control channels can be included in (or provided by) the user device interface 575. One such channel is the HID standard originally introduced for USB, which now supports various communication paths, such as Bluetooth, serial, wireless personal area network, etc. The HID channel can support various different types of devices. The user input device (e.g., connection resource 120) can present a HID descriptor to the host. The HID descriptor can include a byte array that describes the data packet structure used by the device, the number of data packets supported by the indicator device, the size of the data packet, the purpose of each byte and bit in the data packet, and / or other relevant information. The user input device can send actual data in a subsequent HID report record that conforms to the HID descriptor sent previously.

[0245] The connection resources 120 , through a device control module that may be included in (or implemented by) the user device interface 575 , can emulate various input device types to the user device 110 (and / or associated source application).

[0246] An example input device type is an "absolute pointing device." This type of device may be included in the HID category used to describe single-point or multi-point pointing devices (e.g., digitizers, styluses, single-point or multi-point trackpads, etc.). The main feature of this type of device is that the connection resource 120 can send absolute coordinates for single-point and / or multi-point touches, thereby simulating touch events on, for example, the main screen of the user device 110.

[0247] In some embodiments, an absolute pointing device may be initialized by generating or otherwise preparing a HID descriptor at the user device interface 575. The descriptor may define the dimensions or limits of the HID coordinates sent via the HID report or message. The user device interface 575 may send the HID descriptor to the user device 110 via an external control channel (e.g., via USB) and / or via other available communication channels.

[0248] If the projection client 570 receives pointing device input event data (e.g., a touch screen, mouse, joystick, external touchpad, and / or other such input event data), absolute pointing device input data may be generated. For example, the projection client 570 may evaluate the received coordinates to determine whether the coordinates are within any input regions defined in the configuration input message. If there are multiple contacts (associated with the same event) located in different regions, the event data may be processed by different handlers based on the location of the regions. For example, if a contact originates in a first input region and moves to a second input region, the information may be ignored. If a contact originates in a first input region and remains within the first input region, the event information may be processed by an event handler associated with the first input region.

[0249] If the coordinates are within the external input area, the event information can be forwarded to the user device interface 575, which can convert the input coordinates (for each associated touch point), send a HID report (which can include multiple reports or entries for each touch point based on the capabilities of the user device 110) over the external control channel or other available communication path, and send an "external input notification" message to the gateway 100. The conversion of the input coordinates can include scaling and transposing the received input coordinates from the projected client area 220 to the target action area 210 coordinates, and then scaling and converting them to HID coordinates (taking into account the screen orientation reported in the configure input message).

[0250] Another example input device type is a "relative pointing device." Such a device may be included in the HID class used to describe individual pointing devices such as mice, trackpads, trackballs, and the like. The primary characteristic of these types of devices is that the resource 120 can send relative coordinates of cursor movements on the main screen of the user device 110. Related "click" type events (for one, two, or more buttons in various configurations) may be sent as part of the HID report.

[0251] If the connected resource 120 uses a touchscreen as an input device and needs to control the user device 110 via a relative pointing device interface, it must convert absolute coordinates into relative cursor movements and clicks. To accomplish this conversion, the connected resource 120 must understand the characteristics of the associated pointing device, such as the number of steps, current location, and direction. This information can be discovered through a calibration process. In some embodiments, to ensure recovery from errors, the gateway 100 can utilize cursor docking.

[0252] A calibration process, through which the connection resource 120 can discover various characteristics of the pointing device supported by the user device 110, such as the cursor step size (the number of steps it takes to move the cursor one step on the screen via HID). There may be separate step size values ​​for each x and y axis. In some embodiments, step size values ​​can be specified based on the speed of movement (if there is an acceleration curve supported by the user device 110). The calibration process may further discover characteristics such as the current position of the cursor and the docking step number (the relative number of steps that should be sent via HID to dock the cursor). Different step size values ​​can be specified for each docking angle.

[0253] Calibration can typically be performed for each newly connected user device 110 and / or when the OS or settings of the user device 110 are updated. To determine whether calibration is required, the gateway 100 can determine whether the current calibration parameters are valid. This calibration check can include displaying the full screen (e.g., a calibration check screen) from the gateway 100 to the user device 110 and initiating the capture of all input events from the full screen. The gateway 100 can activate the desired pointing device by sending a control input message including the current calibration parameters to the projection client 570. The projection client 570 can pass the message to the user device interface 575, which can store the device parameters and activate the HID device. The projection client 570 can detect the cursor position by sending a Send Input Event message that provides a zero (e.g., 0,0) movement offset and a button press state. The user device interface 575 can send one or more HID reports indicating zero movement and down / up button states. A simulated click can be received at the gateway 100 via the calibration check screen. The gateway 100 can store the initial position associated with the simulated click.

[0254] The gateway 100 can send a series of pointing device moments and click events via one or more send input event messages. The gateway 100 can send absolute coordinates that fall within the calibration check screen size. The user device interface 575 can send one or more HID reports via a HID channel (e.g., an external control channel) based on the received send input event message. The absolute coordinates can be converted into a relative event set (based on the current calibration parameters). The gateway 100 can receive and capture clicks from the calibration check screen and associate the clicks with input events. The gateway 100 can calculate an error score based on the difference between the received cursor position and the expected position based on the current calibration parameters. Based on the error score, the gateway 100 can determine whether the calibration is valid (e.g., if the error score is less than a specified threshold). If calibration is not required, the projection client can start sending UI event data.

[0255] The calibration process can be a collaboration between the gateway 100 and the projection client 570. To start the calibration process, the gateway 100 can send a "control input" message to the projection client 570, which sets the input area associated with the current HID device to the "calibration" state. The user device interface 575 can send the HID descriptor of the relevant pointing device to the user device 110 via an external control channel or other appropriate channel. The user device interface 575 can send an external input notification message to the gateway 100 including the input area ID and a status indicating that the calibration process has begun.

[0256] The gateway 100 can display a full screen (e.g., a calibration screen) on the user device 110 and begin capturing all received events from the calibration screen. For operating systems such as iOS and Android, there are APIs that allow third-party applications to enter full-screen mode. There may be system areas where the gateway 100 cannot receive HID input. The gateway 100 can detect these areas and manage the calibration process accordingly.

[0257] The gateway 100 may send a calibration event by sending an input event message. For example, the gateway 100 may send different calibration sequences, such as [8, -8], [-8, 8], [16, -16], [-16, 16], [24, -24], [-24, 24], [32, -32], and [-32, 32], where each pair represents a relative motion along the x and y axes (e.g., [<offset x>, <offset y>]) followed by a click event to initiate detection by the gateway 100. After each event, the projection client 570 may respond with an external input notification message.

[0258] The gateway 100 can capture and record events from the calibration screen during the calibration sequence. The gateway 100 can filter out any unexpected events. For example, the end user may accidentally interact with the user device screen and generate input events. Unexpected or incorrect events can be filtered based on various relevant criteria. For example, a specific button press can be used to indicate that the data is associated with a calibration click (e.g., a middle mouse button click, or a series of clicks such as left click-right click-left click). As another example, a special up / down click sequence can be used to indicate whether the click is associated with calibration. Such a sequence may be difficult to recreate accidentally (e.g., each point to be checked may have a down / up / down / up sequence). As another example, the captured events can be matched with notifications received from the projection client 570. The gateway 100 can record the received screen points (in user device screen coordinates) and associate the received screen points with simulated events. The gateway 100 can store a timestamp indicating when the event was received.

[0259] The gateway 100 can calculate calibration characteristics based on the recorded points. The step length for each coordinate axis can be calculated by determining the relative movement of the mobile screen pointer (from the screen point) relative to the input device using the provided input points. If there are different step lengths based on the speed of the event, they will be recorded as separate steps for that speed. The current cursor position can be extracted from the last recorded screen point. The docking step length can be calculated based on the estimated step length and the size of the user device screen.

[0260] The gateway 100 may send the calibration characteristics to the projection client 570 via a control input message including the device ID of the calibration device and the calibration parameters. The projection client 570 may apply the calibration parameters and store them for future sessions. The gateway 100 may perform a calibration check and, if the calibration check is unsuccessful, retry the calibration, where the number of retries may be configurable.

[0261] One challenge with relative pointing devices is that they are designed to control the movement of a cursor that is visible to the end user. Therefore, the target device does not report the actual position through the HID interface. The projection client 570 needs to track the cursor position. The expected cursor position can be calculated based on the input HID report provided and the previously calculated calibration parameters. However, errors may occur, and the actual cursor position may not be correct. For example, the movement step size calculated during the calibration process may have a small error. Even relatively small errors can accumulate after a few movements, and the final estimated cursor position and the actual position may differ significantly.

[0262] To overcome such errors, the gateway 100 can implement an error correction mechanism using cursor docking. Docking may involve moving the cursor to a known location so that the projection client 570 can confirm the cursor location and perform accurate relative movements from there. Docking relies on the fact that excessive pointer movement to a corner of the screen will cause the cursor to "get stuck" in that corner, resulting in a known location.

[0263] The cursor can be docked at any of the four corners of the screen. To dock the cursor, the projection client 570 can send a docking command set, such as a relative move event, that equals or exceeds the user device screen size. To account for errors in the calibration data, the calculated screen size may be multiplied by 2 (or some other appropriate factor) to ensure that the cursor docks at a corner.

[0264] In some cases, if the screen resolution is high, sending too many points may degrade performance. In this case, the multiplication factor can be adjusted to reduce excessive motion (e.g., by changing the factor from two to one tenth and two tenths). Docking can be performed by the projection client 570 as part of the normal event sending sequence.

[0265] Various docking modes can be used. Such docking modes can include a "pre-click" mode, in which the projection client 570 can first dock the cursor and then move the cursor to the desired location from the known docking position. This method ensures that the final position is the most accurate. The disadvantage is that the end user may see the cursor move to the docking position and then back.

[0266] The docking mode may include a "post-click" mode, in which docking is performed after receiving a click event. This approach reduces the visual movement of the cursor before the click. If there are other methods of moving the cursor that cannot be detected by the projection client 570 (for example, the user may be interacting with the user device 110), the estimated cursor position for the next event may be inaccurate.

[0267] The docking mode may include an "after inactivity" mode, in which docking may be performed after a specified duration of inactivity. This approach reduces the visual movement of the cursor to and from the docking point. Similar to the post-click mode, if there are other methods of moving the cursor that the projection client 570 cannot detect, the estimated cursor position for the next event may be inaccurate.

[0268] The projection client 570 may select a docking mode and a target location based on information received via a control input message from the gateway 100. Docking may be performed with reference to a specific corner or the nearest docking location.

[0269] Once the pointing device has been calibrated and / or verified to function correctly, the gateway 100 can simulate UI event information, such as touch event information received from the connection resource 120, to the user device 110 as relative pointing device event information. During use, the projection client 570 can receive or recognize pointing device input events (e.g., touch screen, mouse, joystick, external touchpad, and / or other such inputs).

[0270] Projection client 570 can determine whether the received coordinates fall within any input zones received via the configuration input message. If multiple touch points fall within different zones, these points may be processed by different handlers based on the zones. If a touch point originates in one zone and moves to a second zone, the associated event information can be ignored. If a touch point originates in a specific zone and remains in that zone, the associated event information can be processed by the corresponding handler for that specific zone.

[0271] If the received input coordinates fall within the external input area, the event information can be forwarded to the device control module of the user device interface 575. The device control module can convert the received coordinates, for each matching touch point, by scaling and translating from projected client area coordinates to target operating area coordinates, and then scaling and converting to HID coordinates (taking into account the orientation reported in the configure input message).

[0272] The device control module can perform docking based on the configuration input settings. The device control module can calculate the necessary relative movement events based on the current input position and the target screen cursor position. The relative movement calculation can use calibration parameters received through the configuration input message. The device control module can send HID reports (where each contact may be associated with multiple reports) through, for example, an external control channel. The device control module can send external input notification messages to the gateway 100.

[0273] In addition to or in lieu of a touch screen or other touch-based input, HID elements may include other input types, such as a keyboard, game controller, etc. These elements may be or include physical and / or virtual components. For example, some connected devices 120 may include a physical keyboard or keypad, while other such devices may present a keyboard or similar resource via a touch screen display.

[0274] When a keyboard HID is available, the projection client 570 can report support for this keyboard HID via an input capability message. As a result, the gateway 100 can request activation of the keyboard to receive data from an input field or simulate certain actions on the source application 545. Before activating the keyboard, the gateway 100 can configure the keyboard device by sending a configure input message.

[0275] For example, the keyboard HID can be activated when an input field in the source application 545 requires data. For example, the user may be asked to enter a search term, username, password, or similar information. The gateway 100 can automatically detect that the text input field has application focus (e.g., based on information received from the source application and / or OS interface, if any) and instruct the projection client 570 to activate the keyboard HID by sending a control input instruction specifying the device ID for the previously configured keyboard device. As a result, the projection client 570 can use an external keyboard connected to the connection resource 120 or display an on-screen virtual keyboard. Any button press information received from the keyboard can be sent to the user device 110 via the external control channel.

[0276] The keyboard HID can be used to simulate or emulate various actions at the active source application 545. For example, the keyboard HID can simulate a "back" button, media play / pause, next / previous, etc. The gateway 100 can initiate such emulation by launching the keyboard HID via a control input message and specifying only the key codes to be supported (e.g., key codes associated with a consumer control device such as a mouse, keyboard, etc.).

[0277] When a specific command needs to be simulated based on the received UI event information, the gateway 100 can send an input event message that specifies the virtual key code for the command simulation. As a result of the message, the device control module can send the required HID report with the key code. The device control module can convert the key code into a value that the active HID device understands or recognizes.

[0278] The HID protocol is highly configurable, and many different types of input can be provided using the protocol. Consumer controlled input devices (such as game controllers) are handled similarly to the keyboards, absolute, and relative pointing devices mentioned above.

[0279] Any new input device can be easily added to the system via input capability messages, where the projection client 570 can report the available HID capabilities and the gateway 100 can configure these devices via configuration input messages and then start using a specific device via control input messages. User interactions with the connected resources 120 can be sent as HID events, or the gateway 100 can instruct the projection client 570 to simulate specific HID events by sending input event messages.

[0280] In addition to HID, the user device 110 may also support other proprietary or open external input methods, such as iPod controls, control via an audio interface, etc. Similar operations to those described above may be used for these interfaces.

[0281] In addition to or in lieu of external user input, user input to the source application 570 may be provided through the gateway 100. Depending on the capabilities, version, and limitations of the user device 110, different APIs may be provided to third-party applications that allow for simulation of input events. For example, if the necessary permissions are provided to the gateway 100, various Android third-party applications may be used to simulate keyboard and touch events through the accessibility service (e.g., using the accessibility interaction client class). If the gateway 100 has system permissions (e.g., root access), the gateway 100 may simulate input events by interacting with the input subsystem of the OS.

[0282] In order to send input events from the gateway 100, the projection client 570 can receive or identify pointing device input events (e.g., touch screen, mouse, joystick, external touchpad and / or other such inputs). The projection client 570 can determine whether the received coordinates are located in any input area defined by configuring the input message. If there are multiple touch points that fall into different areas, different points can be processed by different handlers associated with different areas. If the touch point is initiated in the first input area and moves to the new input area, the input event information can be ignored. If the touch point is initiated in a specific area and remains in the specific area, the touch point can be processed by the handler associated with the specific area.

[0283] If the coordinates are within the gateway input region, the event data may be sent in a send input event message to gateway 100. In response to the send input event message, gateway 100 may send the event information to source application 545 using one of the supported channels and / or algorithms.

[0284] Figure 8 An example process 800 is shown for identifying a source application executed by a user device 110. Source application identification is described above with reference to Figure 3 Process 800 may be performed whenever a new application is launched (e.g., based on the generation or receipt of a current application message indicating a source application launch, a screen output change, and / or other relevant criteria). In some embodiments, process 800 may be performed by gateway 100.

[0285] As described above, some user device operating systems may provide an interface or other resource through which gateway 100 may receive active or current application information. Process 800 may be performed when no such interface is available and / or when different and / or additional recognition algorithms are implemented than those provided by the OS interface. In addition, process 800 may be used to filter content provided through connection resources 120. For example, during certain operating modes (e.g., modes associated with a moving vehicle), content areas that display video or other distracting elements may be filtered or blocked.

[0286] As shown, process 800 may include receiving (at 810) source application media output. Such output may include, for example, a set of screen images or other video content, audio content, and / or other media content. Such content may be received by a component such as application capture element 505.

[0287] Process 800 may include selecting (at 820) portions of the media output for analysis. These portions may be selected or otherwise specified in various suitable ways. For example, if the data is received in a video stream format, each complete frame may be selected for analysis. As another example, some source applications may include graphical user interface (GUI) elements, such as display windows that may be indicated in various ways, depending on the format of the received data. In this case, areas associated with the video display or other active content may be selected for analysis, while areas associated with text, static images, or other such elements may be ignored. As another example, as described above, various areas may be used for GUI elements, logos, or other such content.

[0288] The process may include analyzing (at 830) the selected media output portion using an available recognition model (and / or other suitable means). The recognition model may define elements such as the location of the output portion to be analyzed, a color profile, matching image elements, and / or other suitable information. In addition, each recognition model may be associated with a specific source application (as indicated by an application ID).

[0289] As described above, the recognition model can indicate various image comparison algorithms, including comparison techniques such as color bin-based analysis (as described in reference color matching), pixel-by-pixel comparison (e.g., when a number of pixels that have changed exceeds a threshold or ratio), perceptual hashing algorithms (e.g., algorithms that use hashing to generate fragments or fingerprints of various media forms), threshold-based scene detection (e.g., the intensity and / or brightness of the current screen can be compared to a specified threshold, and analysis can be triggered when the intensity and / or brightness exceeds the threshold), and / or other appropriate direct image comparison algorithms.

[0290] As shown, process 800 may include determining (at 840) whether the application is recognized. Such a determination may be made based on evaluation criteria (e.g., a minimum match score or other metric threshold) indicated by the relevant recognition model. If the application is recognized, the process may include providing (at 850) an application ID and / or other appropriate information, where the application ID may be retrieved from the matching model used to identify the source application.

[0291] The process may include (at 860) providing a default identification (if available) or otherwise indicating that no matching application was identified. Such a default application ID may be associated with the type of application or content. For example, some embodiments may include a default video application associated with a source application that provides full-screen video output. Such a default application may be associated with various associated restrictions (e.g., no video when the vehicle is moving) and / or other attributes described with reference to the application catalog.

[0292] Figure 9 An example process 900 is shown for receiving commands and / or other UI event information at a connection resource 120 and applying the received commands at a user device 110. Such UI events can be received via an external control channel or as described gateway user input. The process can be performed when a connection is established between the gateway 100 and the projection client 570. In some embodiments, the process 900 can be performed by the gateway 100.

[0293] As shown, process 900 may include receiving (at 910) UI event information from connection resource 120. Such event information may include various types of event data, depending on various UI features or properties that may be associated with various input areas (e.g., whether absolute or relative pointing is used). The event information may include information related to multiple events and / or sub-events (e.g., a click event included in a click and drag operation).

[0294] Process 900 may include extracting (at 920) event attributes from the received UI event information. For example, such event attributes may include a set of coordinates, an action indicator (e.g., "down," "up," etc.), and / or other appropriate information (e.g., an identification of a keyboard button or key).

[0295] The process may include converting (at 930) event attributes based on the available user device interface and / or other appropriate criteria. For example, absolute coordinates may be converted to relative movement information. As another example, an action location may be mapped from a resource such as the projected target area 230 to the target action area 210 (or vice versa).

[0296] As shown, process 900 may include emulating (at 940) an event at the user device to apply the UI event to the source application. As described herein, such emulation may utilize various messaging algorithms and / or elements, and / or other suitable means of emulating or emulating event data.

[0297] Figure 10 An example process 1000 is shown for filtering application content based on the operating mode of a connected resource 120. The process may be performed whenever the gateway 100 provides content to a projection client 570. In some embodiments, the process 1000 may be performed by the gateway 100.

[0298] In addition to detecting the active source application, in some embodiments, the current screen or mode of the application can be detected. Such detection can be used to limit certain functions or UI elements depending on the operating mode of the connected resource 120 (and / or related elements, such as a vehicle). For example, if the connected resource 120 is an in-vehicle display unit and the vehicle is moving, in many countries, driver distraction regulations prohibit the unit from displaying moving images (such as video) or allowing the user to enter text using a keyboard. Process 1000 can detect whether such content is being displayed and block such content or block the projection of the entire application. Similar application restrictions may be required or otherwise implemented for other industries and use cases.

[0299] As shown, process 1000 may include receiving (at 1010) connected resource sensor data, if available. Depending on the type of connected resource 120 (e.g., a vehicle), various different types of sensor data may be available and / or relevant. For example, motion data may be provided (or calculated) based on information received from a GPS element (which may be included at or associated with the connected resource 120), an accelerometer, a gyroscope, etc.

[0300] Process 1000 may include receiving (at 1020) user device sensor data. In addition to and / or in lieu of connection resource sensor data, some embodiments may receive sensor data, such as location or movement information, from a user device.

[0301] The process may include (at 1030) identifying an operating mode associated with the source application 545, the connected resource 120 (and / or associated systems or devices, such as a vehicle), and / or other related components. The process 1000 may identify various parameters and / or attributes associated with such operating modes (e.g., the speed of the moving vehicle, the processor load of the connected resource 120, the network connectivity of the user device 110, etc.). Such data may be received from various elements, determined based on received data such as sensor data, and / or generated or received in other ways.

[0302] The gateway 100 can detect the current mode of the source application 545. This detection can be performed by resources such as the OS interface 560, the notification development toolkit 555, and / or the recognition engine 525 (e.g., using the various image recognition algorithms described herein). The current screen (or other source application output) can be reported to the application manager 515. The application manager 515 can use the application mode restriction information from the application catalog 590 to determine whether any or all UI elements should be blocked or filtered.

[0303] Similarly, the gateway 100 can detect the operating mode of the connected resource 120, such as whether the resource is stationary or mobile. Such a determination can be made by comparing received connected resource and / or user device sensor data with various pattern recognition models. Such models can indicate, for example, the minimum distance and / or speed required to indicate vehicle movement. The operating mode may be related to time attributes (e.g., daytime versus nighttime operation). For example, displayed content may be dimmed or brightened based on the time of day or sensed light conditions (e.g., using data received from a user device camera or connected resource camera or other sensor).

[0304] In a similar manner, gateway 100 can detect the operating mode of associated user device 110. For example, gateway 100 can determine the available connection type (e.g., Wi-Fi, cellular network, etc.) from user device 110 to a resource such as a media server. As another example, gateway 100 can receive user device information such as processor load, memory usage, and / or other metrics that can be used to detect various operating modes.

[0305] As shown, process 1000 may include determining (at 1040) whether any mode-related restrictions apply. Such a determination may be made based on various identified operating modes, restriction modes, and / or other operating rules and / or other appropriate criteria.

[0306] In order to manage various types of connection resources 120 and support different restriction modes, the connection resource 120 reports a general restriction level value through a "connection device status" message that may include a restriction level element. The current restriction level value can be passed to the application manager 515. Depending on the type of connection resource 120, there may be different restriction level values.

[0307] For example, possible restriction level values ​​for the in-vehicle system may include "none," "minor," "major," "full," and / or other similar values ​​and / or indicators (e.g., the example values ​​may be mapped to values ​​from zero to three). In this example, a restriction level of none (or zero) may indicate that no restrictions should be applied (e.g., a full UI may be allowed) and may typically be used when the vehicle is not moving and the user is not engaged in driving (e.g., when the parking brake is engaged or the automatic transmission is in "park").

[0308] The secondary (or one) restriction level may indicate that minor restrictions should be applied (e.g., most UI may be allowed) and may typically be used when the vehicle is moving very slowly (e.g., less than 8 kilometers per hour) or has stopped at an intersection (e.g., when no motion is detected and the automatic transmission is in "Drive" or "Reverse").

[0309] The major (second) restriction level may indicate that major restrictions should be applied (e.g., only UI elements related to driver assistance functions are allowed) and may typically be used when the user is fully engaged in driving and the vehicle may be traveling at relatively high speeds (e.g., over 8 kilometers per hour).

[0310] The full (three) restriction level may indicate that full restrictions should apply (eg, no distracted driving is allowed and the UI may be completely blocked) and may typically be used when the vehicle is traveling at high speeds and / or operating in hazardous conditions.

[0311] The above examples describe various possible restriction level values ​​for automotive implementations. Various other implementations may include various other restriction level values ​​and / or related attributes based on the specific requirements of a particular implementation. For example, medical and / or industrial implementations may be associated with various other sets of restriction level values.

[0312] In some cases, the connected resource 120 may report its operating mode. However, in some embodiments, the gateway 100 may determine the operating mode (and / or any associated restriction level) independently of the connected resource 120. This approach may be useful in situations where the connected resource 120 could be tampered with and cause an unsafe condition. For example, in automotive systems, the host computer typically uses a wire from the parking brake to detect whether the vehicle is in motion. Such a wire can be easily bypassed by the end consumer, eliminating possible driver distraction restrictions in the system.

[0313] The gateway 100 can independently verify that the vehicle is moving using the location and / or movement sensors of the user device 110 , thereby reporting the correct operating mode and identifying the correct restriction level. The gateway 100 can report the identified restriction level to the application manager 515 .

[0314] The application mode restriction record in the application catalog 590 may be used to determine whether to restrict the currently projected source application 545 based on the current operating mode of the connection resource 120 (and / or other components).

[0315] The possible structure of an application mode restriction for a given application in the catalog is described in Table 2 below. This example utilizes a hierarchical structure stored in text or binary form. The storage format can utilize JSON, XML, Protocol Buffers, etc. An application mode restriction record can be associated with a list of matching rules that define restrictions and permissions for the application and / or associated widgets. An application mode restriction record may follow a hierarchical matching structure similar to Cascading Style Sheets (CSS). Table 2 below describes various example high-level attributes that may be associated with each rule record:

[0316]

[0317]

[0318] Table 2

[0319] Restriction rules may overlap depending on the definitions of widgets and screens. The main matching principle is to select the most specific rule before the more general one. For example, a widget is more specific than a screen, and a widget ID is more specific than a widget type.

[0320] When the application manager 515 matches rules to the screen being projected, the application manager 515 can first apply the screen rules (if any), then apply the rules applicable to the widget type, and then the rules for the specific widget. In this way, for example, an application can be configured to block the entire screen and only allow interaction with one widget by defining a permission rule for that widget ID.

[0321] Restrictions may filter or block specific actions (or data) from the UI associated with the projected application. Restrictions may be associated with an application (or screen, widget, etc.) via the application catalog 590 as a mode restriction element 690 and / or other appropriate means. Restrictions may be defined as flags so that multiple restrictions can be associated with a single rule. Table 3 below lists some example restrictions that may be supported.

[0322]

[0323] Table 3

[0324] If the process determines (at 1040) that pattern-related restrictions should be applied, process 1000 may include (at 1050) applying the pattern-related restrictions identified in the matching rule. Such restrictions may be applied in various appropriate ways depending on the connection resource attributes, rule content, and / or other relevant factors. For example, a block display restriction may cause streaming data to be paused or not displayed at the connection resource 120.

[0325] If the process determines (at 1040) that mode-related restrictions should not be applied, the process can include (at 1060) applying default restrictions, application-specific restrictions, or applying no display or interaction restrictions.

[0326] As shown, process 1000 may include analyzing (at 1070) content received from a source application. This analysis may include applying various recognition models to the screen content (or portions thereof). Some embodiments may analyze and apply such recognition models to other types of content. As an example, a difference calculation may be used to identify areas of a UI or screen that may include distracting content such as a video.

[0327] Process 1000 may include determining (at 1080) whether any restriction criteria are met, for example by evaluating the received content relative to any available restriction rules. Such a determination may be made based on content analysis, various identified operating modes, restriction modes, and / or other operating rules, and / or other appropriate criteria. This approach may allow for automatic detection of content types and application of restriction rules to those content types. Thus, for example, even though the source application may not directly provide video or similar content, UI features and / or other elements may produce similar effects.

[0328] If the process determines (at 1080) that the restriction criteria have been met, the process may include (at 1090) applying filters to the received content, blocking such content, and / or otherwise manipulating the content as appropriate.

[0329] Figure 11 An example process 1100 for applying machine learning to generate or update an application recognition model and / or other models, rules, or elements described herein is shown. This process can be performed whenever additional model training data is available. In some embodiments, process 1100 can be performed by gateway manager 610, and can specifically be performed by model trainer 650.

[0330] Machine learning algorithms based on neural networks (NNs) have shown very good results in various computer vision and image classification tasks. In some embodiments, the gateway manager 610 can be used to generate, train and / or otherwise manipulate recognition models and / or associated objects, such as rule objects. A rule object that is part of a recognition model may include parameters associated under the algorithm-specific data section, such as NN model and NN model type. The NN model parameters can store the trained neural network model in a format that can be read by the machine learning library used by the recognition engine 525. The NN model type parameter can describe the type of neural network model provided in the NN model field.

[0331] There are many readily available tools and libraries that allow for the rapid training and use of neural network-based image classification models. One such library is Google MLKit. Such tools can be run on a user device 110 and classify images using a provided custom neural network model. Another such library is Apple's Vision framework, which is available on iOS devices. The Vision framework is optimized for iOS devices and allows for the use of custom Core ML models to perform various machine learning tasks. Core ML models can be associated as NN models with rules that are part of a larger recognition model. The recognition engine 525 can utilize multiple neural network libraries and select based on rule data (e.g., NN model type).

[0332] As shown, process 1100 may include receiving (at 1110) application content. Such content may include video content, screen content, UI elements, photo elements, and / or other content associated with a source application. Such content may be received from a resource such as gateway manager 610 via repository 630.

[0333] Process 1100 may include receiving (at 1120) an application identification associated with the received content (if available). For example, a screen (or other content) used by each gateway 100 to identify an application (or other element) may be sent to the gateway manager 610 along with the identity of any matching application identified by the gateway 100.

[0334] The process may include receiving (at 1130) a recognition model associated with the identity received at 1120. Such information may be provided by the gateway 100 to the gateway manager 610 for storage and association with the application content and / or identity, if any.

[0335] As shown, process 1100 may include receiving (at 1140) recognition feedback, which may be used to train the model received at 1130. For example, a user (or other ML recognition model) may identify an application based on the received content.

[0336] Process 1100 may include updating (at 1150) the recognition model based on the received feedback. For example, if the application is correctly recognized by the model, a reward may be applied to the model, while if the application is incorrectly recognized, a penalty may be applied to the model.

[0337] Process 1100 and / or similar processes may be used to apply machine learning to the various models, rules, algorithms, etc. described herein.

[0338] A separate model (stored as a separate rule element) can be trained for each supported application. However, this approach may slow down the classification process at the gateway 100 because the recognition engine 525 must evaluate multiple models. Alternatively, all supported applications can be trained in a single NN model within a rule. Depending on the accuracy of the combined model, a hybrid approach may be used where most applications are trained into a single NN model, but a subset of available applications and / or screen modes (e.g., keyboard mode) may be trained in separate NN models and rules.

[0339] A meta-learning approach can be used during training to determine whether to use one rule for all applications or some combination of individual rules, which experiments with different combinations and selects the one that produces the best overall results (e.g., as indicated by the highest accuracy and performance).

[0340] The various image recognition algorithms described herein can utilize training models to correctly classify captured screen images. This training can be performed by the gateway manager 610 as an offline process.

[0341] To train the recognition model, the model trainer 650 may require a large number of example screen images from each supported application. In addition, non-application images (e.g., "negative" examples) are also required to assist with training and model evaluation tasks. Screen capture can be done manually, through an automated process, or a combination of both. The captured images can be stored in an image repository (e.g., an image repository included in the repository 630).

[0342] During manual screen capture, one or more human operators (or "analysts") can run the application on user device 110 or a user device simulator and collect captured screen images. The analysts can then annotate the captured screen images with the appropriate tags. During this process, the analysts should attempt to access every possible screen and utilize different application operating modes. Screen images can be collected for different user device models, screen resolutions, screen orientations, display modes (e.g., "night," "day," etc.), and so on.

[0343] Using manual screen collection can be laborious and time consuming, especially if the process needs to be repeated for each new version of the target application. In some embodiments, an automated application capture process can be used. The automated process can interact with the user device 110 in a manner similar to the screen capture and user input described herein. The automated application capture process can utilize an application debugging interface (such as ADB for Android or XCTest for iOS) that allows detection of active applications, screens, and / or UI widgets. Thus, the automated application capture process can "walk through" all screens and application modes and capture sample screenshots in the process. In addition, the automated application capture process may be able to correctly label images because the current application and screen ID are known.

[0344] In some embodiments, a human operator may generate an application interaction path by recording a set of input events, which may be replayed by an automated application capture process and collecting screenshots during the playback.

[0345] Screen capture images collected using manual or automated processes can be stored in an image repository, which can include a database of training and evaluation images captured for different versions of an application. Such an image repository can be a component of repository 630. The image repository can store label information associated with the captured images. Images can be categorized by application, application version, and / or other appropriate attributes.

[0346] The model training process of some embodiments may perform model training. The model training process may be configured to determine which type of model to train and which algorithm to use. The model training process may be configured with a variety of possible algorithms and input parameters. The model training process may utilize meta-training to determine the optimal combination of algorithms, input parameters, and / or other relevant properties.

[0347] Images can be retrieved from an image repository and run through various training algorithms to generate a recognition model. The retrieved images may include training images and evaluation images (which can be manually specified or randomly selected from the training set). The recognition model can be evaluated and scored using a set of evaluation images (also retrieved from the image repository). If the model meets certain quality criteria (e.g., exceeds a score threshold), or improves over other models, the updated model may be stored in the model repository and tagged with the correct version.

[0348] The model library can be a database that stores the latest recognition models generated by the model training process. The recognition models can be versioned and have associated evaluation scores or other performance metrics. If the evaluation score exceeds a specified threshold, the recognition model can be uploaded to the gateway manager 610 and the associated application catalog record 590.

[0349] As a specific model (and / or rule) training example, a training image set can be used to train color matching rules by using model data parameters as input parameters of a training algorithm.

[0350] Input parameters to the training algorithm may include a list of allowed image operations and a merge color bin threshold, which may define the maximum number of additional color bins allowed for merging two rules. The input parameters may be defined by a human operator and / or may be learned using standard meta-learning techniques.

[0351] The training process can generate or retrieve an empty training model. For each training image, the training process can extract the labels associated with the image. These labels can typically be extracted from the file name or a separate lookup table. For each training image and each supported image operation, the training process can apply the image operation to the training image and store the result as the current training image, calculate the color bins as described in the reference color matching classification algorithm, and store the result as a candidate rule.

[0352] For each training image, the training process may evaluate the candidate rules for the current training image and select the rule with the fewest number of color bins.

[0353] For each candidate rule, the training process may search the trained model for other rules with the same label. If no such rule is found, the candidate rule may be added to the trained model. If a rule with the same label is found, the candidate rule may be merged with the existing rule by checking whether the candidate rule has fewer than the merge color bin threshold number of extra colors (or other specified threshold). If the candidate rule is smaller than the threshold, the color bins from the candidate rule may be combined with the color bins from the existing rule. If the candidate rule has more than the merge color bin threshold number of extra colors, the candidate rule may be added to the trained model.

[0354] Those skilled in the art will recognize that processes 700-1100 may be implemented in a variety of different ways without departing from the scope of this disclosure. For example, the elements may be implemented in a different order than shown. As another example, some embodiments may include additional elements or omit various listed elements. Elements or sets of elements may be executed iteratively and / or based on meeting certain performance criteria. Non-dependent elements may be executed in parallel.

[0355] The above processes and modules may be implemented at least in part as a software process, which may be specified as one or more sets of instructions recorded on a non-transitory storage medium. These instructions may be executed by one or more computing elements (e.g., microprocessors, microcontrollers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), other processors, etc.), which may be included in various appropriate devices to perform the actions specified by the instructions.

[0356] As used herein, the terms "computer-readable medium" and "non-transitory storage medium" are entirely restricted to tangible, physical objects that store information in a form readable by an electronic device.

[0357] Figure 12 A schematic block diagram of an exemplary device (or system or device) 1200 for implementing some embodiments is shown. Figure 1-6 The described systems and / or devices may be implemented at least in part using device 1200. As another example, reference Figure 7-11 The described processes may be implemented, at least in part, using device 1200 .

[0358] Device 1200 can be implemented using various appropriate components and / or sub-devices. For example, device 1200 can be implemented using one or more personal computers (PCs), servers, mobile devices (e.g., smartphones), tablet devices, wearable devices, and / or any other suitable devices. The various devices can work individually (e.g., device 1200 can be implemented as a single smartphone) or in conjunction (e.g., some components of device 1200 can be provided by the mobile device, while other components can be provided by the server).

[0359] As shown, device 1200 may include at least one communication bus 1210 , one or more processors 1220 , memory 1230 , an input component 1240 , an output component 1250 , and one or more communication interfaces 1260 .

[0360] The bus 1210 may include various communication paths that allow communication between the components of the device 1200. The processor 1220 may include a processor, a microprocessor, a microcontroller, a digital signal processor, a logic circuit, and / or may be capable of interpreting and executing instructions and / or otherwise manipulating data. The memory 1230 may include a dynamic and / or non-volatile memory structure and / or a device that can store data and / or instructions for use by other components of the device 1200. Such a memory device 1230 may include space within a single physical memory device or space across multiple physical memory devices.

[0361] Input components 1240 may include elements that allow a user to communicate information to a computer system and / or manipulate various operations of the system. Input components may include a keyboard, a cursor control device, an audio input device and / or a video input device, a touch screen, a motion sensor, etc. Output components 1250 may include a display, a touch screen, an audio element such as a speaker, an indicator such as a light emitting diode (LED), a printer, a tactile or other sensory element, etc. Some or all of the input and / or output components may be connected to device 1200 wirelessly or optically.

[0362] Device 1200 may include one or more communication interfaces 1260 that can connect to one or more networks 1270 or other communication paths. For example, device 1200 can be coupled to a web server on the Internet so that a web browser executed on device 1200 can interact with the web server when a user interacts with an interface operating in the web browser. Device 1200 can access one or more remote storage 1280 and one or more external components 1290 via communication interface 1260 and network 1270. Communication interface 1260 may include one or more application programming interfaces (APIs) that can allow device 1200 to access remote systems and / or storage, and can also allow remote systems and / or storage to access device 1200 (or its components).

[0363] Those skilled in the art will recognize that any or all components of computer system 1200 may be used in conjunction with some embodiments. In addition, those skilled in the art will appreciate that many other system configurations may be used in conjunction with some embodiments or components of some embodiments.

[0364] In addition, although the illustrated examples may illustrate many individual modules as separate elements, those skilled in the art will recognize that these modules can be combined into a single functional block or element. Those skilled in the art will also recognize that a single module can be divided into multiple modules.

[0365] The device 1200 can perform various operations in response to the processor 1220 executing software instructions stored in a computer-readable medium (e.g., the memory 1230). These operations may include operations on the output component 1250 (e.g., display of information, tactile feedback, audio output, etc.), operations on the communication interface 1260 (e.g., establishing a communication channel with another device or component, sending and / or receiving a message set, etc.), and / or operations on other components of the device 1200.

[0366] The software instructions may be read into memory 1230 from another computer-readable medium or from another device. The software instructions stored in memory 1230 may cause processor 1220 to perform the processes described herein. Alternatively, hard-wired circuits and / or dedicated components (e.g., logic circuits, ASICs, FPGAs, etc.) may be used in place of or in combination with the software instructions to implement the processes described herein. Thus, the implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0367] The actual software code or dedicated control hardware used to implement the embodiments does not limit the embodiments. Therefore, the operation and behavior of the embodiments are described without reference to specific software code, and it should be understood that software and control hardware can be implemented based on the description herein.

[0368] Although certain connections or devices are shown, additional, fewer, or different connections or devices may be used. Furthermore, although various devices and networks are shown separately, the functionality of multiple devices may be provided by a single device, or the functionality of one device may be provided by multiple devices. Furthermore, multiple instances of the illustrated networks may be included in a single network, or a particular network may include multiple networks. Although some devices are shown as communicating with a network, some such devices may be fully or partially integrated as part of the network.

[0369] Some implementations are described herein in conjunction with threshold values. To the extent that the term "greater than" (or similar terms) is used herein to describe the relationship of a value to a threshold value, it should be understood that the term "greater than or equal to" (or similar terms) may be considered similarly, even if not explicitly stated. Similarly, to the extent that the term "less than" (or similar terms) is used herein to describe the relationship of a value to a threshold value, it should be understood that the term "less than or equal to" (or similar terms) may be considered similarly, even if not explicitly stated. In addition, when used in relation to a threshold value, the term "satisfies" may refer to "greater than a threshold value," "greater than or equal to a threshold value," "less than a threshold value," "less than or equal to a threshold value," or other similar terms, depending on the appropriate context.

[0370] Unless expressly stated, any element, action or instruction used in this application should not be interpreted as critical or essential. As used herein, an example using the term "and" does not necessarily exclude the interpretation that the phrase "and / or" is intended in that example. Similarly, an example using the term "or" as used herein does not necessarily exclude the interpretation that the phrase "and / or" is intended in that example. In addition, as used herein, the article "a" is intended to include one or more items and can be used interchangeably with the phrase "one or more". In the case of only one item, the term "one", "single", "only" or similar language is used. In addition, unless expressly stated otherwise, the phrase "based on" is intended to mean "based at least in part on".

[0371] The foregoing relates to illustrative details of exemplary embodiments and may be modified without departing from the scope of the present disclosure. Although particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the possible implementations of the present disclosure. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. For example, although each dependent claim listed below may directly depend on only one other claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

Claims

1. A device for projecting, controlling, and managing applications, comprising: One or more processors configured to: Establish a connection from the gateway to the connection resource; Identifying the source application at the gateway based on screen image data generated by the source application includes: receiving at the gateway at least one screen image from the source application; receiving a set of application matching rules, each application matching rule being associated with a particular source application; and applying each application matching rule in the application matching rule set to the at least one screen image to identify the source application; and projecting content received from the source application from the gateway to the connection resource; receiving, at the gateway, user input information from the connection resource; and Apply the received user input information to the source application.

2. The device according to claim 1, wherein Identifying the source application further comprises: generating a set of matching scores; and The source application is identified by selecting a particular source application associated with the application matching rule that generates a highest match score from the set of match scores.

3. The device according to claim 2, wherein The application matching rules are received from an external gateway manager service.

4. The apparatus of claim 1 , wherein the one or more processors are further configured to: determining that a parallel control path from the connection resource is available; wherein the parallel control path includes a user device interface associated with the connection resource and is separate from a projection path carrying projected content received from the source application; and and sending input configuration parameters defining an input mode from the gateway to the connection resource, wherein applying the received user input information to the source application comprises receiving absolute movement event information associated with the user input event and converting the received absolute movement event information into relative movement event information based at least in part on the input mode and receiving the relative movement event information via the parallel control path.

5. The device according to claim 4, wherein Applying the received user input information to the source application further includes automatically docking a cursor by receiving a docking command set on the parallel control path to correct a relative movement error, and applying the received docking command set to a user device interface associated with the gateway, wherein the docking command set includes a relative movement event that exceeds a size of the user device interface associated with the gateway.

6. The apparatus according to claim 1, wherein The content that the projection receives from the source application includes: identifying, at a gateway, an operating mode associated with the connection resource; identifying, at the gateway, a mode-related set of restrictions associated with the operating mode; and The mode-dependent set of restrictions is applied to the projected content at the gateway.

7. The apparatus according to claim 6, wherein Identifying an operating mode associated with the connected resource includes determining that movement of the connected resource exceeds a specified speed threshold, and applying the mode-related set of restrictions to projected content includes preventing at least one user interface element from being projected.

8. A non-transitory computer-readable medium storing a plurality of processor-executable instructions to: Establish a connection from the gateway to the connection resource; Identifying the source application at the gateway based on screen image data generated by the source application includes: receiving at the gateway at least one screen image from the source application; receiving a set of application matching rules, each application matching rule being associated with a particular source application; and applying each application matching rule in the application matching rule set to the at least one screen image to identify the source application; projecting, from the gateway to the connected resource, content received from the source application; receiving, at the gateway, user input information from the connection resource; and Apply the received user input information to the source application.

9. The non-transitory computer-readable medium of claim 8, wherein: Identifying the source application further comprises: generating a set of matching scores; and The source application is identified by selecting a particular source application associated with the application matching rule that generates a highest match score from the set of match scores.

10. The non-transitory computer-readable medium of claim 9, wherein: The application matching rules are received from an external gateway manager service.

11. The non-transitory computer-readable medium of claim 8, the plurality of processor-executable instructions to further: determining that a parallel control path from the connection resource is available; wherein the parallel control path includes a user device interface associated with the connection resource and is separate from a projection path carrying projected content received from the source application; and and sending input configuration parameters defining an input mode from the gateway to the connection resource, wherein applying the received user input information to the source application comprises receiving absolute movement event information associated with the user input event and converting the received absolute movement event information into relative movement event information based at least in part on the input mode and receiving the relative movement event information via the parallel control path.

12. The non-transitory computer-readable medium of claim 11, wherein: Applying the received user input information to the source application further includes automatically docking a cursor by receiving a docking command set on the parallel control path to correct a relative movement error, and applying the received docking command set to a user device interface associated with the gateway, wherein the docking command set includes a relative movement event that exceeds a size of the user device interface associated with the gateway.

13. The non-transitory computer-readable medium of claim 8, wherein: The content that the projection receives from the source application includes: identifying, at a gateway, an operating mode associated with the connection resource; identifying, at the gateway, a mode-related set of restrictions associated with the operating mode; and The mode-dependent set of restrictions is applied to the projected content at the gateway.

14. The non-transitory computer-readable medium of claim 13, wherein: Identifying an operating mode associated with the connected resource includes determining that movement of the connected resource exceeds a specified speed threshold, and applying the mode-related set of restrictions to projected content includes preventing at least one user interface element from being projected.

15. A method for projecting, controlling, and managing an application, comprising: Establish a connection from the gateway to the connection resource; Identifying the source application at the gateway based on screen image data generated by the source application includes: receiving at the gateway at least one screen image from the source application; receiving a set of application matching rules, each application matching rule being associated with a particular source application; and applying each application matching rule in the application matching rule set to the at least one screen image to identify the source application; projecting, from the gateway to the connected resource, content received from the source application; receiving, at the gateway, user input information from the connection resource; and Apply the received user input information to the source application.

16. The method according to claim 15, wherein Identifying the source application further comprises: generating a set of matching scores; and The source application is identified by selecting a particular source application associated with the application matching rule that generates a highest match score from the set of match scores.

17. The method according to claim 16, wherein The application matching rules are received from an external gateway manager service.

18. The method according to claim 15, further comprising: determining that a parallel control path from the connection resource is available; wherein the parallel control path includes a user device interface associated with the connection resource and is separate from a projection path carrying projected content received from the source application; and and sending input configuration parameters defining an input mode from the gateway to the connection resource, wherein applying the received user input information to the source application comprises receiving absolute movement event information associated with the user input event and converting the received absolute movement event information into relative movement event information based at least in part on the input mode and receiving the relative movement event information via the parallel control path.

19. The method according to claim 18, wherein Applying the received user input information to the source application also includes automatically docking a cursor by receiving a docking command set on the parallel control path to correct a relative movement error, and applying the received docking command set to a user device interface associated with the gateway, wherein the docking command set includes a relative movement event that exceeds a size of the user device interface associated with the gateway.

20. The method of claim 15, wherein: The content that the projection receives from the source application includes: identifying, at a gateway, an operating mode associated with the connection resource; identifying, at the gateway, a mode-related set of restrictions associated with the operating mode; and Applying the mode-related set of restrictions to the projected content at the gateway; and identifying the operating mode associated with the connection resource includes: Determining that movement of the connected resource exceeds a specified speed threshold, and applying the mode-dependent set of restrictions to the projected content includes preventing at least one user interface element from being projected.

Images