System and method for personalized photo sharing

Abstract

A system and method of personalized photo sharing includes receiving image data captured by a sensor system, the image data including an object proximate to a vehicle, identifying the object in the image data as a user of the vehicle, receiving a user context of the user of the vehicle, and receiving a vehicle context of the vehicle. The system and method also include determining that photo sharing is appropriate based on a vehicle context of the vehicle, retrieving a photo associated with a user account of the user based on the user context and the vehicle context, and displaying the photo associated with the user account on a screen in communication with the data processing hardware.

Description

[0001] introduction

[0002] The information provided in this section is for the purpose of presenting the general context of this disclosure. The work of the currently named inventors, to the extent described in this section, and in aspects that may not qualify as prior art at the time of filing, is neither expressly nor implicitly acknowledged as prior art to this disclosure. Technical Field

[0003] This disclosure generally relates to personalized photo sharing within the vehicle's user interface. Background Technology

[0004] Typically, when a user enters the vehicle, the user interface is used by the vehicle's applications (such as...). There was a short period of time before (etc.) took over. However, in an increasingly connected world, users are increasingly interested in getting personalized experiences from their connected devices.

[0005] Therefore, instead of displaying a blank screen on the user interface, users can benefit from a more personalized experience that learns from their movement and context to infer what will be useful or interesting to them. For example, photo sharing can be used to remind users of specific tasks (e.g., having lunch, picking someone up from daycare), and / or to highlight celebratory moments such as birthdays and anniversaries. However, accurately identifying users and selecting relevant photos within a network of cameras with different viewpoints can be challenging. Furthermore, photos should only be shared when it is safe to do so to prevent distraction. Additionally, personalizing photos and / or filters on photos can be challenging without high-quality methods for tracking user movement and network activity. Summary of the Invention

[0006] One aspect of this disclosure provides a computer-implemented method for personalized photo sharing, which, when executed on data processing hardware, causes the data processing hardware to perform operations including receiving image data captured by a sensor system, the image data including objects approaching a vehicle, and identifying the objects in the image data as a user of the vehicle. The operations also include receiving a user context of the vehicle, receiving a vehicle context, and determining, based on the vehicle context, that photo sharing is appropriate. The operations further include retrieving photos associated with a user's user account based on the user context and the vehicle context, and displaying the photos associated with the user account on a screen in communication with the data processing hardware.

[0007] Implementations of this disclosure may include one or more of the following optional features. In some implementations, displaying a photo associated with a user account on a screen communicating with data processing hardware includes applying filters to the photo based on user context. In some examples, user context includes a knowledge scope. In these examples, a knowledge scope may include one of time-independent, slowly changing, or time-sensitive. Additionally or alternatively, retrieving photos associated with a user's user account based on user context may include selecting photos based on the knowledge scope of the user context.

[0008] In some implementations, determining that photo sharing is appropriate based on vehicle context includes determining that the vehicle context indicates one or more of the following: the vehicle is parked, the vehicle is connected to a network, a screen communicating with data processing hardware is on, or a screen communicating with data processing hardware is available. In some instances, the sensor system includes multiple cameras. In these examples, identifying an object in the image data as a user of a vehicle may include generating a corresponding confidence level for each of the multiple cameras that the object in the image data is a user of a vehicle. Here, identifying an object in the image data as a user of a vehicle may include determining that the corresponding confidence level of one of the multiple cameras exceeds a threshold. In some implementations, identifying an object in the image data as a user of a vehicle includes identifying the object as a registered user.

[0009] Another aspect of this disclosure provides a system for personalized photo sharing, the system including data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations including receiving image data captured by a sensor system, the image data including objects approaching a vehicle, and identifying the objects in the image data as a user of the vehicle. The operations also include receiving a user context of the vehicle, receiving a vehicle context, and determining, based on the vehicle context, that photo sharing is appropriate. The operations further include retrieving photos associated with a user's account based on the user context and the vehicle context, and displaying the photos associated with the user account on a screen in communication with the data processing hardware.

[0010] This aspect may include one or more of the following optional features. In some implementations, displaying photos associated with a user account on a screen communicating with data processing hardware includes applying filters to the photos based on user context. In some examples, user context includes a knowledge scope. In these examples, the knowledge scope may include one of time-independent, slowly changing, or time-sensitive. Additionally or alternatively, retrieving photos associated with a user's user account based on user context may include selecting photos based on the knowledge scope of the user context.

[0011] In some implementations, determining that photo sharing is appropriate based on vehicle context includes determining that the vehicle context indicates one or more of the following: the vehicle is parked, the vehicle is connected to a network, a screen communicating with data processing hardware is on, or a screen communicating with data processing hardware is available. In some instances, the sensor system includes multiple cameras. In these examples, identifying an object in the image data as a user of a vehicle may include generating a corresponding confidence level for each of the multiple cameras that the object in the image data is a user of a vehicle. Here, identifying an object in the image data as a user of a vehicle may include determining that the corresponding confidence level of one of the multiple cameras exceeds a threshold. In some implementations, identifying an object in the image data as a user of a vehicle includes identifying the object as a registered user.

[0012] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the specification, drawings, and claims. Attached Figure Description

[0013] The accompanying drawings described herein are for illustrative purposes only for the selected configurations and are not intended to limit the scope of this disclosure.

[0014] Figure 1 This is a schematic diagram of an example system for personalized photo sharing.

[0015] Figure 2 yes Figure 1 A schematic diagram of an example component of the system.

[0016] Figure 3 Is using Figure 1 The system displays a sample user interface for personalized photos.

[0017] Figure 4 When the user approaches Figure 1 The system is a flowchart of the user's trajectory.

[0018] Figure 5 This is a flowchart illustrating an example of the operational setup for determining when personalized photo sharing is appropriate.

[0019] Figure 6 yes Figure 1 A schematic diagram of an example component of the system.

[0020] Figure 7 This is a flowchart illustrating an example layout of a method for personalized photo sharing.

[0021] In all the accompanying drawings, the corresponding reference numerals denote the corresponding parts. Detailed Implementation

[0022] The example configuration will now be described more fully with reference to the accompanying drawings. The example configuration is provided so that this disclosure will be thorough and will fully communicate the scope of this disclosure to those skilled in the art. Specific details, such as examples of specific components, apparatus, and methods, are set forth to provide a thorough understanding of the configuration of this disclosure. It will be apparent to those skilled in the art that specific details are not required, the example configuration may be embodied in many different forms, and the specific details and example configuration should not be construed as limiting the scope of this disclosure.

[0023] The terminology used herein is for the purpose of describing a particular exemplary configuration only and is not intended to be limiting. As used herein, the singular articles “a,” “an,” and “the” may also be intended to include plural forms unless the context clearly indicates otherwise. The terms “comprises,” “comprising,” “including,” and “having” are inclusive and therefore specify the presence of features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. The method steps, processes, and operations described herein should not be construed as requiring them to be performed in the specific order discussed or shown, unless specifically identified as such. Additional or alternative steps may be employed.

[0024] When an element or layer is referred to as “on another element or layer,” “joined to,” “connected to,” “attached to,” or “linked to” another element or layer, it may be directly on, joined to, connected to, attached to, or linked to the other element or layer, or there may be intermediate elements or layers present. Conversely, when an element is referred to as “directly on another element or layer,” “directly joined to,” “directly connected to,” “directly attached to,” or “directly linked to” another element or layer, there may be no intermediate elements or layers present. Other terms used to describe relationships between elements should be interpreted in a similar manner (e.g., “between” vs. “directly between,” “adjacent” vs. “directly adjacent,” etc.). As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0025] The terms “first,” “second,” “third,” etc., may be used herein to describe various elements, components, regions, layers, and / or parts. These elements, components, regions, layers, and / or parts should not be limited by these terms. These terms may be used only to distinguish one element, component, region, layer, or part from another. Unless the context clearly indicates otherwise, terms such as “first,” “second,” and other numerical terms do not imply order or sequence. Therefore, without departing from the teachings of the example configuration, the first element, component, region, layer, or part discussed below may be referred to as the second element, component, region, layer, or part.

[0026] In this application, including the following definitions, the term "module" may be replaced by the term "circuit". The term "module" may refer to, be part of, or include the following: application-specific integrated circuit (ASIC); digital, analog, or mixed-signal analog / digital discrete circuit; digital, analog, or mixed-signal analog / digital integrated circuit; combinational logic circuit; field-programmable gate array (FPGA); processor (shared, dedicated, or grouped) that executes code; memory (shared, dedicated, or grouped) that stores code executed by the processor; other suitable hardware components that provide the described functionality; or combinations of some or all of the foregoing, such as in a system-on-a-chip.

[0027] The term "code" as used above can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or objects. The term "shared processor" includes a single processor that executes some or all of the code from multiple modules. The term "group processor" includes processors that, in combination with additional processors, execute some or all of the code from one or more modules. The term "shared memory" covers a single memory that stores some or all of the code from multiple modules. The term "group memory" includes memory that, in combination with additional memory, stores some or all of the code from one or more modules. The term "memory" can be a subset of the term "computer-readable medium." The term "computer-readable medium" does not include transient electrical and electromagnetic signals propagating through the medium and can therefore be considered tangible and non-transitory memory. Non-limiting examples of non-transitory memory include tangible computer-readable media, which include non-volatile memory, magnetic memory, and optical memory.

[0028] The apparatus and methods described in this application can be implemented, partially or entirely, by one or more computer programs executed by one or more processors. The computer program includes processor-executable instructions stored on at least one non-transitory tangible computer-readable medium. The computer program may also include and / or depend on stored data.

[0029] A software application (i.e., a software resource) can refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0030] Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0031] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0032] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system, which includes at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0033] The processes and logical flows described in this specification can be executed by one or more programmable processors (also known as data processing hardware) that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logical flows can also be executed by special-purpose logic circuitry (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). As an example, processors suitable for executing computer programs include both general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to receive data from or transfer data to, or both. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0034] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen) for displaying information to the user and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0035] refer to Figure 1In some implementations, system 100 includes a vehicle 10 communicating with a remote system 60 via network 40 (e.g., via wired or wireless communication). Vehicle 10 and / or remote system 60 execute photo-sharing system 200. In short, and as described further in detail below, photo-sharing system 200 is configured to apply probabilistic methods to improve user identification of user 222 using multiple camera inputs, determine the appropriateness of photo sharing based on vehicle 10, and select photos 254 and / or filters 256 of photos 254 by inferring what user 222 wants to see. Notably, by inferring the appropriate photos 254 for a specific user 222 approaching vehicle 10 and which effects to apply to photos 254, photo-sharing system 200 provides user 222 with a highly personalized vehicle experience.

[0036] In the example shown, the photo-sharing system 200 is implemented within vehicle 10. However, the photo-sharing system 200 can be implemented in any other propulsion system, such as, but not limited to, motorcycles, trucks, off-road vehicles, farm equipment, trains, airplanes, etc. Vehicle 10 includes data processing hardware 12 and memory hardware 14 storing instructions that, when executed on the data processing hardware 12, cause the data processing hardware 12 to perform operations. Vehicle 10 also includes a user interface 30 with a screen 32. Figure 3 The screen 32 is configured to display a photo 254 selected by the photo sharing system 200. The user interface 30 can be implemented in the infotainment system of the vehicle 10; however, it should be understood that the user interface 30 can be implemented in other computing devices (e.g., computing devices communicating with the vehicle 10), such as, but not limited to, head-up displays, smartphones, tablets, smart displays, desktop / laptop computers, smartwatches, smart appliances, or smart glasses / headsets. The vehicle 10 also includes an imaging system 16, 16a-16d (e.g., one or more cameras) configured to capture image data 20 of the environment 104 of the vehicle 10. For example, the image data 20 may include one or more data segments including objects 102 approaching the vehicle 10. Here, the one or more cameras 16a-16d may continuously or periodically collect image data 20 of the environment of the vehicle 10 and provide it as input to the photo sharing system 200 for further downstream processing (e.g., object detection, user recognition, etc.).

[0037] The remote system 60 (e.g., a server, a cloud computing environment) also includes data processing hardware 62 and memory hardware 64 for storing instructions that, when executed on the data processing hardware 62, cause the data processing hardware 62 to perform operations. In some embodiments, the execution of the photo sharing system 200 is shared across vehicle 10 and / or remote system 60. See below for reference. Figure 2 and 4As described in more detail in section -6, the photo-sharing system 200 executes a photo-sharing model 210, which includes a user identifier model 220, a sharing determiner model 500, and an inference model 240. In some embodiments, the photo-sharing model 210 may access a user data storage 250, which records / stores a photo identifier 252 of user 222 (e.g., created during the account registration process), a photo 254 belonging to user 222, a filter 256 for photo 254, and / or a previous user context 22 of user 222. The user data storage 250 may be stored on any of the memory hardware 24, 64. The photo-sharing model 210 is configured to receive image data 20 of the environment 104 of vehicle 10, the user context 22 of user 222, and the vehicle context 24 of vehicle 10, and, based on the user context 22 and the vehicle context 24, display the photo 254 associated with the user account of user 222 on the screen 32 of the user interface 30.

[0038] refer to Figure 1 , Figure 2 and Figure 4 The user identifier model 220 can be configured to receive image data 20 captured by the imaging system 16. Specifically, the imaging system 16 may include an indoor camera 16a (such as an electrical camera or security system) located inside the building 106 where the user 222 is located, an outdoor camera 16b (such as a security camera) mounted on the exterior surface of the building 106, a vehicle exterior camera 16c mounted on the exterior surface of the vehicle 10 and configured to capture the environment 104 from the perspective of the vehicle 10, and a cabin camera 16d configured to capture the interior of the vehicle 10 (such as the position and eye movements of the user 222). Here, the cabin camera 16d can be positioned to optimally view the user 222 and verify the user 222's identity. However, identifying the user 222 before the user enters the cabin of the vehicle 10 is crucial for ensuring the presence of additional applications (e.g., Android It is crucial that the photo 254 be displayed in a timely manner before the system (etc.) takes over control of the user interface 30. Thus, the photo-sharing system 200 relies on cameras 16a-16c to identify user 222 before user 222 enters the compartment of vehicle 10. Since each of the cameras 16a-16c in the imaging system 16 has a different viewpoint that is not the optimal angle for identifying user 222, each camera 16a-16c alone may not be able to identify user 222. In other words, each of the cameras 16a-16c may collect image data 20 containing one or more segments of object 102; however, the cameras 16a-16c may not individually identify / recognize object 102 as user 222. It should be understood that the imaging system 16 may include any number of cameras 16 that collect image data 20 for identifying user 222 based on the trajectory of the user approaching vehicle 10.

[0039] Continuing this example, as the user identifier model 220 receives image data 20 from the corresponding cameras 16-16c, it continuously calculates the probability that an object 102 in the image data 20 is user 222 of vehicle 10. For example, during the initialization process of vehicle 10, user 222 may have registered a user account that includes a photo identifier 252 of user 222. As used herein, the photo identifier 252 of user 222 may refer to a high-order feature representation (e.g., embedding) of user 222's facial features, which the user identifier model 220 compares with the image data 20 when determining whether object 102 represents user 222. For example, the user identifier model 220 may extract features from object 102 in the image data 20 and attempt to align the extracted features of object 102 with one or more of the photo identifiers 252 in user data storage 250. When the alignment between the extracted features of object 102 and the photo identifier 252 in user data storage 250 exceeds a probability threshold, user identifier model 220 can identify / identify object 102 as user 222.

[0040] Special Reference Figure 1 and Figure 4User identifier model 220 can perform user identification on image data 20 using a probabilistic method based on incoming image data 20 collected by camera 16 previously passed by user 222. Here, and as mentioned above, each of cameras 16a to 16c may not be able to independently identify object 102 as user 222, and therefore, cameras 16a to 16c can compensate for each other using a probabilistic method. When object 102 approaches vehicle 10, and when object 102 passes each camera 16, user identifier model 220 can perform user identification on object 102 and generate a corresponding confidence level 410 that object 102 is user 222 of vehicle 10. When the corresponding confidence level 410 of a particular camera 16 exceeds a threshold confidence level, user identifier model 220 can identify object 102 as user 222 and initiate / trigger the model (i.e., sharing determiner model 500 and inference model 240) to begin further processing of photo sharing of the identified user 222 approaching vehicle 10.

[0041] refer to Figure 4 Example flowchart 400 shows the trajectory of object 102 as it moves through environment 104 and approaches vehicle 10. Here, object 102 may pass by indoor camera 16a, outdoor camera 16b, vehicle exterior camera 16c, and cabin camera 16d. At each time step, and for each camera 16a-16c, user identifier model 220 can receive image data 20 from each corresponding camera 16 and calculate the probability (i.e., confidence level 410) that the image data 20 including object 102 corresponds to registered user 222.

[0042] In this example, at t=1, registered user 222 can leave building 106 housing indoor camera 16a. User identifier model 220 can receive image data 20 detected by indoor camera 16a and calculate a confidence level 410a that object 102 in image data 20 is registered user 222. If the confidence level 410a of indoor camera 16a does not exceed a threshold confidence level, then at t=2, when registered user 222 passes outdoor camera 26b, user identifier model 220 can receive image data 20 detected by outdoor camera 16b and the confidence level 410a of indoor camera 16a, and calculate a confidence level 410b that object 102 in image data 20 is registered user 222. Here, user identifier model 220 adjusts the confidence level 410b conditionally based on the confidence level 410a calculated for indoor camera 16a. If the confidence level 410b of the outdoor camera 16b does not exceed the threshold confidence level, then at t=3, when the registered user 222 approaches the vehicle exterior camera 16c installed outside the vehicle 10, the user identifier model 220 can receive the image data 20 detected by the vehicle exterior camera 16c and the corresponding confidence levels 410a and 410b of cameras 16a and 16b, and calculate the confidence level 410c that the object 102 in the image data 20 is the registered user 222. Here, the user identifier model 220 adjusts the confidence level 410c based on the corresponding confidence levels 410a and 410b calculated for the previous cameras 16a and 16b. If the confidence level 410c of the external camera 16c does not exceed the threshold confidence level, then at t=4, when the registered user 222 enters the compartment of the vehicle 10, the user identifier model 220 can receive the image data 20 detected by the internal camera 16d and the corresponding confidence levels 410a-410c of the cameras 16a-16b, and calculate the confidence level 410d that the object 102 in the image data 20 is the registered user 222.

[0043] In some implementations, because the vehicle interior camera 16d is optimally located to identify the registered user 222, the user identifier model 220 can use the corresponding confidence level 410d associated with the vehicle interior camera 16d to perform a posterior probability update of the probabilistic model for future identification of the user 222 of the vehicle. For example, the user identifier model 220 can update the conditional probability of the outdoor camera 16b based on the confirmation of the vehicle interior camera 16d. Figure 4 In the example shown, the corresponding confidence level 410d of the outdoor camera 16b may exceed the confidence threshold (e.g., 0.7), and therefore, the user identifier model 220 triggers the photo sharing model 210 to continue determining whether photo sharing is suitable for the identified specific user 222.

[0044] refer to Figure 2 and Figure 5The sharing determiner model 500 is configured to receive the identity of the identified user 222 and the vehicle context 24 of the vehicle 10, and determine whether photo sharing is appropriate based on the vehicle context 24. In other words, the sharing determiner model 500 uses the vehicle context 24 of the vehicle 10 to determine whether to display photo 254 to the user 222. As used herein, the vehicle context 24 can refer to any state or system of the vehicle 10, whether active or inactive, such as, but not limited to, the connectivity of the vehicle 10, the battery life and / or fuel level of the vehicle 10, the gear position of the vehicle 10, and / or whether other applications are currently controlling the systems of the vehicle 10. (Specific Reference) Figure 5 At operation 510, the shared determiner model 500 receives the identity of the identified user 222, thereby triggering the shared determiner model 500 to perform an operation. At operation 520, the shared determiner model 500 determines whether the vehicle situation indicates that the user interface 30 is powered on. When the vehicle situation 24 indicates that the user interface 30 of the vehicle 10 is currently not powered on, at operation 530, the shared determiner model 500 enters a waiting / standby mode.

[0045] If vehicle scenario 24 detects that user interface 30 is powered on, the sharing determiner model 500 proceeds to operation 540, which determines whether vehicle scenario 24 indicates that vehicle 10 can access the Internet (e.g., network 40). At operation 550, when vehicle scenario 24 indicates that vehicle 10 is not connected to network 40, photo sharing model 210 can display photo 254 on a mobile device (not shown) associated with user 222. Conversely, if the sharing determiner model 500 determines that vehicle scenario 24 indicates that vehicle 10 is connected to network 40, at operation 560, the sharing determiner model 500 determines whether vehicle scenario 24 indicates that vehicle 10 is parked. When vehicle scenario 24 indicates that vehicle 10 is not parked, at operation 570, the sharing determiner model 500 determines that photo sharing is inappropriate because it may distract user 222 of vehicle 10.

[0046] Conversely, when vehicle scenario 24 indicates that vehicle 10 is parked, the sharing determiner model 500 proceeds to operation 580, which determines whether vehicle scenario 24 indicates that user interface 30 is currently not occupied by other applications (i.e., available). At operation 590, when vehicle scenario 24 indicates that user interface 30 is currently being used by other preferred applications of vehicle 10, photo sharing model 210 can display photo 254 on the mobile device associated with user 222. Conversely, when vehicle scenario 24 indicates that user interface 30 is not currently being used by other applications of vehicle 10, the sharing determiner model 500 determines that photo sharing is appropriate and proceeds to select and display photo 254 in user interface 30 at operation 590.

[0047] In some implementations, the sharing determiner model 500 utilizes additional context from user 222 when determining whether photo sharing is appropriate. For example, user 222's position relative to user interface 30. Here, if user 222 is sitting in a rear passenger seat (e.g., the third row), user 222 may not have a view of user interface 30. Based on user 222's position, the sharing determiner model 500 can determine that delivering photo 254 to user 222's mobile device instead of user interface 30 is appropriate. Furthermore, in some implementations, additional context from user 222 may include whether user 222 is asleep and unable to view photo 254. In these examples, the sharing determiner model 500 can determine that photo sharing is inappropriate because user 222 is not awake to view photo 254. In these implementations, the vehicle 10's in-vehicle intelligent algorithm can detect the additional context of user 222.

[0048] After the sharing determiner model 500 determines that photo sharing is appropriate based on the vehicle context 24 of vehicle 10, the inference model 240 can retrieve the photo 254 associated with the user account of registered user 222 based on the user context 22 and the vehicle context 24. Here, the user context 22 can refer to known facts, such as the current state, location, or environment of user 222, and the user's historical routines, such as where user 222 has driven vehicle 10 and / or what user 222 has done at a particular location. The inference model 240 receives the user context 22 and the vehicle context 24, and infers which photo 254 of the user account should be displayed to user 222 via the user interface 30 based on the user context 22 and the vehicle context 24.

[0049] refer to Figure 6In some implementations, user data storage 250 stores historical user context 22 and / or vehicle context 24 associated with user 222's account as knowledge scope 260. Knowledge scope 260 may include the historical user context 22 and vehicle context 24 of the user account encoded as an ontology in a knowledge graph using semantic network terminology. As shown, knowledge scope 260 may include time-insensitive knowledge 262, slowly changing knowledge 264, and time-sensitive knowledge 266. Time-insensitive knowledge 262 may refer to information such as the relationships between user 222 and other users (e.g., friends or family). For example, registered user 222 may be Bob, and time-insensitive knowledge 262 may be that user 222 Bob has a wife named Sheila and a daughter named Colin. Slowly changing knowledge 264 may refer to relatively static events, such as when registered user 222 Bob usually eats lunch, the location of Bob's office, or the location of his daughter Colin's school. Time-sensitive knowledge 266 can typically refer to a specific schedule of user 222 Bob, such as, but not limited to, the time Bob spends in his office, or the time Bob usually picks up his daughter Colin from school. It is noteworthy that while the inference model 240 can always base its inference about which photo 254 to select for a user account on time-independent knowledge 262 and slowly changing knowledge 264, the inference model 240 can choose not to apply time-sensitive knowledge 266 outside the time window relevant to it. For example, if Colin's school pick-up is between 4 pm and 5 pm, the inference model 240 can ignore the time-sensitive knowledge 266 reminding Bob to pick up Colin outside of those times and apply only time-independent knowledge 262 and slowly changing knowledge 264, and further apply time-sensitive knowledge 266 that is only relevant to the current time window, different from the school pick-up time window.

[0050] In some cases, a knowledge graph of knowledge scope 260 can be developed from various data streams. For example, knowledge scope 260 can align and merge data streams from proprietary knowledge bases, such as ontology update application programming interfaces, crawlers from service sites, updates to private facts (e.g., birthdays, weddings, graduations, etc.) collected by custom interfaces (such as websites and / or applications designed by the manufacturer of vehicle 10), and / or reminders and calendar information stored in event logging applications associated with user 222. For example, the picking up and dropping off of children can be encoded as an ontology in the knowledge graph using time terms (e.g., 5 pm). The location where user 222 has traveled can be monitored by the trajectory of vehicle 10 and encoded in the knowledge graph. Similarly, points of interest (e.g., home, coffee shop, gym) can be encoded in the knowledge graph using GPS and / or map data. Passengers of vehicle 10 can be determined by the internal camera 16d and also encoded in the knowledge graph to record who user 222 is usually traveling with in vehicle 10.

[0051] Continue to refer to Figure 2 and Figure 3 The inference model 240 receives a user context 22 including a knowledge scope 260 and a vehicle context 24, retrieves a photo 254 associated with the user account of user 222, and displays the photo 254 on a screen 32 communicating with the data processing hardware 12, 62. In some embodiments, the inference model 240 further retrieves a filter 256 for the photo 254. Here, the filter 256 for the photo 254 can be selected based on the user context 22 (e.g., knowledge scope 260). For example, the inference model 240 can apply rule-based inference to determine not only which photo 254 to retrieve, but also which filter 256 to apply to the photo 254 (if any).

[0052] Special Reference Figure 2 Vehicle context 22 can indicate that the current date is the birthday of user 222 Bob's daughter Colin (i.e., time-independent knowledge 262), user 222 Bob is currently in vehicle 10, and it is 4:35 PM (i.e., within the time-sensitive knowledge 266 of Colin's school pick-up between 4 and 5 PM). Inference model 240 can determine that a rule is satisfied to display a photo 254 of Colin on screen 32 of user interface 30, and apply a birthday filter 256 to photo 254 to remind user 222 Bob to pick up Colin from school. As shown, filter 256 includes balloons covering the photo 254 of user 222 Bob's daughter Colin crawling. It should be understood that although birthday rules are described herein, inference model 240 can apply any number of rules to remind user 222 of something, such as, for example, when to stop lunch by displaying a photo 254 of food on user interface 30, and simply to bring joy to user 222 by showing a photo 254 that is relevant to user context 22.

[0053] Figure 7 A flowchart illustrating an example arrangement of a method 700 for personalized photo sharing on screen 32 of display 30 in vehicle 10. Data processing hardware (e.g., Figure 1 Data processing hardware 12, 62) can execute data stored in memory hardware (e.g., Figure 1 An example arrangement of instructions on the memory hardware (14, 64) to perform the operation of method 700. At operation 702, method 700 includes receiving image data 20 captured by the sensor system 16 of vehicle 10. Here, image data 20 includes an object 102 approaching vehicle 10.

[0054] At operation 704, method 700 further includes identifying object 102 in image data 20 as user 222 of vehicle 10. Method 700 further includes user scenario 22 of receiving user 222 of vehicle 10 at operation 706. At operation 708, method 700 further includes vehicle scenario 24 of receiving vehicle 10.

[0055] At operation 710, method 700 further includes determining, based on vehicle context 24, that photo sharing is appropriate. At operation 712, method 700 further includes retrieving, based on user context 22 and vehicle context 24, a photo 254 associated with user account 222. Method 700 further includes, at operation 714, displaying, on screen 32 communicating with data processing hardware 12, 62, the photo 254 associated with user account 222.

[0056] Many embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the appended claims.

[0057] The foregoing description is provided for illustrative and descriptive purposes. It is not intended to be exhaustive or limiting of this disclosure. Elements or features of a particular configuration are generally not limited to that particular configuration, but are interchangeable where applicable and can be used in selected configurations, even if not specifically shown or described. They can also be varied in many ways. Such variations should not be considered as departing from this disclosure, and all such modifications are intended to be included within the scope of this disclosure.

Claims

1. A computer-implemented method that when executed on data processing hardware, causes the data processing hardware to perform operations comprising: receiving image data captured by a sensor system, the image data including an object proximate to a vehicle; identifying the object in the image data as a user of the vehicle; receiving a user context of the user of the vehicle; receiving a vehicle context of the vehicle; determining that photo sharing is appropriate based on the vehicle context; retrieving photos associated with a user account of the user based on the user context and the vehicle context; and displaying the photos associated with the user account on a screen in communication with the data processing hardware.

2. The method of claim 1, wherein, displaying the photos associated with the user account on the screen in communication with the data processing hardware includes applying a filter to the photos based on the user context.

3. The method of claim 1, wherein, the user context includes a knowledge horizon.

4. The method of claim 3, wherein, the knowledge horizon includes one of time-agnostic, slow-changing, or time-sensitive.

5. The method of claim 3, wherein, retrieving the photos associated with the user account of the user based on the user context includes selecting the photos based on the knowledge horizon of the user context.

6. The method of claim 1, wherein determining that photo sharing is appropriate based on the vehicle context includes determining that the vehicle context indicates one or more of: the vehicle is parked; the vehicle is connected to a network; the screen in communication with the data processing hardware is open; or the screen in communication with the data processing hardware is available.

7. The method of claim 1, wherein, the sensor system includes a plurality of cameras.

8. The method of claim 7, wherein, identifying the object in the image data as the user of the vehicle includes, for each camera of the plurality of cameras, generating a respective confidence that the object in the image data is the user of the vehicle.

9. The method of claim 8, wherein, identifying the object in the image data as the user of the vehicle includes determining that a respective confidence of a camera of the plurality of cameras exceeds a threshold.

10. The method of claim 1, wherein, identifying the object in the image data as the user of the vehicle includes recognizing the object as a registered user.

Images