Method and apparatus for object identification and search scope adjustment

Abstract

To effectively identify an object of interest to an occupant of a vehicle.SOLUTION: The method includes processing a query to identify an object external to the vehicle, the query being made by an occupant of the vehicle. The method also includes determining a focus direction of the occupant when the query is received. The method further includes, in response to the query, setting one or more search areas extending from the vehicle based on the focus direction of the occupant. In addition, the method includes adjusting the one or more search areas based on the focus direction of the occupant. The method also includes setting one or more geo-fences corresponding to the one or more search areas for the vehicle on a geographic map in response to the query. The method further includes searching map data associated with the geographic map to identify candidate objects of interest within the one or more geo-fences.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] Device and vehicle manufacturers are constantly challenged to provide products and services that provide value and benefits to users, such as vehicle occupants. Vehicle occupants often see objects or points of interest from their vehicles while driving and inquire about what these objects or points of interest are. Some vehicles have infotainment or navigation systems that utilize on-board cameras and / or sensors to capture gestures made by the vehicle occupant and / or track the eye movements of the vehicle occupant. Summary of the Invention

[0002] An aspect of the present disclosure relates to a method. The method includes processing a query for identifying an object outside the vehicle, the query being made by an occupant of the vehicle. The method also includes determining a focus direction of the occupant when the query is received. The method further includes, in response to the query, setting one or more search areas extending from the vehicle based on the focus direction of the occupant. Additionally, the method includes adjusting a size or shape of the one or more search areas based on the focus direction of the occupant. The method also includes, in response to the query, setting one or more geographic fences corresponding to the one or more search areas on a geographic map relative to the vehicle. The method further includes searching map data associated with the geographic map to identify candidate objects of interest within the one or more geographic fences. Additionally, the method includes searching a database for information about candidate objects of interest within the one or more geographic fences. The method also includes, in response to the query, generating an output including information about the candidate objects of interest.

[0003] An aspect of the present disclosure relates to an apparatus. The apparatus includes a processor and a memory, the memory having instructions stored therein that, when executed by the processor, cause the apparatus to perform the steps of processing a query to identify an object outside the vehicle, the query being made by an occupant of the vehicle. The apparatus also performs the step of determining a focus direction of the occupant when receiving the query. The apparatus further performs the step of setting one or more search areas extending from the vehicle based on the focus direction of the occupant in response to the query. In addition, the apparatus performs the step of adjusting the size or shape of the one or more search areas based on the focus direction of the occupant. In response to the query, the apparatus also performs the step of setting one or more geographic fences corresponding to the one or more search areas for the vehicle on a geographic map. The apparatus further performs the step of searching map data associated with the geographic map to identify candidate objects of interest within the one or more geographic fences. In addition, the apparatus performs the step of searching a database for information about candidate objects of interest within the one or more geographic fences. In response to the query, the apparatus also performs the step of generating an output including information about the candidate objects of interest.

[0004] An aspect of the present specification relates to a non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor, cause an apparatus to perform the steps of processing a query to identify an object outside the vehicle, the query being made by an occupant of the vehicle. The apparatus also performs the step of determining a focus direction of the occupant when receiving the query. The apparatus further performs the step of setting one or more search areas extending from the vehicle based on the focus direction of the occupant in response to the query. In addition, the apparatus performs the step of adjusting the size or shape of the one or more search areas based on the focus direction of the occupant. In response to the query, the apparatus also performs the step of setting one or more geographic fences corresponding to the one or more search areas for the vehicle on a geographic map. The apparatus further performs the step of searching map data associated with the geographic map to identify candidate objects of interest within the one or more geographic fences. In addition, the apparatus performs the step of searching a database for information about candidate objects of interest within the one or more geographic fences. In response to the query, the apparatus also performs the step of generating an output including information about the candidate objects of interest. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a flowchart of a method for adjusting a search scope to identify objects of interest to a vehicle occupant, according to one or more embodiments. [Figure 2] FIG. 2 is a perspective view of a vehicle according to one or more embodiments. [Figure 3] FIG. 3 is a graphical user interface according to one or more embodiments. [Figure 4] FIG. 4 is a graphical user interface according to one or more embodiments. [Figure 5] FIG. 5 is a block diagram of a system for adjusting a search scope to identify objects of interest to a vehicle occupant, according to one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0006] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. It should be noted that, in accordance with standard practice in the industry, various features have not been drawn to scale. In fact, the dimensions of various features may be arbitrarily increased or decreased for clarity of discussion.

[0007] The following disclosure provides many different embodiments or examples for implementing various features of the provided subject matter. Specific examples of components, values, operations, materials, arrangements, or equivalents thereof are described below to simplify the disclosure. It should be understood that these are merely examples and are not intended to be limiting. Other components, values, operations, materials, arrangements, or equivalents thereof are contemplated. For example, the formation of a first feature above or on a second feature in the following description includes embodiments in which the first and second features are formed in direct contact with each other, as well as embodiments in which an additional feature is formed between the first and second features such that the first and second features are not in direct contact with each other. Additionally, the present disclosure may repeat reference numerals and / or letters in various examples. This repetition is for purposes of simplicity and clarity and does not, in itself, dictate a relationship between the various embodiments and / or configurations discussed.

[0008] Additionally, spatially relative terms such as "below," "below," "below," "above," "on top," and the like may be used herein to describe the relationship of one element or feature to another element or feature when shown in the figures for ease of description. Spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors may likewise be interpreted accordingly.

[0009] Vehicle occupants often see objects or points of interest from their vehicles while driving and inquire about what these objects or points of interest are. Some vehicles have infotainment or navigation systems that can search for addresses associated with points of interest based on user input. Some infotainment or navigation systems have one or more displays and navigation capabilities and receive user input from vehicle occupants via a touchscreen, microphone, buttons, knobs, joystick, trackpad, motion sensors, cameras, or other suitable controllers.

[0010] However, problems arise for infotainment or navigation systems that provide responses to vehicle occupant queries regarding objects and / or points of interest because the object or point of interest identification function may fail to detect and / or identify the object or point of interest on the map if the query concerns an object or point of interest that is outside the search area or range. In some situations, the failure may occur based on the direction of travel. For example, if the object is on the right side of the vehicle, the object is located relatively farther from the vehicle or driver than if the object is on the left side of the vehicle. This problem arises depending on the number of lanes between the vehicle and the object, and / or the side of the vehicle on which the driver is sitting, and / or the distance between the vehicle and the side of the road.

[0011] In some countries, cars usually drive on the left side of the road with the driver sitting on the right side of the car. In other countries, cars usually drive on the right side of the road with the driver sitting on the left side of the car. Therefore, if an object is located on the side of the road opposite to the side on which the car is driving and the search range for the area on the left side of the car is the same as the search range for the area on the right side of the car, the search range may not be large enough to identify the object of interest when the object is located on the side of the road opposite to the side on which the car is driving, for example, because there is at least one extra lane between the vehicle and the object on the other side of the road compared to the nearby side of the road close to the lane on which the vehicle is driving.

[0012] The present disclosure includes methods and systems for adjusting a search scope to identify objects of interest to vehicle occupants.

[0013] FIG. 1 is a flowchart of a method 100 for adjusting a search scope to identify objects of interest to a vehicle occupant, according to one or more embodiments.

[0014] In some embodiments, method 100 is implemented using system 500 (FIG. 5). In some embodiments, method 100 is implemented using a system other than system 500 (FIG. 5).

[0015] In some embodiments, method 100 is implemented in vehicle 200 (FIG. 2). In some embodiments, method 100 is implemented in a vehicle other than vehicle 200 (FIG. 2).

[0016] In some embodiments, method 100 includes detecting a query about an object of interest that a vehicle occupant is viewing based on user input, such as a detected pointing gesture, gaze direction, verbal query, or some other suitable manner in which a vehicle occupant may interact with the vehicle's infotainment system or navigation system.

[0017] In operation 101, a query to identify an object outside the vehicle is processed. The query is made by an occupant of the vehicle. In some embodiments, the query is based at least in part on a detected pointing gesture by the occupant of the vehicle. In some embodiments, a gaze direction of the occupant of the vehicle is detected. In some embodiments, the query is based on a verbal command or words spoken out loud by the occupant of the vehicle.

[0018] In some embodiments, the vehicle occupant is the driver of the vehicle. In some embodiments, the vehicle occupant is a passenger of the vehicle. In some embodiments, the driver of the vehicle sits behind the steering wheel of the vehicle. In some embodiments, the vehicle passenger sits in a seat other than the driver's seat, such as a front passenger seat or a rear passenger seat. In some embodiments, the vehicle occupant is any passenger of the vehicle who is sitting or standing in the vehicle.

[0019] In some embodiments, one or more of the pointing motion or gaze direction are detected by one or more sensors or cameras in the vehicle. In some embodiments, the vehicle sensor or camera detecting the pointing motion and / or gaze direction of the vehicle occupant is an interior sensor or camera, such as a sensor or camera facing the interior of the vehicle. In some embodiments, the vehicle sensor or camera detecting the pointing motion and / or gaze direction of the vehicle occupant is an outward-facing sensor or camera, such as a sensor or camera located inside the vehicle cabin and facing outward from the cabin. In some embodiments, the interior and exterior vehicle sensors or cameras detecting the pointing motion and / or gaze direction of the vehicle occupant are physically located inside the vehicle. In some embodiments, the interior vehicle sensor or camera detecting the pointing motion and / or gaze direction of the vehicle occupant is located on the exterior of the vehicle facing the interior of the vehicle. In some embodiments, the exterior vehicle sensor or camera detecting the pointing motion and / or gaze direction of the vehicle occupant is located on the exterior of the vehicle facing outward from the cabin. In some embodiments, the pointing and / or gaze direction is detected by a combination of an internal vehicle sensor or camera that detects the pointing and / or gaze direction of the vehicle occupant and an external vehicle sensor or camera that detects the pointing and / or gaze direction of the vehicle occupant.

[0020] In some embodiments, verbal commands or words spoken by a vehicle occupant are included in the context data, which is audio data received by a microphone associated with the vehicle, within a predetermined time period that includes the time the pointing gesture and / or gaze direction is detected. In some embodiments, the microphone is an interior microphone configured to capture sounds within the vehicle cabin. In some embodiments, the microphone is an exterior microphone configured to capture sounds outside the vehicle. In some embodiments, the microphone is configured to capture audio commands as user input. In some embodiments, the context data includes one or more of a conversation between two or more vehicle occupants, a conversation between a vehicle occupant and a person outside the vehicle, such as via a telephone, video call, or other suitable form of communication, a conversation between a vehicle occupant and a person outside the vehicle, such as through a window or through the vehicle's exterior speakers, a verbal inquiry made by a vehicle occupant, music, other sounds, externally captured sounds, or other suitable interior or exterior sounds that can be captured by one or more microphones associated with the vehicle.

[0021] In operation 103, the focus direction of the occupant when the query is received is determined.

[0022] In operation 105, one or more search areas are established extending from the vehicle in response to the query and based on the occupant's focus direction.

[0023] In operation 107, one or more search areas are adjusted based on the occupant's focus direction.

[0024] In some embodiments, the first side of the road on which the vehicle is traveling is the side of the road closest to the vehicle, and the second side of the road is located opposite the first side of the road with the vehicle located between the first side of the road and the second side of the road, a first search area extends from the vehicle toward the first side of the road, a second search area extends from the vehicle toward the second side of the road, a third search area extends from the vehicle in a direction of travel of the vehicle, and a fourth search area extends from the vehicle in a direction opposite to the direction of travel of the vehicle.

[0025] In some embodiments, traffic rules are processed to identify which side of the road the vehicle is traveling on is legally permitted to travel on. In some embodiments, the traffic rules are obtained by querying a database associated with the vehicle, an infotainment system, a navigation system, or other suitable system that allows the vehicle to access traffic laws corresponding to various regions upon query. In some embodiments, local traffic laws are stored locally in the vehicle's memory or communicated to the vehicle via a wired or wireless connection.

[0026] In some embodiments, the first side of the road is the side of the road on which the vehicle is legally permitted to travel, and the second side of the road is located opposite the first side of the road, with the vehicle located between the first side of the road and the second side of the road.

[0027] In some embodiments, the first search area is smaller than the second search area. In some embodiments, the third search area and the fourth search area are larger than at least one of the first search area or the second search area. In some embodiments, one or more of the search areas are each a triangle with each vertex of the triangle located at the vehicle.

[0028] In some embodiments, the one or more search areas are adjusted based on vehicle speed to accommodate the vehicle speed. For example, based on an increase in vehicle speed, the one or more search areas increase in size from a first set size to a different size based on the vehicle speed. In some embodiments, the one or more search areas are adjusted based on a lane change of the vehicle to accommodate a lane change. For example, based on a detected lane change, the one or more search areas increase or decrease in size accordingly from a first set size based on the detected lane change.

[0029] In some embodiments, one or more search areas are reduced in size based on the detected lane change and the lane the vehicle is traveling in. For example, if the vehicle moves into the right lane and the vehicle occupant points to the right, the size of the search area toward the right side of the vehicle is reduced. Similarly, if the vehicle moves into the right lane and the vehicle occupant points to the left, the size of the search area toward the left side of the vehicle is increased.

[0030] In some embodiments, the first search area extends beyond a first side of the road by a first distance, and the second search area extends beyond a second side of the road by a second distance, the first distance being equal to the second distance.

[0031] In some embodiments, the size of the search area is adjusted based on one or more of the direction the vehicle occupant is pointing or the direction the vehicle occupant is looking, for example, in countries where vehicles drive on the left, if the vehicle occupant points to the right side of the road, the search area is increased to extend beyond the right edge of the road.

[0032] In some embodiments, if the vehicle occupant points a finger in the direction the vehicle is traveling, in the direction the vehicle is facing, behind the vehicle opposite the direction the vehicle is traveling, or in the direction opposite the direction the vehicle is facing, the search area is increased to facilitate searching for objects of interest that are far away.

[0033] In some embodiments, one or more search areas are adjusted to accommodate changes in the direction of vehicle movement, increasing or decreasing in size accordingly from a first set size based on detected lane changes.

[0034] In some embodiments, the first search area has a first default size, the second search area has a second default size, a first distance between a first side of the road and the vehicle is detected, and a second distance between a second side of the road and the vehicle is detected, and adjusting the one or more search areas includes, in response to determining that the first distance is also greater than the second distance amount, increasing the size of the first search area from the first default size such that the first search area is larger than the first default size and the second search area is larger than a second default size.

[0035] In some embodiments, the first search area has a first default size, the second search area has a second default size, the third search area has a third default size, and the fourth search area has a fourth default size, and adjusting the one or more search areas includes adjusting one or more of the first search area, the second search area, the third search area, or the fourth search area by changing a size of the one or more of the first search area, the second search area, the third search area, or the fourth search area to a different size from a corresponding default size of the first default size, the second default size, the third default size, or the fourth default size based on the occupant's focus direction.

[0036] In some embodiments, adjusting the one or more search areas includes adjusting a first search area to extend beyond a first side of the road in response to determining that the occupant's focus direction is toward the first side of the road.

[0037] In some embodiments, adjusting the one or more search areas includes adjusting the first search area to extend beyond the first side of the road and adjusting at least one of the third search area or the fourth search area to extend beyond the first side of the road in response to determining that the occupant's focus direction is toward the first side of the road in an area between the first search area and one of the third search area or the fourth search area.

[0038] In some embodiments, adjusting the one or more search areas includes adjusting the first search area to extend beyond the first side of the road, adjusting the third search area to extend beyond the first side of the road, and adjusting the fourth search area to extend beyond the first side of the road in response to determining that the occupant's focus direction is toward the first side of the road and determining that the vehicle speed exceeds a predetermined threshold. For example, if the vehicle is traveling at a high speed and the occupant of the vehicle points or looks toward an object to inquire or take a photograph, the vehicle may pass the object so quickly that an object located in the third search area may be located in the fourth search area by the time the photograph is taken. Similarly, if the vehicle is traveling at a high speed and the occupant of the vehicle points or looks toward an object to inquire or take a photograph, the vehicle may pass the object so quickly that an object located in the first search area may be located in the fourth search area by the time the photograph is taken.

[0039] In some embodiments, the size of one or more search areas is gradually increased until at least one candidate object of interest is identified in map data searched for information regarding the candidate object of interest. In some embodiments, contextual information is processed as a component for determining how large the search area should be and / or for determining when incremental increases in the search area will capture potential candidate object of interest. For example, if a vehicle occupant points to a distant mountain and talks about it, even though there is a store in the search area between the vehicle and the mountain, the system recognizes that the vehicle occupant is inquiring about the mountain and expands the search area to include the distant mountain.

[0040] In operation 109, in response to the query, one or more geographic fences corresponding to one or more search areas are established for the vehicle on the geographic map.

[0041] In some embodiments, the first search area, the second search area, the third search area, and the fourth search area are all included within a search area corresponding to one or more geographic fences for identifying candidate objects of interest.

[0042] In some embodiments, the first search area, the second search area, the third search area and the fourth search area are all included within a search area corresponding to one or more geographic fences for identifying potential objects of interest and one or more additional potential objects of interest around the vehicle at the time of the query.

[0043] In some embodiments, setting the one or more search areas includes one or more of a first search area, a second search area, a third search area, or a fourth search area based on a focus direction of the occupant, and the system refrains from including one or more of the first search area, the second search area, the third search area, or the fourth search area based on the focus direction of the occupant. In some embodiments, the focus direction of the occupant is based on a detected pointing gesture made by the occupant. In some embodiments, the focus direction of the occupant is based on a detected gaze direction of the occupant. In some embodiments, the focus direction of the occupant is based on a detected pointing gesture made by the occupant and a detected gaze direction of the occupant.

[0044] In operation 111, map data associated with the geographic map is searched to identify candidate objects of interest within one or more geographic fences.

[0045] In operation 113, a database is searched for information about candidate objects of interest within one or more geographic fences.

[0046] In operation 115, in response to the query, an output is generated that includes information about the object candidates.

[0047] In some embodiments, the output is a graphical user interface output by a display of the vehicle or a display of a mobile device communicatively coupled to the vehicle, hi some embodiments, the display is associated with the vehicle's infotainment system or navigation system.

[0048] In some embodiments, the graphical user interface includes a geographic map, a location icon indicating the location on the geographic map where the query was made, graphical objects representing one or more search areas or one or more geographic fences, and descriptive information associated with the candidate objects of interest. The graphical objects extend from the location icons in the graphical user interface.

[0049] In some embodiments, the graphical object representing the one or more search areas or the one or more geographic fences is a polygon. In some embodiments, the graphical object representing the one or more search areas or the one or more geographic fences is a triangle. In some embodiments, the graphical user interface includes a three-dimensional display and the graphical object representing the one or more search areas or the one or more geographic fences is a cone or other suitable shape.

[0050] Those skilled in the art will recognize that modifications to method 100 are within the scope of this disclosure. In some embodiments, method 100 includes at least one additional operation. In some embodiments, the order of operations in method 100 is adjusted.

[0051] 2 is a perspective view of a vehicle 200 according to some embodiments. The vehicle 200 is capable of implementing the method 100 (FIG. 1). In some embodiments, the vehicle 200 is capable of implementing the method 100 (FIG. 1) using a system 500 (FIG. 5) mounted on the vehicle. In some embodiments, the vehicle 200 is capable of implementing the method 100 (FIG. 1) based on receiving instructions from a system 500 (FIG. 5) that is remote or separable from the vehicle 200. In some embodiments where the system 500 (FIG. 5) is remote or separable from the vehicle 200, the vehicle 200 is configured to receive instructions to implement the method 100 (FIG. 1) wirelessly or via a wired connection.

[0052] Vehicle 200 includes one or more vehicle systems for implementing the operation of the vehicle. In some embodiments, the one or more vehicle systems include one or more of an infotainment system or a navigation system having one or more displays 201, one or more interior or exterior sensors 203, one or more interior or exterior cameras 205, and at least one camera 207 with a field of view outside the vehicle for taking pictures. In some embodiments, vehicle 200 includes one or more vehicle systems only in the front portion of the cabin. In some embodiments, vehicle 200 includes one or more vehicle systems in both the front portion of the cabin and the rear portion of the cabin.

[0053] FIG. 3 is a graphical user interface 300 according to one or more embodiments.

[0054] Graphical user interface 300 shows a geographic map 301 with a vehicle 303 on the right side of the road. In response to a query made by an occupant of vehicle 303 regarding a pointing and / or eye direction toward the left side of the road, a search area represented by graphical object 305 expands to encompass potential objects of interest on the left side of the road, such as a coffee cafe 307, a clothing store 309, and a park 311.

[0055] When a query is made regarding the pointing and / or gaze direction toward the right side of the road, the virtual search area represented by graphical object 313 is smaller than the search area represented by graphical object 305, which extends from vehicle 303 toward the left side of the road toward the potential object of interest on the right side of the road.

[0056] Similarly, when a query is made regarding a pointing and / or looking direction toward the front of vehicle 303 in the direction of travel, the virtual search area represented by graphical object 315 is directed toward potential objects of interest in front of vehicle 303, and when a query is made regarding a pointing and / or looking direction toward the rear of vehicle 303 in the opposite direction of travel, the virtual search area represented by graphical object 317 is directed toward potential objects of interest behind vehicle 303.

[0057] In some embodiments, two or more of the graphical objects 305, 313, 315 and 317 are optionally included in the graphical user interface to indicate that the corresponding two or more search areas represented by the graphical objects 305, 313, 315 and 317 are included as the basis of a geographic fence used to identify candidate objects of interest within the map data.

[0058] FIG. 4 is a graphical user interface 400 according to one or more embodiments.

[0059] Graphical user interface 400 includes an image of a candidate object of interest, in this example "Coffee Cafe" 401, and descriptive information 403 associated with the candidate object of interest. In this example, descriptive information 403 includes the name and address of the candidate object of interest. In some embodiments, graphical user interface 400 is displayed based on user input received via graphical user interface 300 (FIG. 3). In some embodiments, graphical user interface 400 is displayed simultaneously with graphical user interface 300.

[0060] The candidate object of interest 401 in the graphical user interface 400 is within the graphical object 305 in the graphical user interface 300 and was identified by searching the map data according to a geographic fence set based on a search area corresponding to the graphical object 305.

[0061] 5 is a block diagram of a system 500 for adjusting a search scope to identify objects of interest to a vehicle occupant, according to one or more embodiments. The system 500 includes a hardware processor 502 and a non-transitory computer-readable storage medium 504 that is encoded with, i.e., stores, computer program code 506, i.e., a set of executable instructions. The computer-readable storage medium 504 is also encoded with instructions 507 for interfacing with a manufacturing machine to produce memory arrays. The processor 502 is electrically coupled to the computer-readable storage medium 504 via a bus 508. The processor 502 is also electrically coupled to an input / output (I / O) interface 510 by the bus 508. A network interface 512 is also electrically coupled to the processor 502 via the bus 508. The network interface 512 is connected to a network 514 such that the processor 502 and the computer-readable storage medium 504 can connect to external elements via the network 514. Processor 502 is configured to execute computer program code 506 encoded on computer-readable storage medium 504 to enable system 500 to perform some or all of the operations described in method 100 (FIG. 1) or implemented by vehicle 200 (FIG. 2).

[0062] In some embodiments, processor 502 is a central processing unit (CPU), a multiprocessor, a distributed processing system, an application specific integrated circuit (ASIC), and / or other suitable processing unit.

[0063] In some embodiments, computer-readable storage medium 504 is an electronic, magnetic, optical, electromagnetic, infrared, and / or semiconductor system (or apparatus or device). For example, computer-readable storage medium 504 includes semiconductor or solid-state memory, magnetic tape, removable computer diskette, random access memory (RAM), read-only memory (ROM), rigid magnetic disk, and / or optical disk. In some embodiments using an optical disk, computer-readable storage medium 504 includes a compact disk-read-only memory (CD-ROM), a compact disk-read / write (CD-R / W), and / or a digital video disk (DVD).

[0064] In some embodiments, storage medium 504 stores computer program code 504 configured to cause system 500 to perform some or all of the operations as described in method 100 ( FIG. 1 ) or implemented by vehicle 200 ( FIG. 2 ). In some embodiments, storage medium 504 also stores information used to perform some or all of the operations as described in method 100 ( FIG. 1 ) or implemented by vehicle 200 ( FIG. 2 ), as well as information generated during the performance of some or all of the operations as described in method 100 ( FIG. 1 ) or implemented by vehicle 200 ( FIG. 2 ), such as input data parameters 516, user profile parameters 518, notification data parameters 520, vehicle state parameters 522, and / or a set of executable instructions for performing some or all of the operations as described in method 100 ( FIG. 1 ) or implemented by vehicle 200 ( FIG. 2 ).

[0065] In some embodiments, storage medium 504 stores instructions 507 for interfacing with an external device, such as a mobile device, that enable processor 502 to generate or receive instructions readable by the external device during performance of some or all of the operations described in method 100 (FIG. 1) or implemented by vehicle 200 (FIG. 2).

[0066] System 500 includes an I / O interface 510. I / O interface 510 is coupled to external circuitry. In some embodiments, I / O interface 510 includes a keyboard, keypad, mouse, trackball, trackpad, touchscreen, and / or cursor direction keys for communicating information and commands to processor 502.

[0067] System 500 also includes a network interface 512 coupled to processor 502. Network interface 512 enables system 500 to communicate with a network 514 to which one or more other computer systems are connected. Network interface 512 may include a wireless network interface such as WiFi, Bluetooth, WiMAX, GPRS, or WCDMA; a wired network interface such as a LAN, Ethernet, WAN, USB, or IEEE-1394; or other suitable network interface. In some embodiments, some or all of the operations described in method 100 (FIG. 1) or implemented by vehicle 200 (FIG. 2) are implemented in more than one system 500, and information such as sensor data, window transmittance, forecast information, or vehicle status is exchanged between the various systems 500 via network 514.

[0068] Supplementary Note 1 An aspect of the present disclosure relates to a method. The method includes processing a query for identifying an object outside the vehicle, the query being made by an occupant of the vehicle. The method also includes determining a focus direction of the occupant when the query is received. The method further includes, in response to the query, setting one or more search areas extending from the vehicle based on the focus direction of the occupant. Additionally, the method includes adjusting a size or shape of the one or more search areas based on the focus direction of the occupant. The method also includes, in response to the query, setting one or more geographic fences corresponding to the one or more search areas on a geographic map relative to the vehicle. The method further includes searching map data associated with the geographic map to identify candidate objects of interest within the one or more geographic fences. Additionally, the method includes searching a database for information about candidate objects of interest within the one or more geographic fences. The method also includes, in response to the query, generating an output including information about the candidate objects of interest.

[0069] Supplementary Note 2 The method of Supplementary Note 1, wherein the occupant focus direction is based on a detected pointing gesture made by an occupant of the vehicle.

[0070] Supplementary Note 3 3. The method of any one of claims 1 to 2, wherein the occupant focus direction is based on a detected line of sight of the vehicle occupant.

[0071] Supplementary Note 4 The method of any one of Supplementary Notes 1 to 3, wherein the one or more search areas are each a triangle, each vertex of the triangle being located on the vehicle.

[0072] Supplementary Note 5 The method of any one of Supplementary Notes 1 to 4, wherein adjusting the size or shape of the one or more search areas is further based on changes in vehicle speed.

[0073] Supplementary Note 6 The method described in any one of Supplementary Notes 1 to 5, wherein the step of adjusting the size or shape of the one or more search areas is further based on lane changes of the vehicle.

[0074] Supplementary Note 7 A method as described in any one of Supplementary Notes 1 to 6, wherein the step of adjusting the size or shape of the one or more search areas is further based on a change in the direction of movement of the vehicle.

[0075] Supplementary Note 8 Supplementary Notes 1 to 7. The method of any one of Supplementary Notes 1 to 7, wherein the one or more search areas include one or more of: a first search area extending from the vehicle toward a first side of a road on which the vehicle is traveling; a second search area extending from the vehicle toward a second side of the road while the vehicle is positioned between the first side of the road and a second side of the road opposite the first side; a third search area extending from the vehicle in a direction of travel of the vehicle; and a fourth search area extending from the vehicle in a direction opposite to the direction of travel of the vehicle.

[0076] Supplementary Note 9 9. The method of Supplementary Note 8, further comprising: setting the one or more search areas to include one or more of the first search area, the second search area, the third search area, or the fourth search area based on a focus direction of the occupant; and refraining from including one or more of the first search area, the second search area, the third search area, or the fourth search area based on a focus direction of the occupant.

[0077] Supplementary Note 10 9. The method of claim 8, wherein the first search area, the second search area, the third search area, and the fourth search area are all included within the search area corresponding to the one or more geographic fences for identifying the candidate object of interest.

[0078] Supplementary Note 11 Supplementary Notes 8-10. The method of any one of Supplementary Notes 8-10, wherein the first search area extends beyond a first side of the road by a first distance, and the second search area extends beyond a second side of the road by a second distance, the first distance being equal to the second distance.

[0079] Supplementary Note 12 12. The method of any one of Supplementary Notes 8 to 11, further comprising processing traffic rules to identify which side of the road the vehicle is legally allowed to travel on, wherein a first side of the road is a side of the road on which the vehicle is legally allowed to travel, and wherein the first search area is smaller than the second search area.

[0080] Supplementary Note 13 13. The method of any one of Supplementary Notes 8 to 12, wherein the third search area and the fourth search area are larger than at least one of the first search area or the second search area.

[0081] Supplementary Note 14 Supplementary Notes 8-13. The method of any one of Supplementary Notes 8-13, wherein the first search area has a first default size and the second search area has a second default size, the method further comprising: detecting a first distance between a first side of the road and the vehicle; and detecting a second distance between a second side of the road and the vehicle, and adjusting the size or shape of the one or more search areas comprises, in response to determining that the first distance is greater than the second distance, increasing the size of the first search area from the first default size such that the first search area is greater than the first default size and the second search area is greater than a second default size.

[0082] Supplementary Note 15 Supplementary Notes 8-14. The method of any one of Supplementary Notes 8-14, wherein the first search area has a first default size, the second search area has a second default size, the third search area has a third default size, and the fourth search area has a fourth default size, and wherein adjusting the size or shape of the one or more search areas comprises adjusting one or more of the first search area, the second search area, the third search area, or the fourth search area by changing a size of the one or more of the first search area, the second search area, the third search area, or the fourth search area to a size different from a corresponding default size among the first default size, the second default size, the third default size, or the fourth default size, based on a focus direction of the occupant.

[0083] Supplementary Note 16 Supplementary Notes 8 to 15, wherein adjusting the size or shape of the one or more search areas includes adjusting the first search area to extend beyond a first side of the road in response to determining that the occupant's focus direction is toward a first side of the road.

[0084] Supplementary Note 17 The method of any one of Supplementary Notes 8 to 16, wherein adjusting the size or shape of the one or more search areas includes adjusting the first search area to extend beyond the first side of the road and adjusting at least one of the third search area or the fourth search area to extend beyond the first side of the road in response to determining that the occupant's focus direction is toward the first side of the road in an area between the first search area and one of the third search area or the fourth search area.

[0085] Supplementary Note 18 The method of any one of Supplementary Notes 8 to 17, wherein the step of adjusting the size or shape of the one or more search areas includes adjusting the first search area to extend beyond the first side of the road, adjusting the third search area to extend beyond the first side of the road, and adjusting the fourth search area to extend beyond the first side of the road in response to determining that the occupant's focus direction is toward the first side of the road and determining that the vehicle speed exceeds a predetermined threshold.

[0086] Supplementary Note 19 An aspect of the present disclosure relates to an apparatus. The apparatus includes a processor and a memory, the memory having instructions stored therein that, when executed by the processor, cause the apparatus to perform the steps of processing a query to identify an object outside the vehicle, the query being made by an occupant of the vehicle. The apparatus also performs the step of determining a focus direction of the occupant when receiving the query. The apparatus further performs the step of setting one or more search areas extending from the vehicle based on the focus direction of the occupant in response to the query. In addition, the apparatus performs the step of adjusting the size or shape of the one or more search areas based on the focus direction of the occupant. In response to the query, the apparatus also performs the step of setting one or more geographic fences corresponding to the one or more search areas for the vehicle on a geographic map. The apparatus further performs the step of searching map data associated with the geographic map to identify candidate objects of interest within the one or more geographic fences. In addition, the apparatus performs the step of searching a database for information about candidate objects of interest within the one or more geographic fences. In response to the query, the apparatus also performs the step of generating an output including information about the candidate objects of interest.

[0087] Supplementary Note 20 An aspect of the present specification relates to a non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor, cause an apparatus to perform the steps of processing a query to identify an object outside the vehicle, the query being made by an occupant of the vehicle. The apparatus also performs the step of determining a focus direction of the occupant when receiving the query. The apparatus further performs the step of setting one or more search areas extending from the vehicle based on the focus direction of the occupant in response to the query. In addition, the apparatus performs the step of adjusting the size or shape of the one or more search areas based on the focus direction of the occupant. In response to the query, the apparatus also performs the step of setting one or more geographic fences corresponding to the one or more search areas for the vehicle on a geographic map. The apparatus further performs the step of searching map data associated with the geographic map to identify candidate objects of interest within the one or more geographic fences. In addition, the apparatus performs the step of searching a database for information about candidate objects of interest within the one or more geographic fences. In response to the query, the apparatus also performs the step of generating an output including information about the candidate objects of interest.

[0088] The foregoing outlines features of several embodiments so that those skilled in the art may better understand aspects of the present disclosure. Those skilled in the art should appreciate that this disclosure may readily be used as a basis for designing or modifying other processes and structures to carry out the same purposes and / or achieve the same advantages as the embodiments introduced herein. Those skilled in the art should also appreciate that such equivalent structures do not depart from the spirit and scope of the present disclosure, and that various changes, substitutions, and alterations can be made herein without departing from the spirit and scope of the present disclosure.

Claims

1. 1. A processor-implemented method comprising: processing a query to identify an object external to the vehicle, the query being made by an occupant of the vehicle; determining a focus direction of an occupant when the inquiry is received; in response to the query, defining one or more search areas extending from the vehicle based on a focus direction of the occupant; adjusting a size or shape of the one or more search areas based on the occupant's focus direction; in response to the query, establishing one or more geographic fences for the vehicle on a geographic map corresponding to the one or more search areas; searching map data associated with the geographic map to identify candidate objects of interest within the one or more geographic fences; searching a database for information about the candidate objects of interest within the one or more geographic fences; generating an output in response to said query, said output including information about said candidate objects of interest; A method comprising:

2. The method of claim 1 , wherein the occupant focus direction is based on a detected pointing gesture made by an occupant of the vehicle.

3. The method of claim 1 or 2, wherein the occupant focus direction is based on a detected line of sight of the vehicle occupant.

4. The method of claim 1 or 2, wherein each of the one or more search areas is a triangle, each vertex of the triangle being located at the vehicle.

5. The method of claim 1 or 2, wherein adjusting the size or shape of the one or more search areas is further based on changes in vehicle speed.

6. The method of claim 1 or 2, wherein adjusting the size or shape of the one or more search areas is further based on lane changes of the vehicle.

7. The method of claim 1 or 2, wherein adjusting the size or shape of the one or more search areas is further based on a change in the direction of travel of the vehicle.

8. The one or more search areas include: a first search area extending from the vehicle toward a first side of a road on which the vehicle is traveling; a second search area extending from the vehicle toward a second side of the road while the vehicle is positioned between the first side of the road and a second side of the road opposite the first side; a third search area extending from the vehicle in the direction of travel of the vehicle; a fourth search area extending from the vehicle in a direction opposite to the direction of travel of the vehicle; 3. The method of claim 1 or 2, comprising one or more of:

9. setting the one or more search areas to include one or more of the first search area, the second search area, the third search area, or the fourth search area based on a focus direction of the occupant; refraining from including one or more of the first search area, the second search area, the third search area, or the fourth search area based on a focus direction of the occupant; The method of claim 8 further comprising:

10. 9. The method of claim 8, wherein the first search area, the second search area, the third search area, and the fourth search area are all included within the search area corresponding to the one or more geographic fences for identifying the candidate object of interest.

11. 9. The method of claim 8, wherein the first search area extends beyond a first side of the road by a first distance and the second search area extends beyond a second side of the road by a second distance, the first distance being equal to the second distance.

12. processing traffic rules to identify on which side of the road the vehicle is legally permitted to travel; the first side of the road is the side of the road on which the vehicle is legally permitted to travel; The method of claim 8 , wherein the first search area is smaller than the second search area.

13. The method of claim 8 , wherein the third search area and the fourth search area are larger than at least one of the first search area or the second search area.

14. the first search area has a first default size and the second search area has a second default size; The method further comprises: Detecting a first distance between a first side of the road and the vehicle; detecting a second distance between a second side of the road and the vehicle; Including, adjusting the size or shape of the one or more search areas, 9. The method of claim 8, further comprising: in response to determining that the first distance is greater than the second distance, increasing a size of the first search area from the first default size such that the first search area is larger than the first default size and a second default size of the second search area.

15. the first search area has a first default size, the second search area has a second default size, the third search area has a third default size, and the fourth search area has a fourth default size; adjusting the size or shape of the one or more search areas, 9. The method of claim 8, comprising adjusting one or more of the first search area, the second search area, the third search area, or the fourth search area by changing a size of the one or more of the first search area, the second search area, the third search area, or the fourth search area to a size different from a corresponding default size among the first default size, the second default size, the third default size, or the fourth default size based on a focus direction of the occupant.

16. adjusting the size or shape of the one or more search areas, 9. The method of claim 8, further comprising: adjusting the first search area to extend beyond the first side of the road in response to determining that the occupant's focus direction is toward the first side of the road.

17. adjusting the size or shape of the one or more search areas, 9. The method of claim 8, further comprising: adjusting the first search area to extend beyond the first side of the road and adjusting at least one of the third search area or the fourth search area to extend beyond the first side of the road in response to determining that the occupant's focus direction is toward a first side of the road in an area between the first search area and one of the third search area or the fourth search area.

18. adjusting the size or shape of the one or more search areas, 9. The method of claim 8, further comprising: adjusting the first search area to extend beyond the first side of the road, adjusting the third search area to extend beyond the first side of the road, and adjusting the fourth search area to extend beyond the first side of the road in response to determining that the occupant's focus direction is toward a first side of the road and determining that the vehicle speed exceeds a predetermined threshold.

19. 1. An apparatus comprising a processor and a memory, The memory has instructions stored therein that, when executed by the processor, cause the device to: processing a query to identify an object external to the vehicle, the query being made by an occupant of the vehicle; determining a focus direction of an occupant when the inquiry is received; in response to the query, defining one or more search areas extending from the vehicle based on a focus direction of the occupant; adjusting a size or shape of the one or more search areas based on the occupant's focus direction; in response to the query, establishing one or more geographic fences for the vehicle on a geographic map corresponding to the one or more search areas; searching map data associated with the geographic map to identify candidate objects of interest within the one or more geographic fences; searching a database for information about the candidate objects of interest within the one or more geographic fences; generating an output in response to said query, said output including information about said candidate objects of interest; A device that performs the following.

20. processing a query to identify an object external to the vehicle, the query being made by an occupant of the vehicle; determining a focus direction of an occupant when the inquiry is received; in response to the query, defining one or more search areas extending from the vehicle based on a focus direction of the occupant; adjusting a size or shape of the one or more search areas based on the occupant's focus direction; in response to the query, establishing one or more geographic fences for the vehicle on a geographic map corresponding to the one or more search areas; searching map data associated with the geographic map to identify candidate objects of interest within the one or more geographic fences; searching a database for information about the candidate objects of interest within the one or more geographic fences; generating an output in response to said query, said output including information about said candidate objects of interest; A computer program that causes a processor to execute the following.

Images