Object information acquisition method and system for implementing the same

Abstract

The present disclosure relates to an object information acquisition method and an object information acquisition system for implementing the method. The method of acquiring object information includes receiving a request from an occupant of a vehicle. The method includes receiving the request from the occupant. The method includes determining a content of the request from the occupant. The method also includes detecting a line-of-sight position of the occupant. The method includes receiving information about an environment around the vehicle based on data collected by a sensor mounted to the vehicle. The method includes identifying a region of interest, ROI, outside the vehicle based on the detected line-of-sight position and the information about the environment around the vehicle. The method includes generating log data based on the ROI and the content of the request. The method includes transmitting the log data to an external device. The method includes receiving information about an object within the ROI that satisfies the content of the request. Thereby, information about an object of interest to an occupant of a vehicle is provided.

Description

[0001] This application relates to the specification of U.S. Patent Application No. 17 / 497846, filed October 8, 2021, which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to a method for obtaining object information and a system for implementing the method. Background Technology

[0003] Vehicle occupants may look out the window at objects of interest. In some cases, occupants may want to identify or learn more about the object. In others, they may use portable devices such as smartphones to capture images of the object and then search for it online to identify or learn more about it. In some situations, capturing images of objects becomes more difficult due to vehicle movement. Furthermore, in some cases, obstacles may pass between the vehicle and the object, obstructing the capture of the image. In some situations, the driver may not be able to safely remove their hands from the steering wheel to use a portable device to capture images.

[0004] In some methods, occupants attempt to identify objects by looking at a map. Furthermore, occupants can search the internet to determine if the object identified using the map is correct, and if so, search for more information about the object. Object identification using maps relies on the occupants' best estimation of the object's position relative to other known landmarks or objects. Summary of the Invention

[0005] The method for obtaining object information according to the implementation includes: receiving a request from a vehicle occupant; receiving the request from the occupant after receiving the request; determining the content of the request from the occupant; detecting the occupant's gaze position; receiving information about the environment surrounding the vehicle based on data collected by sensors installed in the vehicle; identifying a region of interest (ROI) outside the vehicle based on the detected gaze position and the information about the environment surrounding the vehicle; generating log data based on the ROI and the content of the request; sending the log data to an external device; and receiving information related to objects within the ROI that satisfies the request. Attached Figure Description

[0006] Figure 1 This is a block diagram of an object recognition system involved in one of the implementation methods.

[0007] Figure 2 This is a flowchart of an object recognition method involved in one of the implementation methods.

[0008] Figure 3 This is a flowchart of an object recognition method involved in one of the implementation methods.

[0009] Figure 4 This is a diagram of the data structure for occupant requests involved in one of the implementation methods.

[0010] Figure 5 This is a diagram of the data structure of the region of interest data involved in one of the implementation methods.

[0011] Figure 6 This is a diagram of the data structure of the region of interest data involved in one of the implementation methods.

[0012] Figure 7 This is a diagram of the data structure of the region of interest data involved in one of the implementation methods.

[0013] Figure 8 This is a diagram of the user interface involved in one of the implementation methods.

[0014] Figure 9 This is a diagram of the user interface involved in one of the implementation methods.

[0015] Figure 10 This is a diagram of the user interface involved in one of the implementation methods.

[0016] Figure 11 This is a block diagram of a system for performing object recognition according to one of the embodiments. Detailed Implementation

[0017] The technical solutions of this disclosure can be best understood by reading the following detailed description in conjunction with the accompanying drawings. It should be noted that, according to standard practice in the art, the various features are not depicted to a fixed scale. In fact, the dimensions of the various features may be arbitrarily increased or decreased for clarity of description.

[0018] The following disclosure provides numerous different implementations or embodiments of the various features of the provided subject matter. To simplify this disclosure, specific examples of constituent elements, values, actions, materials, configurations, or similar objects are described below. These are merely examples and are not intended to be limiting. Other constituent elements, values, actions, materials, configurations, or similar objects may be considered. For example, in the following description, forming a first feature above or on a second feature may include implementations where the first and second features are in direct contact; additionally, implementations may sometimes include implementations where an additional feature is formed between the first and second features so that the first and second features are not in direct contact. Furthermore, this disclosure sometimes repeats reference numerals and / or reference words in various examples. This repetition is for the purpose of brevity and clarity and does not, in itself, determine the relationship between the various implementations and / or structures discussed.

[0019] Occupants in a moving vehicle often have to exert considerable effort to identify objects of interest. In some cases, occupants cannot accurately identify objects based on either a map or a captured image. In other cases, occupants such as drivers cannot use maps or camera devices like smartphones to attempt to identify objects of interest. To assist occupants in accurately identifying objects of interest, the object recognition method in this specification requests a start command along with line-of-sight data and vehicle sensor data for object recognition. In some embodiments, information about the identified object, such as operating hours and historical data, is also provided.

[0020] By utilizing gaze data, the method can determine the direction an occupant is looking. Combining gaze data with map data and / or vehicle sensor data, it determines what object the occupant is observing at the time the request is initiated. Utilizing request initiation helps reduce processing load and data transmission between the vehicle and external devices such as servers. In one implementation, request initiation includes keywords received from the occupant via voice signals. In another implementation, request initiation includes detecting a predetermined gesture by the occupant. In yet another implementation, request initiation includes receiving input from a user interface (UI) accessible to the occupant.

[0021] Using map data stored in vehicle sensors and / or within the vehicle helps capture information about the vehicle's surroundings without the need for smartphones or other devices, and without requiring drivers or other occupants to take their hands off the steering wheel. This helps reduce distractions for occupants and / or drivers, enabling them to identify objects without operating external devices. Furthermore, using vehicle sensor and map data also helps improve the accuracy of object recognition when other vehicles or other objects obscure the view of the object of interest, or when the object is initially visible but becomes obscured when external devices become usable.

[0022] In some implementations, objects are displayed on the vehicle's UI to aid in object recognition. When an object of interest is identified, the occupant can request additional information related to the identified object. For example, in some implementations, the occupant can request directions to the identified object, the operating hours of the identified object, historical information related to the identified object, or other appropriate information.

[0023] Figure 1This is a block diagram of an object recognition system 100 according to a portion of the embodiments. In the description of the object recognition system 100, the focus is on a vehicle controlled by a driver. However, those skilled in the art will recognize that other vehicles, such as trains or other moving vehicles operated by technicians, and their operators are also within the scope of this specification. The object recognition system 100 includes a vehicle system 110 configured to capture information about vehicle occupants and generate gaze data. The vehicle system 110 also captures request start signals and occupant requests. The object recognition system 100 also includes a server 140 configured to receive the generated gaze data and information collected as log data from the vehicle's sensors. The object recognition system 100 also includes a portable device 160 accessible to the vehicle occupant associated with the occupant request. In some embodiments, some or all of the functionality of the portable device 160 is incorporated into the vehicle system 110. Integrating the functionality of the portable device 160 into the vehicle system 110 enables the occupants to utilize the object recognition system 100 even when they cannot access the portable device or when the portable device's battery is not charged enough to power the device.

[0024] Vehicle system 110 includes an electronic control unit (ECU) 120 configured to receive data from an occupant monitoring (surveillance) camera 112, a front camera 114, a global positioning system (GPS) 116, and a map 118. ECU 120 includes a gaze detection device 122 configured to receive data from the occupant monitoring camera 112 and detect gaze direction and / or gaze depth based on the received data. ECU 120 also includes a region of interest identification device 124 configured to determine the gaze position of an occupant. ECU 120 further includes a position determination device 126 configured to receive data from GPS 116 and map 118 and determine the vehicle's position relative to detected and / or known objects and / or road positions, as well as the vehicle's posture and state. Posture is the vehicle's orientation relative to a reference point such as a lane. In some embodiments, the vehicle's position also means the vehicle's position vector. The vehicle's posture and state mean the vehicle's speed and orientation. In some embodiments, the vehicle's posture and state also mean the vehicle's velocity vector, acceleration vector, and jerk vector. In some embodiments, the position vector, velocity vector, acceleration vector, and jerk vector include an angle vector. In some embodiments, the vehicle's state also refers to whether the vehicle's engine or motor is operating. ECU 120 also includes a log collection device 128 configured to receive information from the front camera 114, the position determination device 126, and the data collection request device 132, and to combine data collection requests from occupants with corresponding sensor data from the vehicle system 110 to aggregate log data that can be used by the server 140 to identify objects of interest. ECU 120 also includes a request receiving device 130 configured to receive data requests from a portable device 160. In some embodiments, the request receiving device 130 is omitted if the functionality of the portable device 160 is incorporated into the vehicle system 110. ECU 120 also includes a data collection request device 132 configured to receive line-of-sight data and region of interest information from the region of interest identification device 124 and to receive occupant request information from the request receiving device 130. The data collection request device 132 is configured to correlate received information and generate a command for the log collection device 128 to collect data associated with the occupant request information from sensors such as the vehicle's front camera 114. The ECU 120 also includes a log sending device 134, which is configured to receive log data from the log collection device 128 and send the log data to the server 140.

[0025] Occupant monitoring camera 112 is configured to capture images of the driver or other occupants of the viewing vehicle. Occupant monitoring camera 112 is connected to the vehicle. In some embodiments, occupant monitoring camera 112 includes a visible light camera. In some embodiments, occupant monitoring camera 112 includes an infrared (IR) camera or other suitable sensor. In some embodiments, occupant monitoring camera 112 is movable relative to the vehicle to capture images of at least one eye of an occupant of various sizes. While capturing images of both eyes of the occupant is preferred, some occupants may only have one eye, or the occupant's head may be turned away from occupant monitoring camera 112, in which case only one eye of the occupant can be captured by occupant monitoring camera 112. In some embodiments, occupant monitoring camera 112 is automatically adjustable. In some embodiments, occupant monitoring camera 112 is manually adjustable. In some embodiments, the captured image includes at least one eye of the occupant. In some embodiments, the captured image includes additional information about the occupant, such as approximate height, approximate weight, hair length, hair color, clothing, or other suitable information. In some embodiments, the occupant monitoring camera 112 includes a plurality of capturing devices that capture images of various areas of the occupant. In some embodiments, multiple occupant monitoring cameras 112 are configured in different locations within the vehicle. For example, in one embodiment, a first occupant monitoring camera 112 is configured near a rearview mirror located in the center of the vehicle, and a second occupant monitoring camera 112 is configured near the driver's side door. Those skilled in the art will recognize that other locations of the occupant monitoring cameras 112 that do not interfere with vehicle operation are also within the scope of this disclosure. In some embodiments, data from the occupant monitoring cameras 112 includes timestamps or other metadata that facilitate synchronization with other data.

[0026] Those skilled in the art will understand that, in some embodiments, vehicle system 110 includes additional cameras for monitoring a plurality of occupants. Each additional camera is similar to the occupant monitoring camera 112 described above. For example, in one embodiment, one or more monitoring cameras are configured within the vehicle to capture images of at least one eye of a front-seat passenger. In one embodiment, one or more monitoring cameras are configured within the vehicle to capture images of at least one eye of a rear-seat passenger. In one embodiment, the additional cameras are activated only in response to the vehicle detecting a corresponding front-seat passenger or rear-seat passenger. In one embodiment, the vehicle operator can selectively deactivate the additional cameras. In embodiments including additional cameras, the captured images are still sent to gaze detection device 122, which is capable of generating gaze results regarding the individual monitored occupants of the vehicle.

[0027] The front camera 114 is configured to capture images of the environment surrounding the vehicle. In some embodiments, the front camera 114 includes a visible light camera and an IR camera. In some embodiments, the front camera 114 is replaced by a LiDAR (Light Detection and Ranging) sensor, a RADAR (Radio Detection and Ranging) sensor, a SONAR (Sound Detection and Ranging) sensor, or other suitable sensors, or used in conjunction with such sensors. In some embodiments, the front camera 114 includes multiple additional cameras disposed in several other locations on the vehicle. For example, in one embodiment, additional cameras are disposed on the sides of the vehicle to detect a wider range of the environment to the left and right of the vehicle. Since vehicle occupants can look out through the side windows, using additional cameras to detect a wider range of the environment surrounding the vehicle helps improve the accuracy of determining what the vehicle occupants are looking at. For example, in one embodiment, additional cameras are disposed at the rear of the vehicle to detect a wider range of the environment behind the vehicle. This information helps to capture more objects that vehicle occupants other than the driver can see through the rear window. Additionally, the front camera 114 can also capture images to determine whether there are obstacles such as a central divider or guardrail between the location of an object and the occupants of the observed vehicle. In some embodiments, the data from the front camera 114 includes timestamps or other metadata to help synchronize the data from the front camera 114 with the data from the occupant monitoring camera 112.

[0028] GPS 116 is configured to determine the vehicle's position. Knowing the observed vehicle's position helps associate objects and directions that attract the occupant's attention with objects and areas at the determined positions on map 118. Knowing the vehicle's direction of travel helps predict which direction the vehicle's occupants are looking in, aiding in the generation of line-of-sight data. Knowing the observed vehicle's speed helps determine how long the vehicle's occupants have the opportunity to look at the object of interest. For example, in one implementation, the vehicle may have passed the object of interest or the vehicle's position relative to the object of interest may have changed before the occupant begins making a request. As a result, knowing the vehicle's position at various points in time helps associate the occupant's request with the object of interest.

[0029] Map 118 contains information associated with lanes and known objects along the lanes. In one embodiment, map 118 can be used with GPS 116 to identify the vehicle's position and direction of travel. In one embodiment, map 118 is received from an external device such as server 140. In one embodiment, map 118 is periodically updated based on information from front camera 114 and / or GPS 116. In one embodiment, map 118 is periodically updated based on information received from an external device. In one embodiment, map 118 is generated from sensor data using a Simultaneous Localization and Mapping (SLAM) algorithm.

[0030] For simplicity, the following description focuses primarily on the analysis of information relating to the driver. Those skilled in the art will understand that the description is equally applicable to other occupants of the vehicle, such as front-seat passengers and rear-seat passengers.

[0031] The gaze detection device 122 is configured to receive data from the occupant monitoring camera 112 and generate a gaze detection result. The gaze detection result includes the direction in which the driver's eyes are looking. In one embodiment, the direction includes azimuth and elevation. By including azimuth and elevation, it is possible to determine whether the driver is looking in a direction parallel to the horizontal line or in a direction perpendicular to the horizontal line. In one embodiment, the gaze detection result also includes depth information. Depth information is an estimated distance from the driver where the driver's visual axis converges. By including depth information, it is possible to determine the distance between the driver and the object being looked at. By combining depth information with azimuth and elevation, the accuracy of the gaze detection result is improved. In one embodiment, when the captured image only contains one eye of the driver, it is difficult to determine depth information, so only the azimuth and elevation are determined by the gaze detection device 122. In one embodiment, the gaze detection device 122 is also configured to receive data from the front camera 114 and associate the detected gaze with the pixel position of the image from the front camera 114 based on the azimuth and elevation.

[0032] In some embodiments, the gaze detection device 122 is not installed in the vehicle. In some embodiments, the gaze detection device 122 is fitted to an occupant observing the vehicle. For example, in some embodiments, the gaze detection device 122 includes smart glasses, other smart clothing, or other such devices capable of determining the wearer's gaze information. In some embodiments utilizing smart glasses, gaze data can be collected from pedestrians, cyclists, or other persons not in the vehicle. The object recognition system 100 can use this gaze data to help identify objects of interest. In embodiments including a user of the gaze detection device 122 not installed in the vehicle, the front camera 114 and the positioning device 126 are still used in combination with the gaze detection device 122.

[0033] The attention area recognition device 124 is configured to receive gaze data from the gaze detection device 122, then filter the gaze data to identify the area within the occupant's visible field of view where the occupant's focus is located. Based on the received gaze data, the attention area recognition device 124 identifies the position relative to the vehicle that attracts the occupant's attention. In some embodiments, the attention area recognition device 124 is further configured to receive information from the front camera 114 and identify pixel areas that attract the occupant's attention from the captured images of the front camera 114. The attention area recognition device 124 helps reduce the amount of log data collected by the log collection device 128, thereby reducing the processing load in the ECU 120.

[0034] The location determination device 126 is configured to receive information from GPS 116 and map 118 to identify the vehicle's position in the world coordinate system or the vehicle's position relative to objects and known objects on map 118. In some embodiments, the location determination device 126 can be used to determine the vehicle's direction of travel and speed. Additionally, the location determination device 126 is also configured to determine the vehicle's state information. In some embodiments, the state information includes the vehicle's speed. In some embodiments, the state information includes the vehicle's velocity vector. In some embodiments, the state information includes the vehicle's direction of travel. In some embodiments, the state information includes the vehicle's acceleration vector. In some embodiments, the state information includes the vehicle's jerk vector. In some embodiments, the state information includes whether the vehicle's engine or motor is operating. In some embodiments, the state information includes other vehicle-related state information such as the operation of the windshield wipers.

[0035] The log collection device 128 is configured to receive images from the front-facing camera 114, receive status information from the location determination device 126, and receive occupant request information from the data collection request device 132. The log collection device 128 is configured to correlate the received data to determine which part of the image from the front-facing camera 114 was being observed by the occupant at the time the occupant request began. Furthermore, the log collection device 128 is configured to determine what information the occupant is seeking, such as object recognition, a route to an object, or other appropriate information. Based on the gaze data analyzed by the region of interest identification device 124 and the data collection request device 132, the log collection device 128 determines the observed portion of the image captured by the front-facing camera 114. Based on the analyzed gaze data, the log collection device 128 can crop the image from the front-facing camera 114 to reduce the amount of data sent to the server for analysis. The log collection device 128 uses status information from the location determination device 126 to complete the analyzed gaze data to improve the accuracy of image cropping.

[0036] The log collection device 128 generates log data based on received and correlated data, such as the cropped image and requested data. Additionally, the log collection device 128 associates timestamp information with the log data to aid in the synchronization of the collected data and for queue prioritization within the server 140. In one embodiment, the log collection device 128 generates log data that also includes world coordinates associated with the cropped image. In another embodiment, the log collection device 128 generates log data that also includes the location on a map associated with the cropped image. In yet another embodiment, the log collection device 128 includes additional information to help improve the accuracy of responses to occupant requests.

[0037] The above description relates to generating log data based on images from the front-facing camera 114, but those skilled in the art will understand that the log collection device 128 is not limited to generating log data based on images. In some embodiments, the log collection device 128 is configured to generate log data based on information from other sensors installed in the vehicle, such as RADAR, LiDAR, or other suitable sensors. In some embodiments, the log collection device 128 can generate log data based on point cloud data received from LiDAR instead of image data. Those skilled in the art will recognize that point cloud data comprises a set of data points in space that can be used to represent a three-dimensional shape or object based on the distances of points relative to the detector. In some embodiments where the occupant wears smart glasses, the log collection device 128 is also configured to generate log data based on information received from the smart glasses.

[0038] The request receiving device 130 is configured to receive a request from the portable device 160. In some embodiments, where the functionality of the portable device 160 is incorporated into the vehicle system 110, the request receiving device 130 is omitted, and the request is transmitted directly to the data collection request device 132. In some embodiments, the request receiving device 130 is configured to receive the request wirelessly. In some embodiments, the request receiving device 130 is configured to receive the request via a wired connection. In some embodiments, the request receiving device 130 is configured to receive the request before it is received. In some embodiments, in response to the start of receiving the request, the request receiving device 130 is configured to notify the data collection request device to begin data collection in the log collection device 128 to help reliably store information from vehicle sensors such as the front camera 114 for generating log data. In some embodiments, the request receiving device 130 is also configured to receive a request containing identification information of the occupant making the request and a timestamp information indicating that the request was made. In some embodiments, the request receiving device 130 is configured to receive information associated with the identity of the occupant making the request.

[0039] The data collection request device 132 is configured to associate an occupant request with region of interest (ROI) information from the region of interest identification device 124. The data collection request device 132 is configured to transform the occupant request and ROI information into a command that the log collection device 128 can use to collect information to satisfy the occupant request. In one embodiment, the data collection request device 132 is configured to determine which sensor can be used to capture information related to a certain area of ​​the vehicle's surrounding environment. In another embodiment, the data collection request device 132 is configured to identify what type of sensor the log collection device 128 should use to satisfy the occupant request. The data collection request device 132 is also configured to identify the timestamp of the occupant request, enabling the log collection device 128 to accurately collect data from the vehicle's associated sensors.

[0040] The log sending device 134 is configured to receive log data from the log collection device 128 and send the log data to the server 140. In some embodiments, the log sending device 134 is configured to send log data wirelessly. In some embodiments, the log sending device 134 is configured to send log data via a wired connection. In some embodiments, the log sending device 134 is configured to send log data to a portable device 160, which in turn sends log data to the server 140. In some embodiments, the log sending device 134 is configured to send log data to the portable device 160 using Bluetooth (registered trademark) or other suitable wireless technology. In some embodiments, the ECU 120 is configured to determine whether the data transmission speed from the portable device 160 to the server 140 is greater than the data transmission speed from the log sending device 134 to the server 140. In response to the determination that the data transmission speed from the portable device 160 to the server 140 is greater, the log sending device 134 is configured to send the log data to be sent to the server 140 to the portable device 160. In response to the determination that the data transmission speed from portable device 160 to server 140 is not high, log sending device 134 is configured to send log data directly to server 140 instead of transmitting it from vehicle system 110 to portable device 160.

[0041] In one embodiment, the vehicle system 110 further includes a memory configured to store sensor data from sensors installed in the vehicle. In another embodiment, the memory is also configured to store information associated with previous occupant requests. In another embodiment, in response to the data collection request device 132 determining that an occupant request matches a previous occupant request, the data collection request device 132 is configured to provide the occupant 180 with the result of the matching previous occupant request. In another embodiment, the previous request is stored as cached data. Those skilled in the art will understand caching as using hardware or software to store data so that future requests for that data can be fulfilled more quickly.

[0042] Server 140 includes a log data receiving device 142 configured to receive log data from log sending device 134. In one embodiment, log data receiving device 142 is configured to receive log data from portable device 160. Server 140 also includes a log storage device 144 configured to store the received log data. Server 140 also includes a log analysis device 146 configured to receive log data from log storage device 144 and information from database 148, identify objects of interest, and / or provide information related to the objects of interest. Server 140 also includes a database 148 configured to store information about the objects. Server 140 also includes an analysis result sending device 150 configured to send the results of log analysis device 146 to portable device 160. Server 140 also includes a log sending device 152 configured to send log identification information to portable device 160.

[0043] The log data receiving device 142 is configured to receive log data from the log sending device 134. In one embodiment, the log data receiving device 142 is configured to receive log data from a portable device 160. In one embodiment, the log data receiving device 142 is configured to receive log data wirelessly. In one embodiment, the log data receiving device 142 is configured to receive log data via a wired connection. In one embodiment, the log data receiving device 142 is configured to append a timestamp of the time the log data was received to the log data.

[0044] Log storage device 144 is configured to store log data received for analysis. In one embodiment, log storage device 144 includes a solid-state storage device. In another embodiment, log storage device 144 includes dynamic random access memory (DRAM). In another embodiment, log storage device 144 includes a non-volatile memory device. In another embodiment, log storage device 144 includes a cloud-based storage device or other suitable storage structure. In another embodiment, log storage device 144 is configured to store log data in a priority-based queue. In another embodiment, the priority is based on the timestamp of when the server 140 receives the log data. In another embodiment, the priority is based on the timestamp of when a passenger request is received. In another embodiment, the priority is based on the size of the log data. In another embodiment, the priority is based on the identity of the passenger 180. For example, in one embodiment, the passenger has an account for services provided on the server 140 to prioritize the fulfillment of passenger requests. In another embodiment, other criteria are used to determine the priority of log data in the queue. In another embodiment, log data is removed from log storage device 144 after being analyzed by log analysis device 146. In some implementations, log data is not protected from being overwritten in log storage device 144 after being analyzed by log analysis device 146.

[0045] The log analysis device 146 is configured to receive log data from the log storage device 144 and determine whether the passenger request in the log data matches a record stored in the database 148. In some embodiments, the log analysis device 146 includes a neural network (NN) learned to compare the log data with known objects in the database 148. After finding a match between the log data and known objects in the database 148, the log analysis device 146 determines the requested data from the log data, such as object identification, the object's operating hours, and historical information of the object. The log analysis device 146 extracts information that satisfies the requested data from the database 148 and transmits the extracted information to the analysis result sending device 150. In some embodiments, the extracted information is transmitted to the analysis result sending device together with the identification information of the log data.

[0046] Database 148 is configured to store object-related information in association with the object's location and an image of the object. In one embodiment, database 148 includes a solid-state storage device. In one embodiment, database 148 includes dynamic random access memory (DRAM). In one embodiment, database 148 includes a non-volatile memory device. In one embodiment, database 148 includes a relational database (RDB). In one embodiment, database 148 includes a key-value database (KVS). In one embodiment, database 148 includes a NoSQL database. In one embodiment, database 148 includes a cloud-based storage device or other suitable storage structure. In one embodiment, database 148 is integrated with log storage device 144. In one embodiment, database 148 is separate from log storage device 144. In one embodiment, database 148 is configured to store information related to the analysis results of previous occupant requests. In one embodiment, log analysis device 146 is capable of reading the results of previous occupant requests in response to a determination that log data matches previous occupant requests. In one embodiment, database 148 stores feature maps generated by a neural network instead of storing image data.

[0047] The analysis result sending device 150 is configured to receive information satisfying the occupant's request from the log analysis device 146. The analysis result sending device 150 is configured to send information to the portable device 160. In some embodiments, the analysis result sending device 150 is configured to send a message to the vehicle system 110 instead of the portable device 160, or to send information to the vehicle system 110 in addition to the portable device 160. In some embodiments, the server 140 is configured to determine whether the data transmission speed from the server 140 to the portable device 160 is greater than the data transmission speed from the server 140 to the vehicle system 110. In response to the determination that the data transmission speed from the server 140 to the portable device 160 is greater, the analysis result sending device 150 is configured to send the information to be sent to the vehicle system 110 to the portable device 160. In response to the determination that the data transmission speed from the server 140 to the vehicle system 110 is greater, the analysis result sending device 150 is configured to send information directly to the vehicle system 110 without going through the portable device 160. In some embodiments, the analysis result transmitting device 150 is configured to transmit information wirelessly. In some embodiments, the analysis result transmitting device 150 is configured to transmit information via a wired connection. In some embodiments, the analysis result transmitting device 150 is configured to also transmit identification information for log data associated with the information. Transmitting identification information for the log data helps the portable device 160 or vehicle system 110 display both the data request and the analysis results to the occupants.

[0048] Log sending device 152 is configured to send information related to the processing of log data by server 140. In one embodiment, log sending device 152 is configured to send information to portable device 160. In another embodiment, log sending device 152 is configured to send information to vehicle system 110. In one embodiment, server 140 is configured to determine whether the data transmission rate from server 140 to portable device 160 is greater than the data transmission rate from server 140 to vehicle system 110. In response to the determination that the data transmission rate from server 140 to portable device 160 is greater, log sending device 152 is configured to send information to be sent to vehicle system 110 to portable device 160. In response to the determination that the data transmission rate from server 140 to vehicle system 110 is greater, log sending device 152 is configured to send information directly to vehicle system 110 without going through portable device 160. In one embodiment, log sending device 152 is configured to send log data to portable device 160 and / or vehicle system 110 for reconfirmation by occupants. In some embodiments, the log sending device 152 is configured to send identification information for the log data to the portable device 160 and / or the vehicle system 110 in response to the log analysis device 146 retrieving log data from the queue of the log storage device 144. In some embodiments, the log sending device 152 sends the information wirelessly. In some embodiments, the log sending device 152 sends the information via a wired connection.

[0049] Portable device 160 includes a log receiving device 162 configured to receive information from log sending device 152. The portable device also includes an analysis result receiving device 164 configured to receive information from analysis result sending device 150. Portable device 160 further includes a UI 166 configured to transmit information to occupant 180 based on information received from log sending device 152 and analysis result sending device 150. UI 166 is also configured to receive input information from occupant 180. Portable device 160 also includes a microphone 168 configured to receive request start information and request data from occupant 180. Portable device 160 also includes a voice recognition device 170 configured to analyze data received from microphone 168 to determine the content of the request start information and request data. Portable device 160 also includes a request sending device 172 configured to send request data to request receiving device 130.

[0050] The log receiving device 162 is configured to receive information from the log sending device 152. In one embodiment, the log receiving device 162 is configured to receive information wirelessly. In another embodiment, the log receiving device 162 is configured to receive information via a wired connection.

[0051] The analysis result receiving device 164 is configured to receive information from the analysis result sending device 150. In one embodiment, the analysis result receiving device 164 is configured to receive information wirelessly. In another embodiment, the log receiving device 162 is configured to receive information via a wired connection.

[0052] UI 166 is configured to receive information from log receiving device 162 and analysis result receiving device 164. UI 166 is configured to convey the received information to occupant 180. In one embodiment, UI 166 includes a touch panel. In one embodiment, UI 166 is part of a smartphone. In one embodiment, UI 166 is incorporated into a vehicle including vehicle system 110. In one embodiment, UI 166 is configured to receive input from occupant 180. In one embodiment, UI 166 is configured to receive input indicating the identity of occupant 180. In one embodiment, UI 166 is configured to receive input corresponding to a data request from occupant 180.

[0053] Microphone 168 is configured to capture voice signals from occupant 180. In one embodiment, microphone 168 is part of a smartphone. In another embodiment, microphone 168 is integrated with a vehicle including vehicle system 110. In another embodiment, microphone 168 includes a directional microphone. In another embodiment, microphone 168 is configured to capture the voice of occupant 180.

[0054] The voice recognition device 170 is configured to receive voice signals from the microphone 168 and determine the content of the voice signals. In some embodiments, the voice recognition device 170 is configured to determine whether the voice signals, such as keywords or key phrases, indicate the start of a request. In some embodiments, the voice recognition device 170 is configured to determine the type of data requested by the occupant 180, such as object recognition or information about the object. In some embodiments, the voice recognition device 170 is also configured to determine the identity of the occupant 180. In some embodiments, the voice recognition device 170 is configured to determine the identity of the occupant 180 based on voice recognition software. In some embodiments, the voice recognition device 170 is configured to determine the identity of the occupant 180 based on the recognition of keywords or key phrases, such as the occupant's name or other identification information. In some embodiments, the voice recognition device 170 is configured to determine the identity of the occupant 180 based on input received in the UI 166. In some embodiments, the voice recognition device 170 is configured to identify the occupant 180 based on input from the vehicle system 110, such as an image of a speaking occupant from the occupant monitoring camera 112.

[0055] The above description relates to requests initiated based on verbal input. Those skilled in the art will recognize that this description is not limited to verbal requests. In some embodiments, a request to begin includes input received in UI 166. In some embodiments, a request to begin includes a gesture detected using the occupant monitoring camera 112. In some embodiments, a request to begin includes input in UI 166 and / or verbal input, or the result of occupant facial recognition, or the result of occupant iris recognition by the gaze detection device 122, or a combination of different inputs, or other suitable combinations. Including a request to begin as part of an occupant request helps minimize unnecessary processing and data transmission, which helps minimize the processing load and power consumption of the vehicle, including vehicle system 110. As more vehicles become electric vehicles (EVs), minimizing power consumption becomes a greater concern for maintaining battery charging and maximizing the range that EVs can travel without recharging.

[0056] The request sending device 172 is configured to receive request information from the voice recognition device 170 and send the information to the request receiving device 130. In one embodiment, the request sending device 172 is configured to send a request start signal in response to the voice recognition device 170 recognizing the start of a request. In another embodiment, the request sending device 172 does not send a response signal when the voice recognition device 170 recognizes the start of a request. Sending a response signal in response to the start of a request helps the vehicle system 110 store sensor data to improve the accuracy and precision of fulfilling occupant requests. However, sending a response signal in response to the start of a request increases the amount of data transmitted and the processing load. The request sending device 172 is configured to send an occupant request based on the analysis performed by the voice recognition device 170. In one embodiment, the request sending device 172 is configured to send the occupant request or other information wirelessly. In another embodiment, the request sending device 172 is configured to send the occupant request or other information via a wired connection.

[0057] Those skilled in the art will understand that modifications to the object recognition system 100 are also within the scope of this disclosure. For example, in one embodiment, the microphone 168 and the voice recognition device 170 are omitted, and the occupant request, including the request to begin, is received via the UI 166. In another embodiment, based on the analysis results obtained from the server 140 and sent to the portable device 160, warnings such as voice warnings or visual warnings are automatically displayed on the portable device 160.

[0058] Figure 2 This is a flowchart of an object recognition method 200 according to one embodiment. In one embodiment, system 100 ( Figure 1) Execution method 200. In some embodiments, system 1100 is used ( Figure 11 ) Execute method 200.

[0059] In action 210, occupant 180 initiates a request. Initiating a request helps avoid unnecessary processing load on portable device 160, vehicle system 110, and server 140 by preventing accidental occupant request processing. In one embodiment, the initiation request includes occupant 180 speaking keywords or key phrases, for example via microphone 168. Figure 1 ) detection. In some implementations, the start request includes occupants, for example, in UI166 ( Figure 1 Touch the button on the device. In some embodiments, the initial request includes a portable device 160 or vehicle system 110, for example, using an occupant monitoring camera 112. Figure 1 The device detects a predetermined gesture made by the occupant 180. When a request is initiated, the portable device activates the request receiving device in action 220, and the occupant 180 can input the request in action 212.

[0060] In action 212, occupant 180 inputs a request. The request is information that occupant 180 wants to know about an object of interest. In one embodiment, the request includes identification information about the object. In another embodiment, the request includes other information about the object, such as business hours, directions to the object, historical information about the object, or other appropriate information. In another embodiment, occupant 180 verbally inputs the request, for example through microphone 168. Figure 1 ) detection. In some implementations, occupant 180 uses a UI, such as UI166 ( Figure 1 In one embodiment, occupant 180 uses a predetermined gesture to input the request, such as gestures input by occupant monitoring camera 112. Figure 1 Detection. In some embodiments, the method for requesting a start is the same as the method for requesting an input, for example, both the start and input are verbally performed by the occupant. In some embodiments, the method for requesting a start is different from the method for requesting an input, for example, the start is performed using a UI, and the input is performed verbally. Other combinations of the start request and the input are also within the scope of this disclosure.

[0061] In action 220, portable device 160 activates request receiving device 130. Activating the request receiving device in response to a start request helps portable device 160 save power by avoiding having the request receiving device constantly monitor for requests from occupant 180. In some embodiments, activating the request receiving device includes a UI, such as UI166 (…). Figure 1The input screen is displayed on the device. In some embodiments, the start request receiving device includes a microphone, such as microphone 168. Figure 1 Initialization. In one embodiment, the start request receiving device includes circuitry within a portable device 160 that initiates the processing of the received request.

[0062] Action 220 is repeated until the portable device 160 receives an input request in action 212. In some embodiments, if no input request is received in action 212 after a predetermined period, such as 10 to 30 seconds, action 220 is terminated, and the receiving device is requested to return to a sleep state or a low-power state. If the predetermined period is too long, power consumption may increase unnecessarily in some cases. If the predetermined period is too short, the occupant 180 may not have enough time to input a request in action 212 in some cases. In some embodiments, action 220 is terminated in response to receiving a cancellation signal, such as that caused by keywords and / or key phrases, input to the UI, or other appropriate input.

[0063] In action 222, portable device 160 receives the request from action 212. In one embodiment, the request is received directly from occupant 180. In another embodiment, the request is received indirectly from occupant 180 via an external device, such as a keyboard or other suitable external device. In another embodiment, actions 212 and 222 utilize the same components of portable device 160, such as microphone 168 or UI 166. Figure 1 )implement.

[0064] In action 224, the request is analyzed and sent. The request is analyzed in a way that determines the type of data requested by occupant 180. In some embodiments, a voice recognition device 170 is used. Figure 1 ) Analyze the request. In order to collect log data to satisfy the request, the analyzed request is sent to vehicle system 110. In one embodiment, a request sending device 172 is used. Figure 1 Send the analyzed request.

[0065] In action 230, the analyzed request is received by vehicle system 110. In one embodiment, the analyzed request is received wirelessly. In another embodiment, the analyzed request is received via a wired connection. In a third embodiment, a request receiving device 130 is used. Figure 1 ) Receive the analyzed request.

[0066] In action 232, one or more images of the occupant are captured. The captured images are associated with timestamp data to determine the time when the one or more images were captured. The one or more images of the occupant capture at least one eye of the occupant. In one embodiment, images of the occupant are captured at regular intervals. In one embodiment, images of the occupant are captured as part of action 220 in response to receiving a signal indicating a start request, such as a signal from portable device 160 to vehicle system 110. In one embodiment, occupant monitoring camera 112 is used. Figure 1 Capture one or more images of the occupants. In one embodiment, only the image of the occupant associated with the occupant request is captured. In one embodiment, images of two or more occupants of the vehicle are captured, and only the image of the occupant associated with the occupant request is used to generate request data later in method 200. In one embodiment, action 232 is performed in response to a signal generated in action 220. In one embodiment, action 232 is performed independently of the reception of the request.

[0067] In action 234, the occupant's gaze is detected based on one or more images captured in action 232. Detecting the occupant's gaze includes identifying the angle of the occupant's gaze relative to the vehicle. In one embodiment, the angle includes azimuth and elevation. In one embodiment, detecting the gaze also includes determining the depth of the gaze relative to the vehicle's position. In one embodiment, a gaze detection device 122 is used ( Figure 1 ) Perform action 234.

[0068] In action 236, a region of interest (ROI) is identified based on the occupant's gaze detected in action 234. The ROI is identified to determine the region of interest (ROI) with respect to occupant 180. In one embodiment, the ROI is identified based on world coordinates. In another embodiment, for example, it is based on the front camera 114 used by the vehicle. Figure 1 The pixel regions of the captured image are used to identify regions of interest. In one embodiment, the regions of interest are identified based on relative coordinates relative to the vehicle. Identifying regions of interest helps reduce the amount of data sent to server 140 for processing. In one embodiment, a region of interest identification device 124 is used. Figure 1 ) Execute action 236.

[0069] In one embodiment, actions 232-236 are performed continuously while the vehicle is being driven. In response to receiving an occupant request, information generated in actions 232-236 is stored in a memory within the vehicle system 110 for analysis. In one embodiment, actions 232-236 are performed as part of action 220 in response to receiving a start request signal. In one embodiment, actions 232-236 are stopped in response to receiving a signal indicating that action 220 was stopped due to failure to successfully receive a timely input request or in response to cancellation of input.

[0070] In action 238, a data collection request is generated based on the received and analyzed request. The data collection request identifies information from actions 232-236 that can be used to satisfy the received and analyzed request. The data collection request identifies which vehicle sensor can be used to satisfy the received and analyzed request. Additionally, the data collection request also identifies the period during which sensor data should be collected based on the timestamp of the request received in action 222. In some embodiments, a data collection request device 132 is used ( Figure 1 ) Execute action 238.

[0071] In action 240, sensor data is collected based on a data collection request. In one embodiment, sensor data is collected from memory within vehicle system 110. In one embodiment, sensor data is collected from a single sensor. In one embodiment, sensor data is collected from multiple sensors. In one embodiment, log collection device 128 is used. Figure 1 Collect sensor data.

[0072] In action 242, the sensor data collected in action 240 is cropped. By cropping the sensor data, the amount of data sent to server 140 is reduced. Here, the term "cropped" is used because the sensor data is image data. However, those skilled in the art will understand that action 242 is used to reduce redundant data based on the region of interest identified in action 236, regardless of the type of sensor data used. In some embodiments, log collection device 128 is used ( Figure 1 ) Perform action 242. In some implementations, the cropped sensor data, along with timestamp information, is considered as log data.

[0073] In action 244, log data is sent to server 140. In some embodiments, if the memory within vehicle system 110 is capable of comparing the received analyzed request with previous occupant requests, action 244 is omitted, and vehicle system 110 directly provides the result of satisfying the received analyzed request. In some embodiments, log data is transmitted wirelessly. In some embodiments, log data is received via a wired connection. In some embodiments, log sending device 134 is used. Figure 1 ) Execute action 244.

[0074] In action 250, server 140 receives log data. In some embodiments, log data receiving device 142 is used. Figure 1 ) Execute action 250. In some embodiments, if the vehicle system 110 is able to provide a result that satisfies the occupant's request, log data is not sent to the server 140, and action 250 is omitted.

[0075] In action 252, log data is stored on server 140. Server 140 stores the log data for subsequent processing. In one implementation, the log data is stored using a priority-based queue. In another implementation, the priority in the queue is based on the time the log data is received by server 140. In another implementation, the priority in the queue is based on the time the occupant request is received, i.e., the time in action 222. In another implementation, the priority in the queue is based on the identity of occupant 180.

[0076] In action 254, the log data is analyzed to determine if the occupant's request is satisfied. This is done by comparing data from the vehicle's sensors with data in the database of server 140. If a match is found between the object in the vehicle sensor data and the data in the database, the database is queried to retrieve information that satisfies the occupant's request. For example, in one embodiment, the database is queried to determine identification information such as the object, the object's operating hours, and the object's location. In one embodiment, the database contains URLs for the occupant 180 to find information about the object. In one embodiment, if no match is found between the vehicle sensor data and the data in the database, action 254 returns a result indicating no match was found. In one embodiment, log analysis device 146 ( Figure 1 ) Execute action 254.

[0077] In action 256, the analysis results from action 254 are transmitted. In one embodiment, the analysis results are transmitted wirelessly. In another embodiment, the analysis results are transmitted via a wired connection. In method 200, the analysis results are transmitted to portable device 160. In one embodiment, the analysis results are transmitted to vehicle system 110 instead of portable device 160, or the analysis results are transmitted to vehicle system 110 in addition to portable device 160. In one embodiment, analysis result transmission device 150 is used. Figure 1 ) Execute action 256.

[0078] In action 260, portable device 160 receives the analysis results. In one embodiment, the analysis results include both the information read from the database in action 254 and log data identification information. Including log data identification information along with the analysis results facilitates rapid analysis and provides additional information about the object if the occupant requests more information about the object after receiving the analysis results. In one embodiment, analysis result receiving device 164 ( Figure 1 ) Execute action 260.

[0079] In action 262, the analysis results are notified to occupant 180. In one embodiment, occupant 180 is notified by providing a URL for accessing information about the object. In another embodiment, occupant 180 is notified by providing requested information about the object. In one embodiment, visual notification is used to notify occupant 180. In another embodiment, voice notification is used to notify occupant 180. In another embodiment, UI166 (…) is used. Figure 1The occupant is notified. In one embodiment, the occupant 180 is notified by at least one of a voice warning and a visual warning automatically issued on the portable device 160 in response to receiving analysis results from the server 140. In one embodiment, the notification to the occupant 180 includes vehicle sensor data, such as cropped images, included as part of the log data to enable the occupant 180 to confirm that the received information corresponds to a predetermined object of interest. In one embodiment, the notification to the occupant 180 includes a request for confirmation regarding the correct identification of the object of interest, which helps to improve the performance of the log data analysis in action 254 by providing the server 140 with the result of the confirmation request. In one embodiment, after action 262, the occupant provides feedback to at least one of the server 140, the portable device 160, and the vehicle system 110 regarding whether the received result is truly related to the occupant's request or whether the occupant likes the information. The feedback provides the neural network (NN) with the ability to adjust or learn to reduce false positives and false negatives over time, enabling the log analysis device 146, the region of interest identification device 124, the data collection request device 132, and the speech recognition device 170 to do so.

[0080] Those skilled in the art will recognize that modifications to method 200 are also within the scope of this disclosure. In some embodiments, additional actions are included in method 200. For example, in one embodiment, method 200 includes updating the database within server 140 based on occupant confirmation following notification analysis results. In some embodiments, at least one action of method 200 is omitted. For example, in one embodiment, action 242 is omitted if there are no concerns about data transmission volume. In some embodiments, the order of actions of method 200 may be changed. For example, in one embodiment, action 234 is performed after action 230 to reduce the processing load in vehicle system 110. Those skilled in the art will recognize that other modifications are also within the scope of this disclosure.

[0081] Figure 3 This is a flowchart of an object recognition method 300 according to one embodiment. In one embodiment, system 100 ( Figure 1 ) Execution method 300. In some implementations, system 1100 is used ( Figure 11 Execute method 300. Method 300 and method 200 ( Figure 2 Similar to the actions of method 200. The actions of method 300, which are similar to those of method 200, have the same reference numerals. For simplicity, only the actions of method 300 that are different from those of method 200 will be described below.

[0082] In action 305, log data is analyzed and associated with object information for objects of interest. Log data is analyzed by comparing data from vehicle sensors with data in the database of server 140. Once a match is found between the object in the vehicle sensor data and the data in the database, a link to the object information for the matching object in the database is associated with the log data. Through the link, occupant 180 can access the database within server 140 to obtain the requested information about the object. In one embodiment, the link contains a Uniform Resource Locator (URL) that the occupant can open using UI 166 (a web browser, etc.). In another embodiment, the link allows occupant 180 to obtain additional information about the object beyond the requested information. In one embodiment, log data is analyzed by comparing feature maps extracted by the NN from data from vehicle sensors with feature maps extracted by the NN from data in the database of server 140. In another embodiment, a log analysis device 146 ( Figure 1 ) Execute action 305.

[0083] In action 310, a link for accessing log data and associated object information from action 305 is sent. In one embodiment, the link is sent wirelessly. In another embodiment, the link is sent via a wired connection. In method 300, the link is sent to portable device 160. In one embodiment, the link is sent to vehicle system 110 instead of portable device 160, or the link is sent to vehicle system 110 in addition to portable device 160. In one embodiment, analysis result sending device 150 is used. Figure 1 ) Execute action 310.

[0084] In action 320, portable device 160 receives a link. In one embodiment, the link includes both a link for accessing a database and log data identification information. Including log data identification information along with the analysis results facilitates rapid analysis, providing additional information about the object when the occupant requests more information about the object after receiving the link, and when the link does not provide access to all information about the object stored in the database. In one embodiment, analysis result receiving device 164 ( Figure 1 ) Execute action 320.

[0085] In action 322, a link is notified to occupant 180. In one implementation, occupant 180 is notified by providing a URL for accessing information about the object. In another implementation, occupant 180 is notified by providing an optional icon for accessing information about the object. In one implementation, visual notification is used to notify occupant 180. In another implementation, voice notification is used to notify occupant 180. In another implementation, UI166 (…) is used. Figure 1 The occupant is notified. In one embodiment, the occupant 180 is notified by at least one of a voice warning and a visual warning automatically issued on the portable device 160 in response to receiving a link from the server 140. In one embodiment, the notification to the occupant 180 includes vehicle sensor data, such as cropped images, included as part of the log data so that the occupant 180 can confirm that the received information corresponds to a predetermined object of interest. In one embodiment, the notification to the occupant 180 includes a request for confirmation regarding the correct identification of the object of interest, which helps to improve the performance of the log data analysis in action 305 by providing the server 140 with the result of the confirmation request.

[0086] Those skilled in the art will recognize that modifications to method 300 are also within the scope of this disclosure. In some embodiments, additional actions are included in method 300. For example, in one embodiment, method 300 includes updating a database within server 140 based on a result of occupant confirmation following a notification link. In some embodiments, at least one action of method 300 is omitted. For example, in one embodiment, action 242 is omitted if there is no concern about data transmission volume. In some embodiments, the order of actions of method 300 may be changed. For example, in one embodiment, action 234 is performed after action 230 to reduce the processing load in vehicle system 110. Those skilled in the art will recognize that other modifications are also within the scope of this disclosure.

[0087] Figure 4 This is a diagram of a data structure 400 for occupant requests related to one implementation. In one implementation, data structure 400 corresponds to data received by microphone 168 and processed by voice recognition device 170. Figure 1 The status of the occupant request processed from occupant 180. In one implementation, data structure 400 corresponds to the status of the occupant request processed in action 222 ( Figure 2 The crew request received in the system.

[0088] Data structure 400 includes occupant identification information 405. Occupant identification information 405 indicates the identity of the occupant who made the occupant request. In one embodiment, this is based on the voice recognition device 170 ( Figure 1The analysis performed is used to determine occupant identification information 405. In some implementations, this is based on the analysis performed in UI166 ( Figure 1 The occupant identification information 405 is determined based on the input from the portable device 160. In some embodiments, this is based on who manages the portable device 160. Figure 1 The occupant identification information 405 is determined based on the iris recognition result of the occupant's eyes identified by the camera of the portable device 160. In some embodiments, the occupant identification information 405 is determined based on the occupant's fingerprint identified by the portable device 160 or by a sensor on the vehicle's steering wheel. The data structure 400 also includes request data 410. Request data 410 contains the content of information requested by the occupant. In some embodiments, request data 410 contains a request for object recognition. In some embodiments, request data 410 contains a request for information about an object, other than or different from object recognition. The data structure 400 also includes timestamp information 415. Timestamp information 415 indicates the time corresponding to the receipt of information requested by the occupant.

[0089] Data structure 400 is merely an example, and those skilled in the art will understand that occupant request data can contain different information. In some embodiments, at least one of the constituent elements is excluded from data structure 400. For example, in one embodiment, occupant identification information 405 is excluded from data structure 400. In some embodiments, data structure 400 includes additional information. For example, in one embodiment, data structure 400 also includes information about the location of occupants within the vehicle.

[0090] Figure 5 This is a diagram of a data structure 500 for region of interest data involved in one implementation. In one implementation, data structure 500 corresponds to the region of interest identification device 124 ( Figure 1 The region of interest is identified. In some implementations, data structure 500 corresponds to the region of interest identified in action 236. Figure 2 The region of interest identified in the data.

[0091] Data structure 500 includes occupant identification information 505. Occupant identification information 505 indicates the identity of the occupant making the occupant request. In one embodiment, this is based on the voice recognition device 170 ( Figure 1 The analysis performed is used to determine occupant identification information 505. In some implementations, this is based on the analysis performed in UI166 ( Figure 1 The occupant identification information 505 is determined based on the input from the portable device 160. In some embodiments, this is based on who manages the portable device 160. Figure 1The occupant identification information 505 is determined based on the recognition result of the occupant's iris detected by the gaze detection device 122 or the camera of the portable device 160. In some embodiments, the occupant identification information 405 is determined based on the occupant's fingerprint detected by the portable device 160 or by a sensor on the vehicle's steering wheel. The data structure 500 also includes timestamp information 510. In some embodiments, timestamp information 510 represents the time corresponding to the receipt of information requested from the occupant. In some embodiments, timestamp information 510 includes information related to the time when data is captured by the vehicle's sensors. In some embodiments, timestamp information 510 includes information related to the time when the region of interest is identified. The data structure 500 also includes region of interest (ROI) information 515. ROI information 515, for example, represents the location within the image where the region of interest is identified. The ROI information 515 is determined based on the correlation between the occupant's gaze data associated with the occupant identification information 505 and the sensor data from the vehicle. ROI information 515 includes a first corner pixel position 520. In one embodiment, the first corner pixel position 520 represents the position within the image of the upper left corner of the region of interest determined based on occupant gaze data. ROI information 515 also includes a second corner pixel position 525. In one embodiment, the second corner pixel position 525 represents the position within the image of the lower right corner of the region of interest determined based on occupant gaze data. Using the first corner pixel position 520 and the second corner pixel position 525, the boundary of the determined region of interest can be set using minimal position information. In one embodiment, ROI information 515 can be used, for example, with log collection device 128 (… Figure 1 Or in action 242 ( Figure 2 Cropping images in )

[0092] Data structure 500 is merely an example, and those skilled in the art will understand that it is possible to include different information in the region of interest data. In some embodiments, at least one of the constituent elements is excluded from data structure 500. For example, in one embodiment, occupant identification information 505 is excluded from data structure 500. In some embodiments, data structure 500 includes additional information. For example, in one embodiment, regarding ROI information 515, data structure 500 also includes additional corner pixel positions.

[0093] Figure 6 This is a diagram of a data structure 600 for region of interest data involved in one implementation. In one implementation, data structure 600 corresponds to the region of interest identification device 124 ( Figure 1The region of interest is identified. In some implementations, data structure 600 corresponds to the region of interest identified in action 236. Figure 2 The region of interest identified in the data structure 600 and data structure 500. Figure 5 Similar to data structure 500, the constituent elements of data structure 600 have the same reference numbers. For simplicity, only the constituent elements of data structure 600, which differs from data structure 500, will be described below.

[0094] Data structure 600 includes ROI information 615, which, in addition to the first corner pixel position 520 and the second corner pixel position 525, also includes depth information 620. The depth information 620 can be used to determine the distance from the vehicle to which the occupant is looking. In some embodiments, a gaze detection device 122 is used. Figure 1 Or in action 234 ( Figure 2 The depth information 620 is determined within the occupant's data. Including depth information 620 helps improve the accuracy of identifying objects for which the occupant is requesting information.

[0095] Data structure 600 is merely an example, and those skilled in the art will understand that it is possible for the region of interest data to contain different information. In some embodiments, at least one of the constituent elements is excluded from data structure 600. For example, in one embodiment, occupant identification information 505 is excluded from data structure 600. In some embodiments, data structure 600 includes additional information. For example, in one embodiment, with respect to ROI information 615, data structure 600 also includes additional corner pixel positions.

[0096] Figure 7 This is a diagram of the data structure for the region of interest data involved in one of the implementation methods. In one implementation method, data structure 700 corresponds to the region of interest identification device 124 ( Figure 1 The region of interest is identified. In some implementations, data structure 700 corresponds to the region of interest identified in action 236. Figure 2 The region of interest identified in the data structure 700 and data structure 500. Figure 5 Similar to data structure 500, the constituent elements of data structure 700 have the same reference numbers. For simplicity, the following description only focuses on the constituent elements of data structure 700, which differs from data structure 500.

[0097] Data structure 700 includes ROI information 715, which includes world coordinate position information 720 instead of the first corner pixel position 520 and the second corner pixel position 525. The world coordinate position information 720 can be used to determine the position of an object in the real world. In some embodiments, a log collection device 128 is used. Figure 1 Or in action 236 ( Figure 2 The world coordinate position information 720 is determined within the data. Including world coordinate position information 720 helps improve the accuracy of identifying objects that have requested occupant information.

[0098] Data structure 700 is merely an example, and those skilled in the art will understand that it is possible for the region of interest data to contain different information. In some embodiments, at least one of the constituent elements is excluded from data structure 700. For example, in one embodiment, occupant identification information 505 is excluded from data structure 700. In some embodiments, data structure 700 includes additional information. For example, in one embodiment, data structure 700 also includes at least a partial image of an object.

[0099] Figure 8 This is a diagram of a user interface 800 according to one implementation. In one implementation, UI 800 corresponds to UI 166 ( Figure 1 In some implementations, UI800 is a portable device 160. Figure 1 In one implementation, UI800 is part of vehicle system 110. Figure 1 Part of ).

[0100] UI 800 includes navigation UI 805 and image UI 810. Image UI 810 includes captured images 815 from vehicle sensors and a highlight 820 of identified objects. UI 800 can be used to notify occupants of objects identified as the cause of occupant requests using image UI 810. UI 800 can also be used to notify occupants of the movement path of objects using navigation UI 805. In some embodiments, UI 800 is configured to receive information from occupants as part of occupant requests, request initiation, confirmation of identified objects, or other such input information. In some embodiments, UI 800 is integrated into the vehicle. In some embodiments, UI 800 can be detached from the vehicle.

[0101] The navigation UI805 is configured to, for example, receive signals from GPS116 ( Figure 1The navigation UI 805 displays GPS information and a map visible to the vehicle's driver. It is also configured to display a path along the map that the vehicle can traverse to reach identified objects. In one embodiment, the navigation UI 805 includes a touch panel. In another embodiment, the navigation UI 805 is configured to receive data from server 140 (…). Figure 1 External devices such as ) receive map and / or movement path updates.

[0102] The image UI 810 includes a captured image 815 from a vehicle sensor and a highlighted display 820 of identified objects. The highlighted display 820 of identified objects is overlaid on the image 815 from the vehicle sensor to identify objects in the image from the vehicle sensor. In one embodiment, the image 815 from the vehicle sensor is a cropped image from the vehicle sensor. In one embodiment, the image UI 810 is capable of receiving input from an occupant to confirm or deny the accuracy of the identified objects. In one embodiment, the image UI 810 includes a touch panel.

[0103] Figure 8 Includes a navigation UI805 separate from the image UI810. In one embodiment, the image UI810 overlaps with the navigation UI805. In another embodiment, the image UI810 is hidden while the vehicle is moving.

[0104] Figure 9 This is a diagram of a user interface 900 according to one embodiment. In one embodiment, UI 900 corresponds to UI 166 ( Figure 1 In some implementations, UI900 is a portable device 160. Figure 1 In one implementation, UI900 is part of vehicle system 110. Figure 1 UI900 is part of UI800. The components of UI900, which are similar to UI800, share the same reference numbers. For simplicity, only the components of UI900 that differ from UI800 will be described below.

[0105] UI900 includes link UI910, which is configured to display links to object information, such as in action 320. Figure 3 The link received in the system. In one embodiment, the link UI910 includes selectable links and is configured to display object information in response to information read after an occupant selects a link. In one embodiment, the link UI910 is configured to display an icon associated with the link. In one embodiment, the link UI910 includes a touch panel.

[0106] Figure 9It includes a navigation UI805 and a link UI910 separated from the image UI810. In one embodiment, at least one of the image UI810 and the link UI910 overlaps with the navigation UI805. In another embodiment, at least one of the image UI810 and the link UI910 is hidden during vehicle movement.

[0107] Figure 10 This is a diagram of a user interface related to one of the implementation methods. In one implementation method, UI1000 corresponds to UI166 ( Figure 1 In some implementations, UI1000 is a portable device 160. Figure 1 In one implementation, UI1000 is part of vehicle system 110. Figure 1 UI1000 is part of UI800. The components of UI1000, which are similar to UI800, share the same reference numbers. For simplicity, the following description focuses only on the components of UI1000 that differ from UI800.

[0108] UI 1000 includes a request history UI 1010, configured to display information related to subsequent requests for occupant requests and additional information about objects. In one embodiment, the request history UI 1010 includes a dialog-style display with occupant requests and sequentially provided object information. In another embodiment, the request history UI 1010 is configured to provide a selectable list of previous occupant requests and, in response to the selection of an occupant request, display information provided in response to the corresponding occupant request. In another embodiment, the request history UI 1010 includes a touch panel.

[0109] Figure 10 It includes a navigation UI805 and a request history UI1010, which are separated from the image UI810. In one embodiment, at least one of the image UI810 and the request history UI1010 overlaps with the navigation UI805. In another embodiment, at least one of the image UI810 and the request history UI1010 is hidden during vehicle movement.

[0110] Figure 11This is a block diagram of a system for performing object recognition according to a portion of an embodiment. System 1100 includes a hardware processor 1102 and a non-transitory computer-readable storage medium 1104 encoded with computer program code 1106, i.e., a set of executable commands; that is, a non-transitory computer-readable storage medium 1104 storing a set of executable commands. Additionally, the computer-readable storage medium 1104 is encoded with commands 1107 for connection to external devices. The processor 1102 is electrically connected to the computer-readable storage medium 1104 via a bus 1108. Furthermore, the processor 1102 is also electrically connected to an input / output (I / O) interface 1110 via the bus 1108. Additionally, a network interface 1112 is electrically connected to the processor 1102 via the bus 1108. The network interface 1112 is connected to a network 1114, thereby enabling the processor 1102 and the computer-readable storage medium 1104 to connect to external elements via the network 1114. Processor 1102 is configured to execute computer program code 1106 encoded in computer-readable storage medium 1104 to enable the use of system 1100 in object recognition system 100. Figure 1 Method 200 Figure 2 ) or method 300 ( Figure 3 Part or all of the action described in ().

[0111] In some embodiments, processor 1102 is a central processing unit (CPU), a multiprocessor, a distributed processing system, an application-specific integrated circuit (ASIC), and / or a suitable processing unit.

[0112] In some embodiments, the computer-readable storage medium 1104 includes electronic, magnetic, optical, electromagnetic, infrared, and / or semiconductor systems (or machines or devices). For example, the computer-readable storage medium 1104 includes semiconductor memory, solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), read-only memory (ROM), rigid magnetic disk, and / or optical disk. In some embodiments using optical disk, the computer-readable storage medium 1104 includes optical disc-read-only memory (CD-ROM), optical disc-read / write (CD-R / W), and / or digital versatile optical disc (DVD).

[0113] In some embodiments, storage medium 1104 is configured to enable system 1100 to perform operations in object recognition system 100. Figure 1 Method 200 Figure 2 ) or method 300 ( Figure 3Computer program code 1106, representing part or all of the actions described in [the document]. Additionally, in some embodiments, storage medium 1104 stores data such as line-of-sight data parameters 1116, object data parameters 1118, vehicle position parameters 1120, request content parameters 1122, and / or object recognition system 100. Figure 1 Method 200 Figure 2 ) or method 300 ( Figure 3 The actions described in the document are part or all of a set of executable commands performed in the object recognition system 100. Figure 1 Method 200 Figure 2 ) or method 300 ( Figure 3 The information required for part or all of the action described in the object recognition system 100, and the information required for performing such actions. Figure 1 Method 200 Figure 2 ) or method 300 ( Figure 3 Information generated during part or all of the action described in the document.

[0114] In one embodiment, storage medium 1104 stores commands 1107 for connection to an external device. Command 1107 enables processor 1102 to generate commands that can be read by the external device to efficiently execute within object recognition system 100. Figure 1 Method 200 Figure 2 ) or method 300 ( Figure 3 Part or all of the action described in ().

[0115] System 1100 includes an I / O interface 1110. The I / O interface 1110 is connected to external circuitry. In one embodiment, the I / O interface 1110 includes a keyboard, keypad, mouse, trackball, and / or cursor arrow keys for transmitting information and instructions to processor 1102.

[0116] Additionally, system 1100 also includes a network interface 1112 integrated with processor 1102. Network interface 1112 enables system 1100 to communicate with a network 1114 connected to one or more other computer systems. Network interface 1112 includes wireless network interfaces such as Bluetooth, WIFI, WIMAX, GPRS, or WCDMA, or wired network interfaces such as Ethernet, USB, or IEEE-1394. In some embodiments, the object recognition system 100 is executed in two or more systems 1100. Figure 1 Method 200 Figure 2 ) or method 300 ( Figure 3The actions described in the document involve the exchange of information such as line-of-sight data parameters 1116, object data parameters 1118, vehicle position parameters 1120, or request content parameters 1122 directly via network 1114 between different systems 1100.

[0117] One technical solution of this specification relates to a method for obtaining object information. The method includes receiving a request start from an occupant of a vehicle. The method includes receiving the request from the occupant after receiving the request start. The method also includes determining the content of the request from the occupant. The method also includes detecting the occupant's gaze position. The method also includes receiving information about the environment surrounding the vehicle based on data collected by sensors installed in the vehicle. The method also includes identifying a region of interest (ROI) outside the vehicle based on the detected gaze position and the information about the environment surrounding the vehicle. The method also includes generating log data based on the ROI and the content of the request. The method also includes sending the log data to an external device. The method also includes receiving information about objects within the ROI that satisfy the request. In one embodiment, receiving the request start includes receiving a request start containing keywords, key phrases, predetermined gestures, or input to a user interface (UI). In one embodiment, receiving information about the environment surrounding the vehicle includes receiving images from a camera installed in the vehicle. In one embodiment, the method also includes cropping the image based on the ROI and generating log data, including generating log data using the cropped image. In some embodiments, receiving object-related information includes responding to a request for object identification by receiving object-related identification information. In some embodiments, the method further includes identifying the occupant, and generating log data includes generating log data based on the occupant's identity. In some embodiments, detecting the occupant's gaze position includes detecting the azimuth angle of the occupant's gaze relative to the vehicle and detecting the elevation angle of the occupant's gaze relative to the vehicle. In some embodiments, detecting the occupant's gaze position also includes detecting the depth of the occupant's gaze relative to the vehicle. In some embodiments, detecting the occupant's gaze position includes detecting the world coordinates of the gaze position, and generating log data includes generating log data based on the world coordinates. In some embodiments, detecting the occupant's gaze position includes capturing images of the occupant using a camera mounted on the vehicle.

[0118] One technical solution of this specification relates to a system for acquiring object information. The system includes an occupant monitoring camera, a front-facing camera, a non-transitory computer-readable medium configured to store commands, and a processor connected to the non-transitory computer-readable medium. The processor is configured to execute commands to initiate receiving a request from an occupant of the vehicle. The processor is also configured to execute commands to receive the request from the occupant after the request has been initiated. The processor is further configured to execute commands to determine the content of the request from the occupant. The processor is also configured to execute commands to detect the occupant's gaze position based on information from the occupant monitoring camera. The processor is further configured to execute commands to receive information about the vehicle's surrounding environment based on the front-facing camera mounted on the vehicle. The processor is further configured to execute commands to identify a region of interest (ROI) outside the vehicle based on the detected gaze position and information about the vehicle's surrounding environment. The processor is further configured to execute commands to generate log data based on the ROI and the content of the request. The processor is further configured to execute commands to generate commands for sending the log data to an external device. The processor is further configured to execute commands to receive information related to an object within the ROI that satisfies the request. In one embodiment, the processor is configured to execute commands for cropping an image from a front-facing camera based on the ROI and generating log data using the cropped image. In another embodiment, the processor is configured to execute commands for responding to a request regarding object recognition and receiving object-related information containing object-related identification information. In another embodiment, the processor is configured to execute commands for identifying the occupant and generating log data based on the occupant's identity. In another embodiment, the processor is configured to execute commands for detecting the occupant's line of sight relative to the vehicle's azimuth angle and for detecting the occupant's line of sight relative to the vehicle's elevation angle. In another embodiment, the processor is configured to execute commands for detecting the occupant's line of sight relative to the vehicle's depth. In yet another embodiment, the processor is configured to execute commands for detecting the world coordinates of the line of sight position and generating log data based on the world coordinates.

[0119] One technical solution of this specification relates to a method for obtaining object information. The method includes receiving a request from a vehicle occupant using a microphone to initiate the process. The method also includes receiving the request from the occupant using a microphone after the request has been initiated. The method further includes detecting the occupant's gaze position. The method also includes receiving information about the environment surrounding the vehicle using a camera mounted on the vehicle. The method further includes generating log data based on the information about the environment surrounding the vehicle and the received request. The method further includes sending the log data to an external device. The method also includes receiving information related to objects within the environment surrounding the vehicle. The method further includes automatically generating an occupant-visible notification in response to the received object-related information. In one embodiment, receiving object-related information includes receiving a link for accessing an external device. In one embodiment, automatically generating the notification includes displaying the link on an occupant-visible user interface.

[0120] The foregoing provides a summary of features of some embodiments to enable those skilled in the art to better understand the technical solutions of this disclosure. Those skilled in the art will understand that this disclosure can be readily used as a basis for designing or modifying other processes and structures to achieve the same objectives and / or obtain the same benefits as the embodiments described in this specification. Furthermore, those skilled in the art will recognize that such equivalent structures do not depart from the spirit and scope of this disclosure, and that various changes, substitutions, and modifications can be made without departing from the spirit and scope of this disclosure.

Claims

1. A method for obtaining object information, comprising: The process begins by receiving a request from the vehicle's occupants; The request is received from the occupant after the request is initiated; The timestamp of the occupant's request is identified; Determine the content of the request from the passenger; Detect the occupant's line of sight position at the timestamp; Based on data collected by sensors installed in the vehicle, information about the environment surrounding the vehicle is received; Based on the detected line-of-sight position and the information about the environment around the vehicle, the region of interest (ROI) outside the vehicle is identified. Identify which type of sensor should be used to satisfy the occupant's request; The occupant's request is correlated with the ROI information, and the occupant's request and the ROI information are transformed into commands that can be used to collect information to satisfy the occupant's request, so as to collect data associated with the occupant's request from the identified sensors; Determine the location of the vehicle at the timestamp; Based on the ROI, the vehicle's location, and the content of the request, log data is generated, and the generated log data is associated with the timestamp information; Send the log data associated with the timestamp information to an external device; and Receive information from the external device that relates to objects within the ROI and satisfies the request.

2. The method according to claim 1, Receiving the request begins by receiving the request containing keywords, key phrases, predetermined gestures, or input to the user interface (UI).

3. The method according to claim 1 or 2, Receiving information about the environment surrounding the vehicle includes receiving images from cameras mounted on the vehicle.

4. The method according to claim 3, It also includes cropping the image based on the ROI. Generating the log data includes generating the log data using the cropped image.

5. The method according to claim 1 or 2, Receiving information related to the object includes responding to a request that the content of the request is about the identification of the object and receiving identification information related to the object.

6. The method according to claim 1 or 2, This also includes identifying the identities of the passengers. Generating the log data includes generating the log data based on the identity of the occupant.

7. The method according to claim 1 or 2, Detecting the occupant's line of sight position includes: Detect the occupant's line of sight relative to the vehicle's azimuth angle, and The angle of elevation of the occupant's line of sight relative to the vehicle is detected.

8. The method according to claim 7, Detecting the occupant's line of sight position also includes detecting the depth of the occupant's line of sight relative to the vehicle.

9. The method according to claim 1 or 2, Detecting the occupant's line of sight position includes detecting the world coordinates of the line of sight position, and generating the log data includes generating the log data based on the world coordinates.

10. The method according to claim 1 or 2, Detecting the occupant's line of sight involves capturing images of the occupant using a camera mounted on the vehicle.

11. An object information acquisition system, which is used to acquire object information, and comprises: Occupant surveillance camera; Front camera; It constitutes a non-transitory computer-readable medium for storing commands; and A processor connected to the non-transitory computer-readable medium, The processor is configured to execute the commands for performing the following processes: The process begins by receiving a request from the vehicle's occupants; The request is received from the occupant after the request is initiated; The timestamp of the occupant's request is identified; Determine the content of the request from the passenger; Based on information from the occupant monitoring camera, the occupant's line of sight at the timestamp is detected; Information about the environment surrounding the vehicle is received based on the front camera installed in the vehicle; Based on the detected line-of-sight position and the information about the environment around the vehicle, the region of interest (ROI) outside the vehicle is identified. Identify what type of sensor should be used to meet the occupant's request; The occupant's request is correlated with the ROI information, and the occupant's request and the ROI information are transformed into commands that can be used to collect information to satisfy the occupant's request, so as to collect data associated with the occupant's request from the identified sensors; Determine the location of the vehicle at the timestamp; Based on the ROI, the vehicle's location, and the content of the request, log data is generated, and the generated log data is associated with the timestamp information; Generate a command for sending the log data associated with the timestamp information to an external device; and Receive information from the external device that relates to objects within the ROI and satisfies the request.

12. The system according to claim 11, The processor is configured to execute the commands for performing the following processes: Cropping the image from the front camera based on the ROI; and The cropped image is used to generate the log data.

13. The system according to claim 11 or 12, The processor is configured to execute the commands for performing the following processes: The system responds to a request that concerns the identification of the object by receiving information related to the object, including identification information associated with the object.

14. The system according to claim 11 or 12, The processor is configured to execute the commands for performing the following processes: Determine the identity of the stated occupants; and The log data is generated based on the identity of the occupant.

15. The system according to claim 11 or 12, The processor is configured to execute the commands for performing the following processes: Detect the occupant's line of sight relative to the vehicle's azimuth angle; and The angle of elevation of the occupant's line of sight relative to the vehicle is detected.

16. The system according to claim 15, The processor is configured to execute the command for detecting the depth of the occupant's line of sight relative to the vehicle.

17. The system according to claim 11 or 12, The processor is configured to execute the commands for performing the following processes: Detect the world coordinates of the line-of-sight position; and The log data is generated based on the world coordinates.

18. A method for obtaining object information, comprising: Begin by using the microphone to receive requests from the vehicle's occupants; Using the microphone, the request is received from the occupant after the request has been initiated; The timestamp of the occupant's request is identified; Detect the occupant's line of sight position at the timestamp; Using a camera installed in the vehicle, information about the environment around the vehicle is received; Identify what type of sensor should be used to meet the occupant's request; Data associated with the occupant's request is collected from the identified sensors; Determine the location of the vehicle at the timestamp; Based on the information about the environment surrounding the vehicle, the vehicle's location, and the received request, log data is generated, and the generated log data is associated with the timestamp information; Send the log data associated with the timestamp information to an external device; Receive information about objects in the environment surrounding the vehicle from the external device; and In response to received information related to the object, the system automatically generates a notification that is visually identifiable to the occupants.

19. The method according to claim 18, Receiving information related to the object includes receiving a link for accessing the external device.

20. The method according to claim 19, The automatic generation of the notification includes displaying the link on a user interface visible to the occupant.

Images