DETERMINING A RELATIONSHIP BETWEEN AN OBJECT AND A HUMAN-PROVIDED INFORMATION LINKED TO THE OBJECT

Abstract

A system for determining a relationship between an object and a human-provided piece of information associated with the object can include a processor and memory. The memory can store an image reception module, a record reception module, a relationship determination module, and a database query module. The image reception module can receive an image that includes a representation of the object but not a representation of the human-provided information associated with the object. The record reception module can receive a record that includes a representation of the human-provided information but not a representation of the object. The relationship determination module can determine whether a relationship exists between the object and the human-provided information.Based on the determination of the existence of the relationship, the database query module can cause a database to generate information about the object in response to a query about a subject of the human-provided information.

Description

TECHNICAL AREA

[0001] The disclosed technologies are designed to determine a relationship between an object and a human-provided piece of information associated with the object. BACKGROUND

[0002] The development of advanced driver assistance systems (ADAS) has led many vehicles to include forward-facing cameras. According to the U.S. Department of Transportation, forward-facing cameras have been mandatory in all production vehicles since 2018. These cameras can be used to support ADAS technologies such as collision warning systems, lane departure warning systems, lane centering systems, lane keeping assist systems, adaptive cruise control systems, traffic sign recognition systems, and similar features. Furthermore, some vehicles are configured to display images from forward-facing cameras on one or more in-vehicle displays.The ability to display images from a forward-facing camera on one or more displays installed in a vehicle can be used, for example, to support ADAS technologies, to record evidence of traffic collisions, to improve visibility for the driver of the vehicle (e.g., if the forward-facing camera provides the driver with an enhanced view of one or more objects in front of the vehicle), and the like. SUMMARY

[0003] In one embodiment, a system for determining a relationship between an object and a human-provided reference associated with the object may include a processor and memory. The memory may store an image reception module, a record reception module, a relationship determination module, and a database query module. The image reception module may include instructions that, when executed by the processor, cause the processor to receive an image containing a representation of the object but not a representation of the human-provided reference associated with the object. The record reception module may include instructions that, when executed by the processor, cause the processor to receive a record containing a representation of the human-provided reference but lacking a representation of the object.The relationship determination module can include instructions that, when executed by the processor, cause the processor to determine whether a relationship exists between the object and the human-provided specification. The database query module can include instructions that, when executed by the processor, cause the processor, based on a determination of the relationship's existence, to generate information about the object from a database in response to a query on a subject of the human-provided specification.

[0004] In another embodiment, a method for determining a relationship between an object and a human-provided reference associated with the object may include receiving an image by a processor that contains a representation of the object but no representation of the human-provided reference associated with the object. The method may also include receiving a record by the processor that contains a representation of the human-provided reference but no representation of the object. Finally, the method may include determining whether the relationship between the object and the human-provided reference exists.The procedure may involve the processor, based on the determination of the existence of the relationship, causing a database to generate information about the object in response to a query about a subject of the human-provided information.

[0005] In another embodiment, a non-transient, computer-readable medium for determining a relationship between an object and a human-provided reference associated with the object may include instructions which, when executed by one or more processors, cause the one or more processors to receive an image containing a representation of the object but lacking a representation of the human-provided reference associated with the object. The non-transmittable, computer-readable medium may include instructions which, when executed by one or more processors, cause the one or more processors to receive a record containing a representation of the human-provided reference but lacking a representation of the object.The non-transitory machine-readable medium may include instructions that, when executed by one or more processors, cause the one or more processors to determine the existence of a relationship between the object and the human-provided information. The non-transitory machine-readable medium may also include instructions that, when executed by one or more processors, cause the one or more processors, based on a determination of the existence of the relationship, to generate information about a database in response to a query about a subject of the human-provided information. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The accompanying drawings, which form part of the description, depict various systems, methods, and other embodiments of the disclosure. The element boundaries shown in the figures (e.g., boxes, groups of boxes, or other shapes) represent one embodiment of the boundaries. In some embodiments, an element may be designed as multiple elements, or multiple elements as one element. In some embodiments, an element shown as an internal component of another element may be implemented as an external component, and vice versa. Furthermore, elements may not be drawn to scale. Fig. 1 includes a diagram that represents an example of an environment at an earlier time in order to determine a relationship between an object and a human-provided indication associated with the object, according to the disclosed technologies. Fig. 2 includes a diagram that provides an example of the environment in which, at a later date, the relationship between the object and the human-provided information associated with the object will be determined according to the disclosed technologies. Fig. Figure 3 is a block diagram that provides an example of a system for determining a relationship between an object and a human-provided piece of information associated with the object, according to the disclosed technologies. Fig. 4A and Fig. 4B includes a diagram illustrating an example of a procedure that involves determining a relationship between an object and a human-provided specification associated with the object, in accordance with the disclosed technologies. Fig. Figure 5 includes a block diagram that provides an example of elements arranged on a vehicle according to the disclosed technologies. DETAILED DESCRIPTION

[0007] The disclosed technologies are designed to determine a relationship between an object and a human-provided piece of information associated with that object. These technologies can enhance database technologies because they allow the existence of such a relationship to be determined without requiring: (1) the existence of the relationship to be predefined in a database schema, or (2) the use of textual information (e.g., via a keyboard interface, speech-to-text technology, or similar methods).

[0008] An image may be received that includes a representation of the object but not a representation of the human-provided cue associated with the object. The human-provided cue may include, for example, one or more hand gestures, a glance, an audible comment, or similar. The image may have been captured by a camera. For example, the object may have been within the camera's field of view at the time the image was taken, but the human-provided cue may have been outside the camera's field of view or otherwise not perceptible to the camera (e.g., the human-provided cue may have been an audible comment). A recording that includes a representation of the human-provided cue but not a representation of the object may be received.

[0009] The existence of a relationship between the object and the human-provided information can be determined. For example: (1) the image may have been generated by the camera at a first time point, (2) the recording may have been generated at a second time point, (3) the human-provided information may denote a specific direction, (4) a location of the object at the second time point may lie in the specific direction from a human who generated the human-provided information, and (5) based on: (a) information about the location of the object at the second time point and (b) information about a relative movement between the camera and the object between the first time point and the second time point, a location of the object at the first time point can be determined to correspond to the representation of the object encompassed in the image.For example, relationship information between the object and the human-provided data can be stored in a database. The database can then be configured to generate information about the object in response to a query about a subject of the human-provided data.

[0010] Fig. Figure 1 comprises a diagram 100, which represents an example of an environment at an earlier time 101, to determine a relationship between an object and a human-provided piece of information associated with the object, according to the disclosed technologies. For example, the diagram 100 may include First Street 102 (along a line of latitude) and Avenue A 103 (along a line of longitude). The diagram 100 may, for example, include a street intersection 104 (e.g., a junction) of First Street 102 and Avenue A 103. First Street 102 may, for example, include a right lane 105 heading west, a left lane 106 heading west, a left lane 107 heading east, and a right lane 108 heading east. For example, Avenue A 102 may include a right lane 109 heading south, a left lane 110 heading south, a left lane 111 heading north, and a right lane 112 heading north.Diagram 100 can, for example, include a bar 113 at a southeast corner of the intersection 104. Diagram 100 can, for example, include a cloud computing platform 114. The cloud computing platform 114 can, for example, include a communication facility 115 and a data storage facility 116.

[0011] Diagram 100 can, for example, include a first vehicle 117, a second vehicle 118, a third vehicle 119, and a fourth vehicle 120. The second vehicle 118, for example, can include one or more of the following: a driver's seat 121, a passenger seat 122, a processor 123, a memory 124, a data storage device 125, a communication device 126, a forward-facing camera 127, a rear-facing camera 128, a cabin-view camera 129, a microphone 130, or a display 131. For example, Ascher 132 can be in the driver's seat 121 and Bryce 133 in the passenger seat 122.The third vehicle 119, for example, can include one or more of the following elements: a driver's seat 134, a passenger seat 135, a processor 136, a memory 137, a data storage device 138, a communication device 139, a forward-facing camera 140, a rear-facing camera 141, a cabin-view camera 142, a microphone 143, or a display 144. For example, Chad 145 can be in the driver's seat 134 and Dylan 146 in the passenger seat 135. The fourth vehicle 120 may, for example, include one or more of the following elements: a driver's seat 147, a passenger seat 148, a processor 149, a memory 150, a data storage device 151, a communication device 152, a forward-facing camera 153, a rear-facing camera 154, a cabin view camera 155, a microphone 156 or a display 157.For example, Evan 158 may be in the driver's seat 147 and Forrest 159 in the passenger seat 148. Diagram 100 may, for example, include a pedestrian 160, Grace Worthington.

[0012] Diagram 100 can, for example, include a first pair of rays 161 and 162, a second pair of rays 163 and 164, and a third pair of rays 165 and 166. For example, the first pair of rays 161 and 162 can define a field of view 167 of the rear-facing camera 128 of the second vehicle 118. For example, the second pair of rays 163 and 164 can define a field of view 168 of the forward-facing camera 140 of the third vehicle 119. The third pair of rays 165 and 166 can, for example, define a field of view 169 of the forward-facing camera 153 of the fourth vehicle 120.

[0013] For example, at earlier time 101: (1) the first vehicle 117 may be located in the left, westbound lane 106 immediately west of the intersection 104 and moving westward, (2) the second vehicle 118 may be located in the left, eastbound lane 107 immediately west of the intersection 104, immediately south of the first vehicle 117, and moving eastward, (3) the third vehicle 119 may be located in the right, northbound lane 112 about thirty meters south of the intersection 104 and moving northward, (4) the fourth vehicle 120 may be located in the right, northbound lane 112 about sixty meters south of the intersection 104 and be parked, and (5) the pedestrian 160 may be on a sidewalk east of the right,The northbound lane 112 is located approximately five meters north of the fourth vehicle 120 and is moving in a southerly direction. Diagram 100 may, for example, include a representation 170 with dashed lines showing the location of the first vehicle 117 at a time prior to the earlier time 101.

[0014] Fig. 2 includes a diagram 200, which presents an example of the environment at a later time 201, to determine the relationship between the object and the human-provided information associated with the object, according to the disclosed technologies. For example, at later time 201: (1) the first vehicle 117 may be located in the left, westbound lane 106 approximately sixty meters west of the intersection 104 and moving westward; (2) the second vehicle 118 may be located in the left, eastbound lane 107 of the intersection 104 and moving eastward; (3) the third vehicle 119 may be located in the right, northbound lane 112 immediately south of the intersection 104; (4) the third vehicle 119 may be located in the right, northbound lane 112 immediately south of the intersection 104, immediately west of the barrier 113.and move in a northerly direction, (4) the fourth vehicle 120 may be located and parked on the right-hand lane 112 in a northerly direction approximately sixty meters south of the intersection 104, and (5) the pedestrian 160 may be located on a sidewalk east of the right-hand lane 112 in a northerly direction immediately east of the fourth vehicle 120 and move in a southerly direction.

[0015] Fig. Figure 3 is a block diagram illustrating an example of a System 300 for determining a relationship between an object and human-provided information associated with the object, according to the disclosed technologies. The System 300 may, for example, comprise a Processor 302 and a Memory 304. The Memory 304 may be communicatively connected to the Processor 302. The Memory 304 may, for example, store an Image Receiving Module 306, a Recording Receiving Module 308, a Relationship Determination Module 310, and a Database Query Module 312.

[0016] For example, the image reception module 306 can include instructions that control the processor 302 to receive an image that contains a representation of the object, but not a representation of the human-provided information associated with the object.

[0017] For example, the recording receiver module 308 can include instructions that control the processor 302 to receive a recording that contains a representation of the information provided by the human, but not a representation of the object.

[0018] The image can be generated by a camera, for example. The recording can be an audio recording, for example. Alternatively or additionally, the image can be a first image and the recording a second image. For example, the first image could be captured by a first camera and the second image by a second camera. The first camera could be, for example, a forward-facing camera or a rear-facing camera on a vehicle. The second camera could be, for example, a cabin-view camera mounted on the vehicle.

[0019] For example, the image may have been created at one point in time and the recording at a second point in time. The second point in time may, for example, be after the first point in time. Alternatively, the second point in time may, for example, be before the first point in time.

[0020] The information provided by a person can include, for example, one or more hand gestures, a glance, an audible comment, or similar means. For example, one or more of the hand gestures can be a gesture pointing in a specific direction, the glance can be directed in that specific direction, the audible comment can include information indicating that specific direction, or similar. Additionally or alternatively, one or more of the hand gestures can express an opinion of the person who made the hand gesture, the glance can express an opinion of the person who made the glance, the audible comment can express an opinion of the person who made the audible comment, or similar.

[0021] For example, the relationship determination module 310 can include instructions that control the processor 302 to determine the existence of the relationship between the object and the information provided by the human.For example: (1) the image may have been generated by a camera at a first time point, (2) the recording may have been generated at a second time point, (3) the human-provided information may denote a specific direction, (4) a location of the object at the second time point may lie in the specific direction of a human who generated the human-provided information, and (5) the instructions for determining the existence of the relationship may include instructions to determine, based on: (a) information about the location of the object at the second time point and (b) information about relative motion between the camera and the object between the first time point and the second time point, that a location of the object at the first time point corresponds to the representation of the object in the image. The relative motion may, for example, be one or more movements of the (e.g.,(camera located on the vehicle) or movement of the object.

[0022] Furthermore, memory 304 can, for example, store a hand gesture module 314. The hand gesture module 314 can, for example, contain instructions that control the processor 302 to initiate an operation of a hand gesture technique in response to a human-provided input containing a hand gesture. The hand gesture technique can, for example, include: (1) operating a gesture recognition technology to determine that the hand gesture is a gesture pointing in the specific direction, and (2) generating a hand gesture vector in the specific direction. The origin of the hand gesture vector can, for example, be a hand positioned to generate the hand gesture.

[0023] Alternatively or additionally, memory 304 can, for example, also store a gaze module 316. For instance, the gaze module 316 can include instructions that control the processor 302 to initiate a gaze technique operation in response to a human-provided input containing a gaze. The gaze technique can, for example, include: (1) operating a moment-point tracking technology to determine that an eye's gaze point is in the specific direction, and (2) generating a gaze vector in the specific direction. The origin of the gaze vector can, for example, be the eye.

[0024] With regard to the Fig. 1 and Fig. 2, for example, with respect to the third vehicle 119: (1) the image may be a first image produced at earlier time 101 by the forward-facing camera 140 and comprising a representation of the bar 113, (2) the recording may be a second image produced at later time 201 by the cabin-view camera 142 and comprising a representation of Dylan 146, (3) a hand gesture vector 202 may be produced from Dylan 146's right hand to the bar 113, and (4) the existence of a relationship between the bar 113 and the hand gesture may be determined by considering, based on: (a) information about the location of the bar 113 at later time 201 and (b) information about a relative motion between the forward-facing camera 140 and the bar 113 between earlier time 101 and later time 201, that a location of the bar 113 at earlier time 201 101 is included with the representation of bar 113 in the first image.Furthermore, the second image may, for example, include a depiction of Dylan 146 making a hand gesture with his left hand that is a thumbs-up gesture, indicating that Dylan 146 has a positive opinion of Bar 113.

[0025] For example, with respect to the fourth vehicle 120: (1) the image may be a first image produced at earlier time 101 by the forward-facing camera 153, comprising a representation of the pedestrian 160; (2) the recording may be a second image produced at later time 201 by the cabin-view camera 155, comprising a representation of Forrest 159 looking in the direction of the pedestrian 160; (3) a gaze vector 203 from one eye of Forrest 159 to the pedestrian 160 may be generated; and (4) the existence of a relationship between the pedestrian 160 and the gaze may be determined based on: (a) information about the location of the pedestrian 160 at later time 201; and (b) information about relative motion between the forward-facing camera 153 and the pedestrian 160 between earlier time 101 and later time 201.that a location of pedestrian 160 at the earlier time 101 corresponds to the representation of pedestrian 160 included in the first image. Furthermore, the disclosed technologies can, for example, employ emotion recognition technology to determine that the representation of Forrest 159 looking in the direction of pedestrian 160, included in the second image, signifies that Forrest 159 has a positive opinion of pedestrian 160.

[0026] For example, with regard to the second vehicle 118: (1) the image may be an image produced at the later time 201 by the rear-facing camera 128 and comprising a representation of the first vehicle 117, (2) the recording may be an audio recording produced at the earlier time 101 by the microphone 130 and comprising a representation of Bryce 133 giving an audible commentary that includes information indicating a direction of the first vehicle 117 (e.g.B, "(3) the existence of a relationship between the first vehicle 117 and the audible comment can be determined by determining, based on: (a) information about the location of the first vehicle 117 at the earlier time 101 and (b) information about relative movement between the rear-facing camera 128 and the first vehicle 117 between the earlier time 101 and the later time 201, that a location of the first vehicle 117 at the later time 201 corresponds to the representation of the first vehicle 117 contained in the image. Furthermore, the disclosed technologies can, for example, employ emotion recognition technology to determine that the representation by Bryce 133, which includes the sound recording and generates the audible comment, is indicative of Bryce 133 having a negative opinion about an operator of the first vehicle 117 (e.g., 'That guy drives like a maniac!')."

[0027] In order to Fig. To return to step 3, memory 304 can, for example, continue to store a database relationship creation module 318. The database relationship creation module 318 can, for example, include instructions that control processor 302 to store relationship information in a database. The relationship information can include, for example: (1) the record that includes the representation of the human-provided specification, (2) information about the existence of the relationship between the object and the human-provided specification, and (3) one or more of: (a) the image that includes the representation of the object, or (b) another image that includes the representation of the object.

[0028] For example, the instructions that cause the information about the relationship to be stored in the database can include instructions to store the information about the relationship in the database. The database can, for example, be stored in a data storage device located in a vehicle. For example, the system 300 can also include a data storage device 320. The data storage device 320 can be communicatively connected to the processor 302. The data in the database can, for example, be intended as a private database for use by the vehicle's occupants.

[0029] Alternatively or additionally, the instructions that cause the relationship information to be stored in the database could, for example, include instructions to: (1) transfer the relationship information to a cloud computing platform and (2) store the relationship information in the database. The database could, for example, be stored in a data store on the cloud computing platform. The data in the database could, for example, be intended as a public database for use by individuals authorized to access the cloud computing platform.

[0030] Additionally, memory 304 can, for example, store an image comment module or image annotation module 322. The image comment module or image annotation module 322 can, for example, include instructions that control processor 302 to annotate or annotate (1) the image containing the representation of the object, or (2) the other image containing the representation of the object, with additional information. The additional information can be based, for example, on one or more of the following: (1) information characterized by the human-provided information, or (2) information generated concurrently with the creation of the human-provided information.The additional information may include, for example, one or more pieces of information about the identification of the object, information about the location of the object, information about a feature of the object, information about a characteristic of the feature of the object, information about an opinion about the object, or similar information.

[0031] With reference to the Fig. 1 and Fig. 2. The image comprising the representation of Bar 113 can be annotated with additional information, for example, in relation to the third vehicle 119. For example, the additional information can be based on the hand gesture produced by Dylan 146's left hand, i.e., the thumbs-up gesture, which indicates that Dylan 146 has a positive opinion about Bar 113. Furthermore, the additional information can be based, for example, on an audio recording produced by microphone 143 simultaneously with the production of the hand gesture produced by Dylan 146's right hand, i.e., the gesture of pointing towards Bar 113, which includes a representation of Dylan 146 producing an audible commentary containing one or more pieces of information about a location of Bar 113, information about a characteristic of Bar 113, information about an opinion about Bar 113, or similar information (e.g.,, “The bar on the corner of First Street and Avenue A with the red double doors is awesome!”).

[0032] With regard to the fourth vehicle 120, for example, the image depicting pedestrian 160 can be annotated with additional information. This additional information could, for instance, be based on a result from emotion recognition technology indicating that Forrest 159 has a positive opinion of pedestrian 160. Furthermore, the additional information could, for example, be based on an audio recording generated by microphone 156 simultaneously with the generation of Forrest 159's gaze toward pedestrian 160, and include a depiction of Evan 158 making an audible comment containing information about the identification of pedestrian 160 (e.g., "Isn't that Grace Worthington?").Furthermore, the disclosed technologies can, for example, use facial recognition technology to determine, based on the representation of pedestrian 160 in the image and the additional information, that the identified pedestrian 160 is Grace Worthington.

[0033] With regard to the second vehicle 118, the image that includes the representation of the first vehicle 117 can, for example, be annotated with additional information. This additional information could, for instance, be based on a result from emotion recognition technology indicating that Bryce 133 has a negative opinion of the operator of the first vehicle 117. Furthermore, the disclosed technologies could, for example, operate automatic number plate recognition (ANPR) to read characters on a vehicle registration plate (i.e., a license plate) of the first vehicle 117. In addition, the additional information could, for instance, be based on a result generated by the ANPR technology that includes the characters on the license plate of the first vehicle 117.

[0034] Back to Fig. Alternatively or additionally, memory 304 can also store, for example, an object recognition and classification module 324. The object recognition and classification module 324 can, for example, contain instructions that control the processor 302 to initiate the recognition and classification of the object.

[0035] Alternatively or additionally, memory 304 can, for example, also store an image transformation module 326. For example, the image transformation module 326 can include instructions that control processor 302 to generate, based on the image containing the representation of the object, another image containing the representation of the object. The other image can, for example, be a transformation of the image. For example: (1) the representation of the object in the image can be associated with a first viewpoint of the object, and (2) the representation of the object in the other image can be associated with a second viewpoint of the object. For example, the second viewpoint can be the viewpoint of a person who generated the human-provided information at the time of production of the recording that includes the representation of the human-provided information.Alternatively, the second viewing angle could be, for example, a viewing angle where a measure of object recognizability is most important. Alternatively, the second viewing angle could be, for example, a viewing angle of an image generated by a different camera. This other camera could be, for example, another camera on the (original) vehicle or a camera on a different vehicle. If the other camera is, for example, the camera on the other vehicle, the image generated by the other camera can be transmitted to the (original) vehicle.

[0036] Alternatively or additionally, the image depicting the object can be part of a series of images depicting the object. For example, the camera that produced the image can be configured to generate images at a specific production rate. This specific production rate can be, for example, ten hertz. Memory 304 can also store an image quality measurement module 328 and an image labeling module 330. The image quality measurement module 328 can, for example, contain instructions that control the processor 302 to select an image from the group of images where the measurement of the object's image quality yields the highest value.For example, the image labeling module can include 330 instructions that control the processor 302 to identify, from the set of images, the one where the measurement of the object's image quality is most important, as the other image that contains the representation of the object. For example, the database relationship creation module can include 318 instructions that control the processor 302 to include the other image in the relationship information stored in the database.

[0037] Alternatively or additionally, memory 304 can, for example, also store a map extension module 332. The map extension module 332 can, for example, include instructions that control processor 302 to cause useful map information to be included in a map of an object's environment. For example, the useful map information can include one or more of the following: (1) one or more of the following: (a) the image containing the representation of the object, or (b) the other image containing the representation of the object, or (2) additional information. The additional information can, for example, be based on one or more of the following: (a) information designated by the human-provided input, or (b) information generated concurrently with the human-provided input.

[0038] For example, the database query module 312 may include instructions that cause the processor 302, based on a determination of the existence of the relationship, to cause a database to generate information about the object in response to a query about a subject of the human-provided specification.

[0039] Additionally, memory 304 can, for example, also store a display presentation module 334. The display presentation module 334 can, for example, contain instructions that control processor 302 to cause the information about the object, generated in response to the query about the subject of the human-provided information, to be displayed on a screen. The screen could, for example, be mounted on a vehicle.

[0040] Fig. 4A and Fig. 4B includes a diagram illustrating an example of a Method 400, which involves determining a relationship between an object and a human-provided piece of information associated with the object, according to the disclosed technologies. Although the Method 400, in combination with the one in Fig. From the description of system 300 shown in Figure 3, a person skilled in the art understands that the method 400 is not limited to the method described in Figure 3. Fig. System 300, as depicted in section 3, is to be implemented. Rather, it is in Fig. System 300, shown in Figure 3, is an example of a system that can be used to carry out Procedure 400. Although Procedure 400 is generally depicted as a serial process, various aspects of Procedure 400 can also be executed in parallel.

[0041] In Fig. 4A, in procedure 400, during operation 402, the image reception module 306 can, for example, receive an image that includes a representation of the object, but does not include a representation of the human-provided information associated with the object.

[0042] In an operation 404, for example, the record-receive module 308 may include instructions that control the processor 302 to receive a recording containing a representation of the information provided by the human, but not a representation of the object.

[0043] The image can be generated by a camera, for example. The recording can be an audio recording, for example. Alternatively or additionally, the image can be a first image and the recording a second image. For example, the first image could be captured by a first camera and the second image by a second camera. The first camera could be, for example, a forward-facing camera or a rear-facing camera on a vehicle. The second camera could be, for example, a cabin-view camera mounted on the vehicle.

[0044] For example, the image may have been created at one point in time and the recording at a second point in time. The second point in time may, for example, be after the first point in time. Alternatively, the second point in time may, for example, be before the first point in time.

[0045] The information provided by a person can include, for example, one or more hand gestures, a glance, an audible comment, or similar means. For example, one or more of the hand gestures can be a gesture pointing in a specific direction, the glance can be directed in that specific direction, the audible comment can include information indicating that specific direction, or similar. Additionally or alternatively, one or more of the hand gestures can express an opinion of the person who made the hand gesture, the glance can express an opinion of the person who made the glance, the audible comment can express an opinion of the person who made the audible comment, or similar.

[0046] In an operation 406, the relationship determination module 310 can, for example, determine the existence of the relationship between the object and the information provided by the human.For example: (1) the image may have been generated by a camera at a first time point, (2) the recording may have been generated at a second time point, (3) the human-provided information may denote a specific direction, (4) a location of the object at the second time point may lie in the specific direction of a human who generated the human-provided information, and (5) the instructions for determining the existence of the relationship may include instructions based on: (a) information about the location of the object at the second time point, and (b) information about a relative motion between the camera and the object between the first time point and the second time point such that a location of the object at the first time point corresponds to the representation of the object in the image. The relative motion may, for example, be one or more movements of the (e.g.,(camera located on the vehicle) or movement of the object.

[0047] In Fig. 4B can, for example, in procedure 400, in an operation 408, cause the database query module 312 to, based on a determination of the existence of the relationship, cause a database to generate information about the object in response to a query about a subject of the human-provided information.

[0048] In Fig. 4A can additionally, in procedure 400, in response to human-provided input that includes a hand gesture, initiate an operation of a hand gesture technique in operation 410. The hand gesture technique may, for example, include: (1) operating a gesture recognition technology to determine that the hand gesture is a gesture pointing in the specific direction, and (2) generating a hand gesture vector in the specific direction. The origin of the hand gesture vector may, for example, be a hand positioned to generate the hand gesture.

[0049] Alternatively or additionally, the gaze module 316 can, for example, in an operation 412, initiate a gaze technique operation in response to human-provided information that includes a gaze. The gaze technique can, for example, include: (1) operating a moment-point tracking technology to determine that an eye's gaze point is in the specific direction, and (2) generating a gaze vector in the specific direction. The origin of the gaze vector can, for example, be the eye.

[0050] Additionally, in an operation 414, the database relationship creation module 318 can, for example, cause relationship information to be stored in a database. For example, the relationship information can include: (1) the record that includes the representation of the human-provided specification, (2) information about the existence of the relationship between the object and the human-provided specification, and (3) one or more of: (a) the image that includes the representation of the object, or (b) another image that includes the representation of the object.

[0051] In operation 414, the database relationship creation module 318 can, for example, store the information about the relationship in the database. The database can be stored, for example, in a data storage device on a vehicle.

[0052] Alternatively or additionally, during operation 414, the database relationship creation module 318 can, for example: (1) transfer the information about the relationship to a cloud computing platform and (2) store the information about the relationship in the database. The database can, for example, be stored in a data store on the cloud computing platform.

[0053] In Fig. 4B can additionally, in operation 416, cause the image commenting module or image annotation module 322 in procedure 400 to, for example, (1) the image containing the representation of the object, or (2) the other image containing the representation of the object, be annotated or provided with additional information. The additional information can be based, for example, on one or more of the following points: (1) Information characterized by the human-provided information, or (2) information generated concurrently with the creation of the human-provided information. The additional information may include, for example, one or more of the following: information about an identification of the object, information about a location of the object, information about a feature of the object, information about a characteristic of the feature of the object, information about an opinion about the object, or similar information.

[0054] Alternatively or additionally, for example in operation 418 the object recognition and classification module 324 can cause the object to be recognized and classified.

[0055] Alternatively or additionally, for example in Operation 420, the image transformation module 326 can generate another image that includes a representation of the object, based on the image that includes the representation of the object. The other image can, for example, be a transformation of the image. For example: (1) the representation of the object in the image can be associated with a first viewpoint of the object, and (2) the representation of the object in the other image can be associated with a second viewpoint of the object. For example, the second viewpoint can be the viewpoint of a person who created the human-provided information at the time of production of the record that includes the representation of the human-provided information. Alternatively, the second viewpoint can, for example, be a viewpoint at which a measure of the object's recognizability has the highest value.Alternatively, the second viewpoint could be, for example, a viewpoint of an image generated by another camera. This other camera could be, for instance, another camera on the (original) vehicle or a camera on a different vehicle. If the other camera is, for example, the camera on the other vehicle, the image generated by the other camera can be transmitted to the (original) vehicle.

[0056] Alternatively or additionally, the image that comprises the representation of the object can be part of a set of images that comprise the representation of the object. For example, the camera that produced the image can be configured to produce images at a specific production rate. For example, the specific production rate can be ten hertz. For example, in operation 422, the image quality measurement module 328 can determine an image from the set of images where a measurement of the object's image quality has the highest value. For example, in operation 424, the image labeling module 330 can designate the image from the set of images where the measurement of the object's image quality has the highest value as the other image that comprises the representation of the object. For example, the database relationship creation module 318 can include the other image in the relationship information stored in the database.

[0057] Alternatively or additionally, the map enhancement module 332 can, for example, in an operation 426, cause useful map information to be included in a map of the object's environment. For example, the useful map information can include one or more of the following: (1) one or more of the following: (a) the image containing the representation of the object, or (b) the other image containing the representation of the object, or (2) additional information. The additional information can be, for example, based on one or more of the following: (a) information designated by the human-provided input, or (b) information generated concurrently with the human-provided input.

[0058] Furthermore, the display presentation module 334 can, for example, in an operation 428, cause the information about the object, generated in response to the query about the subject of the human-provided information, to be displayed on a screen. The screen could, for example, be mounted on a vehicle.

[0059] Fig. Figure 5 includes a block diagram illustrating an example of elements arranged on a vehicle 500 in accordance with the disclosed technologies. As used herein, a “vehicle” can be any form of motorized transport. In one or more embodiments, the vehicle 500 can be an automobile. Although the arrangements described herein relate to motor vehicles, the person skilled in the art understands, in view of the description contained herein, that the embodiments are not limited to motor vehicles. For example, functions and / or operations of one or more of the second vehicles 118 (illustrated in the Fig. 1 and Fig. 2), of the third vehicle 119 (shown in the Fig. 1 and Fig. 2) or the fourth vehicle 120 (shown in the Fig. 1 and Fig. 2) be achieved by vehicle 500.

[0060] In some embodiments, the vehicle 500 can be configured to selectively switch between an automated mode, one or more semi-automatic operating modes, and / or a manual mode. Such switching can be implemented in a suitable manner, now known or later developed. As used herein, the term "manual mode" can refer to the fact that all or most of the navigation and / or maneuvering of the vehicle 500 is performed according to inputs received from a user (e.g., a human driver). In one or more arrangements, the vehicle 500 can be a conventional vehicle configured to operate only in a manual mode.

[0061] In one or more embodiments, the vehicle 500 may be an automated vehicle. As used herein, “automated vehicle” may refer to a vehicle operating in an automated mode. As used herein, “automated mode” may refer to the navigation and / or maneuvering of the vehicle 500 along a route using one or more computer systems to control the vehicle 500 with minimal or no input from a human driver. In one or more embodiments, the vehicle 500 may be highly automated or fully automated. In one embodiment, the vehicle 500 may be configured with one or more semi-automatic operating modes in which one or more computer systems perform part of the navigation and / or maneuvering of the vehicle along a route, and a vehicle operator (i.e., a driver)the driver) provides input to the vehicle 500 to perform part of the navigation and / or maneuvering of the vehicle 500 along a route.

[0062] The standard J3016 202104, Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles, published by the Society of Automotive Engineers (SAE) International on January 16, 2014, and last revised on April 30, 2021, defines, for example, six levels of driving automation. These six levels include: (1) Level 0, no automation, where all aspects of driving dynamics tasks are performed by the human driver; (2) Level 1, driver assistance, where a driver assistance system, if selected, can perform either steering or acceleration / deceleration tasks using information about the driving environment, but all other driving dynamics tasks are performed by the human driver;(3) Level 2, partial automation, in which one or more driver assistance systems, if selected, can perform both steering and acceleration / deceleration tasks using information about the driving environment, but all other driving dynamics tasks are performed by a human driver; (4) Level 3, conditional automation, in which an automated driving system, if selected, can perform all aspects of driving dynamics tasks in the expectation that a human driver will respond appropriately to a request to intervene; (5) Level 4, highly automated driving, in which an automated driving system, if selected, can perform all aspects of dynamic driving tasks even if a human driver does not respond appropriately to a request to intervene;and (6) Level 5, fully automated driving, in which an automated driving system is able to perform all aspects of dynamic driving tasks under all road and environmental conditions that can be handled by a human driver.

[0063] Vehicle 500 can comprise various elements. Vehicle 500 can be any combination of the different elements in Fig. exhibit the 5 elements shown. In various embodiments, it is not necessary for the vehicle 500 to have all of the elements shown. Fig. The vehicle comprises the 5 elements shown. Furthermore, the vehicle can carry 500 additional items beyond those shown. Fig. The five elements shown contain further elements. While the various elements in Fig. Although the elements are depicted as being located inside the vehicle 500, one or more of these elements may also be located outside the vehicle 500. Furthermore, the depicted elements may be spatially separated by large distances. For example, as described, one or more components of the disclosed system may be implemented inside the vehicle 500, while other components of the system may be implemented in a cloud computing environment, as described below. The elements may, for example, include one or more processors 510, one or more data storage devices 515, a sensor system 520, an input system 530, an output system 535, vehicle systems 540, one or more actuators 550, one or more autonomous driving modules 560, a communication system 570, and the system 300 to determine a relationship between an object and a human-provided indication associated with the object.

[0064] In one or more arrangements, the one or more processors 510 can be a main processor of the vehicle 500. For example, the one or more processors 510 can be an electronic control unit (ECU). For example, functions and / or operations of one or more of the processors 123 (shown in Fig. 1 and Fig. 2), of processor 136 (shown in Fig. 1 and Fig. 2), of processor 149 (shown in Fig. 1 and Fig. 2) or of processor 302 (shown in Fig. 3) can be implemented by one or more 510 processors.

[0065] The one or more data memories 515 can, for example, store one or more types of data. The one or more data memories 515 can include volatile memory and / or non-volatile memory. Examples of suitable memory for the one or more data memories 515 include random-access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, magnetic disks, optical disks, hard disks, any other suitable storage medium, or a combination thereof. The one or more data memories 515 can be a component of the one or more processors 510. Additionally or alternatively, the one or more data memories 515 can be operationally connected to the one or more processors 510 for use.As used herein, “operationally connected” can include direct or indirect connections, including connections without direct physical contact. As used herein, the statement that a component can be “configured” to perform an operation can be understood to mean that the component does not require any structural changes, but merely needs to be placed in an operational state (e.g., supplied with electrical power, running an underlying operating system, etc.) in order to perform the operation. Thus, for example, functions and / or operations of one or more of the following elements can be performed: Memory 124 (represented in . Fig. 1 and Fig. 2), Data storage 125 (shown in Fig. 1 and Fig. 2), Memory 137 (shown in Fig. 1 and Fig. 2), Data storage 138 (shown in Fig. 1 and Fig. 2), the storage 150 (shown in Fig. 1 and Fig. 2), the data storage device 151 (shown in Fig. 1 and Fig. 2), memory 304 (shown in Fig. 3) or the data storage 320 (shown in Fig. 3) can be realized by one or more data storage devices 515.

[0066] In one or more arrangements, the one or more data stores 515 can store map data 516. The map data 516 can include maps of one or more geographic areas. In some cases, the map data 516 can include information or data about roads, traffic control devices, road markings, structures, features, and / or landmarks in the one or more geographic areas. The map data 516 can be in any suitable format. In some cases, the map data 516 can include aerial photographs of an area. In some cases, the map data 516 can include ground views of an area, including 360-degree ground views. The map data 516 can include measurements, dimensions, distances, and / or information for one or more elements contained in the map data 516 and / or relating to other elements in the map data 516.Map data 516 can include a digital map with information about road geometry. Map data 516 can be of high quality and / or very detailed.

[0067] In one or more arrangements, the map data 516 may include one or more terrain maps 517. The one or more terrain maps 517 may include information about the soil, terrain, roads, surfaces, and / or other features of one or more geographic areas. The one or more terrain maps 517 may include elevation data of the one or more geographic areas. The map data 516 may be of high quality and / or very detailed. The one or more terrain maps 517 may define one or more land areas, which may include paved roads, unpaved roads, land, and other features that define a land area.

[0068] In one or more arrangements, the map data 516 may include one or more maps of static obstacles 518. The one or more maps of static obstacles 518 may contain information about one or more static obstacles located in one or more geographic areas. A "static obstacle" may be a physical object whose position does not change (or does not change significantly) over a period of time and / or whose size does not change (or does not change significantly) over a period of time. Examples of static obstacles may include trees, buildings, curbs, fences, railings, medians, utility poles, statues, monuments, signs, benches, furniture, mailboxes, large rocks, and hills. The static obstacles may be objects that protrude above ground level.The one or more static obstacles included in the one or more static obstacle maps 518 may contain location data, size data, dimension data, material data, and / or other related data. The one or more static obstacle maps 518 may include measurements, dimensions, distances, and / or information for one or more static obstacles. The one or more static obstacle maps 518 may be of high quality and / or very detailed. The one or more static obstacle maps 518 may be updated to reflect changes within a mapped area.

[0069] In one or more arrangements, the one or more data storage devices 515 can store sensor data 519. As used here, the term "sensor data" can refer to all information about the sensors with which the vehicle 500 may be equipped, including the capabilities and other information about such sensors. The sensor data 519 can refer to one or more sensors of the sensor system 520. For example, in one or more arrangements, the sensor data 519 can include information about one or more lidar sensors 524 of the sensor system 520.

[0070] In some arrangements, at least some of the map data 516 and / or the sensor data 519 may be located in one or more data storage devices 515 on board the vehicle 500. Additionally or alternatively, at least some of the map data 516 and / or the sensor data 519 may be stored in one or more data storage devices 515 located remotely from the vehicle 500.

[0071] The Sensor System 520 can include one or more sensors. As used herein, a "sensor" can refer to any device, component, and / or system capable of detecting and / or sensing something. The one or more sensors can be configured to detect and / or measure something in real time. As used herein, the term "real time" can refer to a processing response that is perceived by a user or system as sufficiently immediate for a particular process or purpose, or that enables the processor to keep pace with an external process.

[0072] In arrangements where the sensor system 520 comprises a plurality of sensors, the sensors can operate independently of one another. Alternatively, two or more of the sensors can operate in combination. In such a case, the two or more sensors can form a sensor network. The sensor system 520 and / or the one or more sensors can operate with the one or more processors 510, the one or more data stores 515, and / or another element of the vehicle 500 (including one of those in Fig. The sensor system 520 can be connected to at least one part of the external environment of the vehicle 500 (e.g., from nearby vehicles). The sensor system 520 can include any suitable type of sensor. Several examples of different types of sensors are described here. However, a person skilled in the art will understand that the embodiments are not limited to the sensors described here.

[0073] The sensor system 520 can comprise one or more vehicle sensors 521. The one or more vehicle sensors 521 can acquire, determine, and / or perceive information about the vehicle 500 itself. In one or more arrangements, the one or more vehicle sensors 521 can be configured to acquire and / or query changes in the position and orientation of the vehicle 500, such as those based on inertial acceleration. In one or more arrangements, the one or more vehicle sensors 521 can comprise one or more accelerometers, one or more gyroscopes, an inertial measurement unit (IMU), a dead reckoning navigation system, a global navigation satellite system (GNSS), a global positioning system (GPS), a navigation system 547, and / or other suitable sensors.The one or more vehicle sensors 521 can be configured to detect and / or sample one or more characteristics of the vehicle 500. In one or more arrangements, the one or more vehicle sensors 521 can include a speedometer to determine the current speed of the vehicle 500.

[0074] Additionally or alternatively, the sensor system 520 may include one or more environmental sensors 522 configured to obtain and / or record driving environment data. As used herein, “driving environment data” may include data or information about the external environment in which a vehicle or one or more parts thereof are located. For example, the one or more environmental sensors 522 may be configured to detect, quantify, and / or record obstacles in at least one part of the external environment of the vehicle 500 and / or information / data about such obstacles. Such obstacles may be stationary objects and / or dynamic objects. The one or more environmental sensors 522 may be configured to detect, measure, quantify, and / or record other things in the external environment of the vehicle 500, such as…Lane markings, signs, traffic lights, traffic signs, lane lines, pedestrian crossings, curbs near the vehicle 500, objects in the terrain, etc. For example, functions and / or operations of sensor 230 (shown in . Fig. 2) can be achieved by one or more environmental sensors 522.

[0075] Several examples of sensors of the sensor system 520 are described here. The example sensors can be part of one or more vehicle sensors 521 and / or one or more environmental sensors 522. However, those skilled in the art understand that the embodiments are not limited to the sensors described.

[0076] In one or more arrangements, the one or more environmental sensors 522 can comprise one or more radar sensors 523, one or more lidar sensors 524, one or more sonar sensors 525, and / or one or more cameras 526. In one or more arrangements, the one or more cameras 526 can be one or more high dynamic range (HDR) cameras or one or more infrared (IR) cameras. For example, the one or more cameras 526 can be used to record a reality of the state of an information element that may appear in the digital map. For example, functions and / or operations of one or more of the forward-facing cameras 127 (shown in Fig. 1 and Fig. 2), the rear-facing camera 128 (shown in Fig. 1 and Fig. 2), the cabin view camera 129 (shown in Fig. 1 and Fig. 2), the forward-facing camera 140 (shown in the Fig. 1 and Fig. 2), the rear-facing camera 141 (shown in the Fig. 1 and Fig. 2), the cabin view camera 142 (shown in the Fig. 1 and Fig. 2), the forward-facing camera 153 (shown in Fig. 1 and Fig. 2), the rear-facing camera 154 (shown in Fig. 1 and Fig. 2) or the cabin view camera 155 (shown in Fig. 1 and Fig. 2) can be achieved by one or more cameras 526.

[0077] The input system 530 can include all devices, components, systems, elements, arrangements, or groups thereof that enable the input of information / data into a machine. The input system 530 can receive input from a vehicle occupant (e.g., a driver or a passenger). The output system 535 can include all devices, components, systems, elements, arrangements, or groups thereof that enable the presentation of information / data to a vehicle occupant (e.g., a driver or a passenger). For example, functions and / or operations of one or more of the microphones 130 (shown in Fig. 1 and Fig. 2), of microphone 143 (shown in Fig. 1 and Fig. 2) or microphone 156 (shown in Fig. 1 and Fig. 2) can be implemented by the input system 530. For example, functions and / or operations of one or more of the displays 131 (shown in Fig. 1 and Fig. 2), of the display 144 (shown in Fig. 1 and Fig. 2) or of display 157 (shown in Fig. 1 and Fig. 2) are implemented through the output system 535.

[0078] Several examples of the one or more vehicle systems 540 are in Fig. Figure 5 illustrates this. However, a person skilled in the art understands that the vehicle 500 may comprise more, fewer, or different vehicle systems. Although certain vehicle systems may be defined separately, each of the systems or parts thereof may be combined in other ways or separated by hardware and / or software within the vehicle 500. For example, the one or more vehicle systems 540 may comprise a propulsion system 541, a braking system 542, a steering system 543, a throttle system 544, a transmission system 545, a signaling system 546, and / or the navigation system 547. Each of these systems may comprise one or more devices, components, and / or a combination thereof, which are known now or may be developed later.

[0079] The navigation system 547 may include one or more devices, applications, and / or combinations thereof, now known or subsequently developed, configured to determine the geographic location of the vehicle 500 and / or to determine a travel route for the vehicle 500. The navigation system 547 may include one or more mapping applications to determine a travel route for the vehicle 500. The navigation system 547 may include a global positioning system, a local positioning system, a geolocation system, and / or a combination thereof.

[0080] The one or more actuators 550 can be any element or combination of elements that, in response to receiving signals or other inputs from the one or more processors 510 and / or the one or more autonomous driving modules 560, can modify, adjust, and / or change one or more of the vehicle systems 540 or their components. Any suitable actuator can be used. For example, the one or more actuators 550 can include motors, pneumatic actuators, hydraulic pistons, relays, solenoids, and / or piezoelectric actuators.

[0081] The one or more processors 510 and / or the one or more autonomous driving modules 560 can be operationally connected to communicate with the various vehicle systems 540 and / or individual components thereof. For example, the one or more processors 510 and / or the one or more autonomous driving modules 560 can communicate to send and / or receive information from the various vehicle systems 540 in order to control the movement, speed, maneuvering, course, direction, etc., of the vehicle 500. The one or more processors 510 and / or the one or more autonomous driving modules 560 can control some or all of these vehicle systems 540 and can therefore be partially or fully automated.

[0082] The one or more processors 510 and / or the one or more autonomous driving modules 560 can be operated to control the navigation and / or maneuvering of the vehicle 500 by controlling one or more of the vehicle systems 540 and / or their components. For example, when operating in an automated mode, the one or more processors 510 and / or the one or more autonomous driving modules 560 can control the direction and / or speed of the vehicle 500. The one or more processors 510 and / or the one or more autonomous driving modules 560 can cause the vehicle 500 to accelerate (e.g., by increasing the fuel supply to the engine), decelerate (e.g., by decreasing the fuel supply to the engine and / or by applying the brakes), and / or change direction (e.g., by turning the two front wheels).As used herein, “to cause” or “to induce” can mean to bring about, to compel, to force, to direct, to command, to instruct and / or to enable an event or action, or at least to be in a state in which such an event or action may occur either directly or indirectly.

[0083] The communication system 570 can include one or more receivers 571 and / or one or more transmitters 572. The communication system 570 can receive and send one or more messages over one or more wireless communication channels. For example, the one or more wireless communication channels can be configured according to IEEE (Institute of Electrical and Electronics Engineers) standard 802.The 11p standard for wireless access in vehicle environments (WAVE) (the basis for Dedicated Short-Range Communications (DSRC)), the 3rd Generation Partnership Project (3GPP) Long-Term Evolution (LTE) Vehicle-to-Everything (V2X) standard (including the LTE Uu interface between a mobile communication device and an Evolved Node B of the Universal Mobile Telecommunications System), the 3GPP New Radio (NR) fifth generation (5G) Vehicle-to-Everything (V2X) standard (including the 5G NR Uu interface), or similar. The Communication System 570, for example, may include "connected vehicle" technology.Connected Vehicle (V2V) technology can include, for example, devices for exchanging communications between a vehicle and other devices in a packet-switched network. Such other devices can include, for example, another vehicle (e.g., Vehicle-to-Vehicle (V2V) technology), roadside infrastructure (e.g., Vehicle-to-Infrastructure (V2I) technology), a cloud platform (e.g., Vehicle-to-Cloud (V2C) technology), a pedestrian (e.g., Vehicle-to-Pedestrian (V2P) technology), or a network (e.g., Vehicle-to-Network (V2N) technology). Vehicle-to-Everything (V2X) technology can integrate aspects of these individual communication technologies. For example, functions and / or operations of one or more of the communication devices 126 (shown in . Fig. 1 and Fig. 2), the communication device 139 (shown in Fig. 1 and Fig. 2) or the communication device 152 (shown in Fig. 1 and Fig. 2) be implemented through the 570 communication system.

[0084] Furthermore, the one or more processors 510, the one or more data storage devices 515, and the communication system 570 can be configured to form one or more microclouds, participate as a member of a microcloud, or assume the role of a microcloud leader. A microcloud can be characterized by the distribution of one or more computing resources or one or more data storage resources among the microcloud members to cooperate in executing operations. The members can include at least connected vehicles.

[0085] The vehicle 500 can comprise one or more modules, at least some of which are described herein. The modules can be implemented as computer-readable program code which, when executed by the one or more processors 510, implements one or more of the various processes described herein. One or more of the modules can be a component of the one or more processors 510. Additionally or alternatively, one or more of the modules can be executed on and / or distributed to other processing systems with which the one or more processors 510 can be operationally connected. The modules can comprise instructions (e.g., program logic) that can be executed by the one or more processors 510. Additionally or alternatively, the one or more data storage devices 515 can contain such instructions.

[0086] In one or more configurations, one or more of the modules described herein may include elements of artificial or computational intelligence, such as neural networks, fuzzy logic, or other machine learning algorithms. Furthermore, in one or more configurations, one or more of the modules may be distributed across a multitude of the modules described herein. In one or more configurations, two or more of the modules described herein may be combined into a single module.

[0087] The vehicle 500 can include one or more autonomous driving modules 560. The one or more autonomous driving modules 560 can be configured to receive data from the sensor system 520 and / or any other type of system capable of acquiring information about the vehicle 500 and / or the vehicle 500's external environment. In one or more configurations, the one or more autonomous driving modules 560 can use such data to generate one or more driving scene models. The one or more autonomous driving modules 560 can determine the position and speed of the vehicle 500. The one or more autonomous driving modules 560 can determine the position of obstacles, barriers, or other environmental features, including traffic signs, trees, bushes, neighboring vehicles, pedestrians, etc.

[0088] The one or more autonomous driving modules 560 can be configured to receive and / or determine location information for obstacles in the external environment of the vehicle 500, which can be used by the one or more processors 510 and / or one or more of the modules described herein to estimate the position and orientation of the vehicle 500, the vehicle position in global coordinates based on signals from a variety of satellites or other data and / or signals that could be used to determine the current state of the vehicle 500 or to determine the position of the vehicle 500 in relation to its environment, either to create a map or to determine the position of the vehicle 500 in relation to map data.

[0089] The one or more autonomous driving modules 560 can be configured to determine one or more driving paths, current automated driving maneuvers for the vehicle 500, future automated driving maneuvers, and / or changes to current automated driving maneuvers based on data acquired by the sensor system 520, driving scene models, and / or data from another suitable source, such as determinations from the sensor data 519. As used herein, "driving maneuver" can refer to one or more actions that affect the movement of a vehicle. Examples of driving maneuvers include: accelerating, decelerating, braking, turning, moving the vehicle 500 laterally, changing lanes, interfering with a lane, and / or reversing, to name only a few. The one or more autonomous driving modules 560 can be configured to perform specific driving maneuvers.The one or more autonomous driving modules 560 can directly or indirectly cause such automated driving maneuvers to be performed. As used herein, "cause" or "provoke" means the device to initiate, command, instruct, and / or enable an event or action to occur, or at least to be in a state in which such an event or action may occur, either directly or indirectly. The one or more autonomous driving modules 560 can be configured to perform various vehicle functions and / or to transmit data to, receive data from, interact with, and / or control the vehicle 500 or one or more of its systems (e.g., one or more of the vehicle systems 540).For example, functions and / or operations of a vehicle navigation system can be implemented by one or more modules for autonomous driving 560.

[0090] Detailed embodiments are disclosed herein. However, the person skilled in the art understands from the description contained herein that the disclosed embodiments are intended only as examples. Therefore, specific structural and functional details disclosed herein are not to be understood as limiting, but merely as a basis for the claims and as a representative basis to show the person skilled in the art how to incorporate the aspects contained herein into virtually any appropriately detailed structure. Furthermore, the terms and expressions used herein are not to be understood as limiting, but rather are intended to provide an understandable description of possible embodiments. Various embodiments are described in the Fig. Figures 1-3, 4A, 4B and 5 are shown, but the embodiments are not limited to the structure or application shown.

[0091] The flowcharts and block diagrams in the figures represent the architecture, functionality, and operation of possible implementations of systems, processes, and computer program products according to various embodiments. In this respect, each block in flowcharts or block diagrams can represent a module, segment, or part of the code that contains one or more executable instructions for implementing the specified logical function(s). Those skilled in the art know that in some alternative implementations, the functions described in a block may not execute in the sequence shown in the figures. For example, two blocks shown consecutively may, in reality, be executed essentially simultaneously, or the blocks may be executed in reverse order, depending on the functionality involved.

[0092] The systems, components, and / or processes described above can be implemented in hardware or a combination of hardware and software, and can be centralized in a processing system or decentralized, i.e., with various elements distributed across several interconnected processing systems. Any type of processing system or other device suitable for carrying out the procedures described herein is suitable. A typical combination of hardware and software might be a processing system with computer-readable program code that, when loaded and executed, controls the processing system to perform the procedures described herein. The systems, components, and / or processes might also be embedded in computer-readable memory, such as…A computer program product or other device for storing data programs that can be read by a machine and that contains a program with instructions that can be executed by the machine to carry out the procedures and processes described herein. These elements may also be embedded in an application product that has all the features necessary to carry out the procedures described herein and that, when loaded into a processing system, is capable of executing these procedures.

[0093] Furthermore, the arrangements described here can take the form of a computer program product embodied in one or more computer-readable media containing, for example, stored, computer-readable program code. Any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signaling medium or a computer-readable storage medium. The term "computer-readable storage medium" here means a device without a transit function. A computer-readable storage medium may be, for example, but not exclusively, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or arrangement, or a suitable combination thereof.More specific examples of computer-readable storage media would include, in a non-exhaustive list, the following: a portable computer floppy disk, a hard disk drive (HDD), a solid-state drive (SSD), a read-only memory (ROM), a erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. A computer-readable storage medium can be any tangible medium capable of containing or storing a program for use by or in conjunction with a command-execution system, device, or apparatus, as used herein.

[0094] In general, modules, as used here, comprise routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific data types. In other respects, memory generally stores such modules. The memory associated with a module may be a buffer or cache embedded in a processor, random access memory (RAM), ROM, flash memory, or another suitable electronic storage medium. In still other respects, a module used herein may be implemented as an application-specific integrated circuit (ASIC), as a hardware component of a system-on-a-chip (SoC), as a programmable logic array (PLA), or as another suitable hardware component (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), or the like) that is configured with a defined set of settings (e.g.,Instructions) for executing the disclosed functions are embedded.

[0095] The program code embodied on a computer-readable medium may be transmitted via any suitable medium, including but not limited to wireless, wired, fiber-optic, cabled, radio frequency (RF) transmission, etc. Computer program code for performing operations on aspects of the disclosed technologies may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java™, Smalltalk, C++, or similar languages, and conventional procedural programming languages ​​such as the programming language "C" or similar languages.The program code can run entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (e.g., via the internet using an internet service provider).

[0096] The term "one," as used here, is defined as one or more than one. The term "multiple," as used here, is defined as two or more than two. The term "another," as used here, is defined as at least one other or more. The expressions "including" and / or "with" used here are defined to encompass the following (i.e., open language). The phrase "at least one of... or..." as used here refers to and includes all possible combinations of one or more of the linked listed items. For example, the phrase "at least one of A, B, or C" includes only A, only B, only C, or any combination thereof (e.g., AB, AC, BC, or ABC). The aspects contained herein can be embodied in other forms without deviating from the basic idea or the essential properties. Accordingly, reference should be made to the following claims and not to the preceding description to specify the scope of the present invention.

Claims

[1] System, with: a processor; and a storage device that stores: an image reception module comprising instructions which, when executed by the processor, cause the processor to receive an image comprising a representation of an object, but not comprising a representation of any human-provided information associated with the object; a recording / receiving module comprising instructions which, when executed by the processor, cause the processor to receive a recording that includes the representation of the information provided by the human, but does not include the representation of the object; a relationship determination module comprising instructions which, when executed by the processor, cause the processor to determine the existence of a relationship between the object and the human-provided information; and A database query module comprising instructions which, when executed by the processor, cause the processor, based on a determination of its existence, to induce a database to generate information about the object in response to a query about a subject of human-provided information. [2] System according to claim 1, wherein at least one(s): the recording is an audio recording, or of the image is a first image and of the recording is a second image. [3] System according to claim 2, wherein: the first image was created by a first camera, and the second image was created by a second camera. [4] System according to claim 3, wherein: The first camera is at least one of the following: a forward-facing camera mounted on a vehicle, or a rear-facing camera mounted on the vehicle, and The second camera is a cabin view camera mounted inside the vehicle. [5] System according to claim 1, wherein the human-provided information comprises at least one of the following: a hand gesture, a look, or a more audible comment. [6] System according to claim 5, wherein at least one: The hand gesture is a gesture to point in a specific direction, the gaze is directed in a specific direction, or The audible commentary includes information that indicates the specific direction. [7] System according to claim 5, wherein at least one: The hand gesture represents the opinion of the person who created the hand gesture. The gaze refers to the opinion of a person who created the gaze, or The term "audible comment" refers to the opinion of a person who produced the audible comment. [8] System according to claim 1, wherein: the image was created by a camera at a first point in time, the recording was created at a second time, the information provided by humans indicates a specific direction, a location of the object at the second time lies in the specific direction of a human who generated the human-provided information, and The instructions for determining the existence of the relationship include instructions for determining based on: Information about the location of the object at the second time, and Information about a relative movement between the camera and the object between the first time point and the second time point, that a location of the object corresponds to the first time of representation of the object that is included in the image. [9] System according to claim 8, wherein the memory further stores at least one of the following: a hand gesture module comprising instructions which, when executed by the processor, cause the processor to initiate an operation of a hand gesture technique in response to the human-provided input comprising a hand gesture, wherein the hand gesture technique comprises: Operating a gesture recognition technology to determine that the hand gesture is a gesture to point in the specific direction, and Generating a hand gesture vector in the specific direction, wherein an origin of the hand gesture vector is a hand positioned to generate the hand gesture, or a gaze module comprising instructions which, when executed by the processor, cause the processor to initiate an operation of a gaze technique in response to the human-provided input comprising a gaze technique, wherein the gaze technique comprises: Operating a momentary point tracking technology to determine that a gaze point of an eye is pointing in the specific direction, and Generating a gaze vector in the specific direction, where one origin of the gaze vector is the eye. [10] System according to claim 1, wherein the memory further comprises a database relationship creation module comprising instructions which, when executed by the processor, cause the processor to initiate the storage of relationship information in the database, wherein the relationship information comprises: the record, which includes the representation of the information provided by the human, Information about the existence of a relationship between the object and the information provided by the human, and at least one of the images that includes a representation of the object, or another image that includes a representation of the object. [11] System according to claim 10, wherein the memory further stores an image annotation module comprising instructions which, when executed by the processor, cause the processor to annotate at least one of the images comprising the representation of the object, or the other image comprising the representation of the object, with additional information, wherein the additional information is based on at least one of the information designated by the human-provided specification, or of the information generated concurrently with the generation of the human-provided specification. [12] System according to claim 10, wherein the memory further comprises an object recognition and classification module comprising instructions which, when executed by the processor, cause the processor to cause the object to be recognized and classified. [13] System according to claim 10, wherein the memory further stores an image transformation module comprising instructions which, when executed by the processor, cause the processor to generate, based on the image comprising the representation of the object, the other image comprising the representation of the object, wherein the other image is a transformation of the image. [14] System according to claim 10, wherein: the image that includes the representation of the object, is an element of a set of images that include the representation of the object, and the memory continues to store: an image quality measurement module comprising instructions which, when executed by the processor, cause the processor to determine an image from the set of images where a measurement of the object's image quality is most valuable; and an image labeling module comprising instructions which, when executed by the processor, cause the processor to designate the image from the set of images where the measurement of the object's image quality is most significant as the other image comprising the representation of the object. [15] System according to claim 10, wherein the memory further stores a map extension module comprising instructions which, when executed by the processor, cause the processor to cause useful map information to be incorporated into a map of an environment of a location of the object, wherein the useful map information comprises at least one of the following: at least one of the images that includes the representation of the object, or of the other image that includes the representation of the object, or additional information, wherein the additional information is based on at least one of the pieces of information designated by the human-provided information or on information generated concurrently with the generation of the human-provided information. [16] Methods, with: Received, by a processor, an image that includes a representation of an object but does not include a representation of any human-provided information associated with the object; Received, by a processor, a recording that includes the representation of the information provided by the human, but does not include the representation of the object; Determine, by the processor, the existence of a relationship between the object and the information provided by the human; and To cause the processor, based on a determination of the existence of a database, to generate information about the object in response to a query about a subject of the human-provided information. [17] Method according to claim 16, wherein: the image was created at a first point in time, and the recording was created at a second time. [18] Method according to claim 17, wherein the second time point is after the first time point. [19] Method according to claim 17, wherein the second time point is before the first time point. [20] Non-transitory computer-readable medium for determining a relationship between an object and a human-provided indication associated with the object, wherein the non-transitory computer-readable medium comprises instructions which, when executed by one or more processors, cause the one or more processors to: to receive an image that includes a representation of the object, but does not include a representation of the human-provided information associated with the object; to receive a recording that includes a representation of the information provided by the human, but does not include a representation of the object; to determine the existence of a relationship between the object and the information provided by the human; and based on a determination of the existence of the relationship, to cause a database to generate information about the object in response to a query about a subject of the human-provided information.

Images