Automatic reference lookup in audio-visual scenes

Abstract

Automatic reference lookup in audiovisual scenes. Embodiments of the present disclosure set forth a computer-implemented method for identifying an object in an environment, the method comprising: receiving first sensor data associated with an environment via at least one sensor; storing the first sensor data in association with a first scene in a memory; and in response to receiving a user request for information associated with the environment: selecting the first scene based on the user request; accessing the first sensor data associated with the first scene via the memory; and analyzing the first sensor data to identify a first object included in the first scene; and causing information associated with the first object to be output via at least one output device.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to computer information systems and, more particularly, to automatic reference lookup in audio-visual (A / V) scenes. Background Art

[0002] The establishment of the Internet has made information on virtually any subject readily available to anyone with an Internet connection. Furthermore, the widespread use of smartphones, wearable devices, and other devices provides many users with Internet connections at virtually any time. Since access to knowledge sources via the Internet is nearly ubiquitous, users can easily search for information on a variety of topics by initiating a text-based keyword or key phrase search. In addition, Internet-connected devices such as Microsoft Cortana can easily TM , Apple Siri TM and Amazon Alexa TM Intelligent personal assistants (IPAs) such as , allow users to initiate information searches on specific topics without looking at a display screen, or to manually enter search parameters as text. Alternatively, users can verbally retrieve information from the Internet by speaking questions or other natural language phrases to the IPA.

[0003] Generally speaking, in order to search for a topic, the user initiating the search must provide at least some specific information about the topic. However, in many cases, the user may not be able to initiate a search because the user may not be able to adequately identify the topic, describe key information related to the specific topic, and / or provide relevant search parameters for the topic to be searched. For example, when a person sees a transient object in an environment, the user may not be able to recognize the object or accurately describe the object. When a user is unable to initiate an Internet search for an object with sufficient accuracy, a device connected to the Internet will not be able to provide search results that include applicable information about the object. Therefore, the user may not be able to effectively utilize the Internet's vast pool of information to determine or otherwise examine objects within a given scene.

[0004] In view of the foregoing, more effective techniques for identifying various aspects of an environment would be useful. Summary of the invention

[0005] An embodiment of the present disclosure describes a computer-implemented method for identifying an object in an environment, the method comprising: receiving first sensor data associated with the environment via at least one sensor; storing the first sensor data in association with a first scene in a memory; and in response to receiving a user request for information associated with the environment: selecting the first scene based on the user request; accessing the first sensor data associated with the first scene via the memory; and analyzing the first sensor data to identify a first object included in the first scene; and causing the information associated with the first object to be output via at least one output device.

[0006] Additional embodiments also provide methods and computer-readable storage media for implementing aspects of the methods set forth above.

[0007] At least one technical advantage of the disclosed technology is that objects within a given scene can be identified even when the objects no longer exist within the scene. Another advantage is that information about objects identified in a scene can be obtained from one or more data sources and provided to a user without the user having to provide detailed information about the objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to be able to understand the above-mentioned features of various embodiments in detail, the inventive concept briefly outlined above can be more particularly described with reference to various embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the drawings only show typical embodiments of the inventive concept and are therefore in no way to be considered limiting of the scope, and there are other equally effective embodiments.

[0009] Figure 1 A block diagram of a reference identification system configured to implement one or more aspects of the present disclosure is shown.

[0010] Figure 2 The present invention shows a method for using various embodiments of the present invention. Figure 1 A reference identification system is a technology for storing sensor data associated with one or more scenes.

[0011] Figure 3 The present invention shows a method for using various embodiments of the present invention. Figure 1 A reference identification system is provided to analyze one or more scenes to identify candidate objects within the scenes.

[0012] Figure 4 The present invention shows a method for using various embodiments of the present invention. Figure 1 A technique for obtaining and providing information associated with a selected object within a scene by a reference identification system.

[0013] Figure 5 is a flow chart of method steps for providing information associated with an object included in a scene according to various embodiments of the present disclosure. DETAILED DESCRIPTION

[0014] In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to one skilled in the art that the inventive concept may be practiced without one or more of these specific details.

[0015] Figure 1 A block diagram of a reference identification system configured to implement one or more aspects of the present disclosure is shown. The reference identification system 100 includes a computing device 110, one or more sensors 120, one or more knowledge agents 130, and one or more input / output (I / O) devices 114. The computing device 110 includes a processing unit 112 and a memory 116. The memory 116 stores a reference identification application 162 and a scenario storage area 164.

[0016] In operation, the processing unit 112 receives sensor data from the sensor 120. The processing unit 112 executes the reference identification application 162 to store a portion of the received sensor data as a scene in the scene storage area 164. After storing the scene, the processing unit 112 detects a signal received from a user via the I / O device 114 as a trigger. In response to detecting the trigger, the reference identification application 162 analyzes one or more scenes stored in the scene storage area 164 to identify candidate objects present within the one or more scenes. After identifying the one or more candidate objects within the scene, the reference identification application 162 selects one of the candidate objects as the selected object.

[0017] In various embodiments, the reference identification application 162 may select a particular candidate object from a plurality of candidate objects, wherein the reference identification application 162 determines the selection based on a relevance score associated with each candidate object. In such a case, the reference identification application 162 may calculate a relevance score for a given candidate object based on one or more relevance factors associated with the candidate object. For example, in some embodiments, the reference identification application 162 may calculate quantitative values ​​of relevance factors such as location, size, amount of movement, historical data, and / or other factors associated with the probability that a given candidate object is associated with a user. In some embodiments, the reference identification application 162 may select a candidate object from a plurality of candidate objects by comparing the relevance scores associated with each candidate object, wherein the reference identification application 162 selects the candidate object associated with the highest relevance score.

[0018] After selecting a candidate object, the reference identification application 162 generates a query associated with the selected object. In various embodiments, the reference identification application 162 can transmit the generated query to one or more knowledge agents 130 to obtain information associated with the selected object. In response, the reference identification application 162 receives a response from the one or more knowledge agents 130, the response including information associated with the selected object. The reference identification application 162 then presents at least a portion of the information received from the one or more knowledge agents 130 to the user via one or more I / O devices 114.

[0019] In some embodiments, the reference identification application 162 can receive feedback associated with the selected object and / or associated with one or more candidate objects for the user. For example, the reference identification application 162 will receive feedback from the user, and the feedback specifies that the reference identification application 162 obtains more information about the selected object. In such a case, the reference identification application 162 can retrieve supplementary information from one or more knowledge agents 130, and the supplementary information can be presented to the user. In another example, the reference identification application 162 will receive feedback to select different candidate objects from multiple candidate objects. In such a case, the reference identification application 162 can select different candidate objects from the remaining multiple candidate objects as the newly selected object, and can retrieve additional information about the newly selected candidate object by querying one or more knowledge agents 130.

[0020] The sensor 120 includes one or more devices that detect the position of an object in the environment by performing measurements and / or collecting data. In some embodiments, one or more sensors 120 may be coupled to and / or included in the computing device 110. In some embodiments, the computing device 110 may receive sensor data via one or more sensors 120, wherein the sensor data reflects the position and / or orientation of one or more objects in the environment. One or more positions and / or one or more orientations of one or more objects may be derived from the absolute position of one or more sensors 120, and / or may be derived from the position of the object relative to one or more sensors 120. The processing unit 112 executes the reference identification application 162 to store the sensor data in the scene storage area 164. In various embodiments, the reference identification application 162 may store data associated with the sensor data obtained at a specified time in association with a specific scene. For example, the reference identification application 162 may store the sensor data collected from one or more sensors 120 within a specified time range as part of a data set including a specific scene. In various embodiments, the reference identification application 162 may also store additional data in a data set for a specific scene. For example, the reference identification application 162 may add metadata associated with a specified time range as part of a data set that includes a particular scene.

[0021] In various embodiments, one or more sensors 120 may include optical sensors, such as RGB cameras, time-of-flight sensors, infrared (IR) cameras, depth cameras, and / or quick response (QR) code tracking systems. In some embodiments, one or more sensors 120 may include position sensors, such as accelerometers and / or inertial measurement units (IMUs). IMUs may be devices such as three-axis accelerometers, gyro sensors, and / or magnetometers. Additionally, in some embodiments, one or more sensors 120 may include audio sensors, wireless sensors (including radio frequency (RF) sensors ( example like , sonar, and radar)), ultrasound-based sensors, capacitive sensors, laser-based sensors, and / or wireless communication protocols (including Bluetooth, Bluetooth Low Energy (BLE), wireless LAN (WiFi), cellular protocols, and / or near field communication (NFC)).

[0022] As described above, computing device 110 may include processing unit 112 and memory 116. Computing device 110 may be a device including one or more processing units 112, such as a system on a chip (SoC), or a mobile computing device, such as a tablet computer, a mobile phone, a media player, etc. In general, computing device 110 may be configured to coordinate the overall operation of reference identification system 100. Embodiments disclosed herein encompass any technically feasible system configured to implement the functionality of reference identification system 100 via computing device 110.

[0023] The processing unit 112 may include a central processing unit (CPU), a digital signal processing unit (DSP), a microprocessor, an application specific integrated circuit (ASIC), a neural processing unit (NPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), etc. In some embodiments, the processing unit 112 may be configured to execute a reference identification application 162 to analyze sensor data acquired by one or more sensors 120, acquire information about one or more candidate objects detected in a scene, and present the acquired information to a user.

[0024] In various embodiments, the processing unit 112 may execute the reference identification application 162 to receive and / or analyze one or more signals generated by the I / O device 114. The reference identification application 162 then determines whether the received signal includes a trigger indicating that the reference identification application 162 should obtain more information about the object in the scene. When the reference identification application 162 determines that a trigger is detected, the reference identification application 162 identifies one or more candidate objects within one or more scenes stored in the scene storage area 164. In various embodiments, the reference identification application 162 may identify one or more candidate objects based on sensor data acquired by one or more sensors 120 and stored in association with one or more scenes. For one or more of the candidate objects identified in the scene stored in the scene storage area 164, the reference identification application 162 calculates a relevance score associated with the candidate object. In some embodiments, the reference identification application 162 may select one of the candidate objects ("selected object") based on the calculated relevance score. In some embodiments, the reference identification application 162 may calculate the relevance score of the candidate object using one or more relevance parameters, which reflect the probability that the candidate object is relevant to the user.

[0025] In various embodiments, once a particular object ( For example, based on the relevance of the object to the user), the reference identification application 162 can query the selected object to one or more knowledge agents 130. In some embodiments, the reference identification application 162 can generate one or more queries associated with the selected object, and can transmit the one or more queries to one or more knowledge agents 130. The processing unit 112 receives responses from one or more knowledge agents 130, wherein one or more knowledge agents 130 generate responses based on the received queries. The reference identification application 162 analyzes the responses received from one or more knowledge agents. In some embodiments, the reference identification application 162 can filter the responses to generate a set of related information for the selected object. Then, the processing unit 112 transmits at least a portion of the set of related information to the user via one or more I / O devices 114. In some embodiments, one or more I / O devices 114 can present the related information via one or more user interfaces. For example, the audio I / O device 114 will emit a spoken audio signal that verbally presents the related information. In another example, the visual I / O device 114 will visually present the related information as text and / or graphics.

[0026] The memory 116 may include a memory module or a collection of memory modules. The reference identification application 162 in the memory 116 may be executed by the processing unit 112 to implement the overall functionality of the computing device 110, thereby coordinating the operation of the reference identification system 100 as a whole.

[0027] The scene storage area 164 may store values ​​and other data retrieved by the processing unit 112 to coordinate the operation of the reference identification system 100. During operation, the processing unit 112 may be configured to store values ​​in the scene storage area 164 and / or retrieve values ​​stored in the scene storage area 164. For example, the scene storage area 164 may store sensor data, metadata, previously stored scenes from external sources, digital signal processing algorithms, transducer parameter data, etc. The scene storage area 164 includes a scene storage area 164 that is retrieved by the user before the reference identification system 100 is queried ( For example , immediately before the user query) for a specific length of time in the past. In various embodiments, the scene storage area 164 may include a buffer that stores a constant, predetermined number of scenes. In such a case, when the user requests information associated with the environment, the reference identification application 162 can retrieve one or more scenes included in the buffer.

[0028] I / O devices 114 include at least one device capable of receiving input (such as a keyboard, mouse, touch-sensitive screen, microphone, etc.) and a device capable of providing output (such as a display screen, speakers, etc.). One or more of I / O devices 114 may be incorporated into computing device 110 or may be external to computing device 110.

[0029] The knowledge agent 130 includes at least one application and / or database that generates a response to the received query. Based on the content of the query, the generated response includes additional information for a given object. In operation, one or more knowledge agents 130 receive a query from a computing device 110 and generate a response to the query. In various embodiments, the response generated by the knowledge agent 130 may include additional information associated with the selected object, wherein the additional information is retrieved based on the content included in the received query. For example, a simple query associated with a bird will cause the knowledge agent 130 to generate a response including general information about the bird. A more detailed query associated with a woodpecker will cause the knowledge agent 130 to include more detailed information associated with the woodpecker, including, among other things, classification data and / or location data. In various embodiments, one or more knowledge agents may retrieve additional information from one or more internal data sources and / or external data sources included in the computing device 110. When retrieving information from an external data source, one or more knowledge agents 130 may generate additional queries to obtain information from the external data source.

[0030] Figure 2 The present invention shows a method for using various embodiments of the present invention. Figure 1 The present invention provides a technology for storing sensor data associated with one or more scenes in a reference identification system. As shown in the figure, the reference identification system 200 includes candidate objects 210, 212, 214, sensors 220, a reference identification application 162, a scene storage area 164, and an external data source 230. The sensor 220 includes an audio sensor 222, an imaging sensor 224, and / or a motion sensor 226.

[0031] During operation, one or more sensors 220 transmit sensor data 228 to the reference identification application 162. In various embodiments, the sensor data 228 may include data related to the position and / or orientation of the candidate objects 210, 212, 214 in the environment within a specified time period. 0 -t 1During the first specified time period, the candidate objects 210, 212, 214 have initial positions; one or more of the candidate objects 210, 214 change positions during the first specified time period. In this example, one or more sensors 220 can acquire sensor data 228 during the first specified time period. Accordingly, the motion sensor 226 acquires motion data, the audio sensor 222 acquires audio data, and / or the imaging sensor 224 acquires imaging data. Therefore, the sensor data 228 transmitted by the sensor 220 to the reference identification application 162 includes the sensor data 228 during the first specified time period t 0 -t 1 The position and / or orientation data associated with the candidate objects 210, 212, 214 is provided within.

[0032] The reference identification application 162 receives the sensor data 228 from the sensor 220 and generates a scene 262 including the sensor data 228. The reference identification application 162 sends the scene 262 to the scene storage area 164, where the scene is stored as scene 262-1. In various embodiments, the reference identification application 162 can receive the sensor data 228 associated with a specified time period. The reference identification application 162 can store the sensor data 228 as part of a data set associated with the scene 262, where the scene 262 corresponds to the specified time period. For example, the reference identification application 162 initially generates a scene 262 in a first specified time period t 0 -t 1 The reference identification application 162 may then generate a first scene s comprising the first sensor data set. 1 .

[0033] In various embodiments, the reference identification application 162 may include metadata 234, 264 associated with the specified time period into the scene 262. In some embodiments, the computing device 110 may generate the metadata 264, or may retrieve the metadata 234 from the external data source 230. For example, the reference identification application 162 may obtain the metadata 234, 264 ( For example , GPS data, data from other applications, data from external data sources, etc.). Then, the reference identification application 162 can 1 The scene 262 includes metadata acquired during the first specified time period and the received sensor data 228. Examples of metadata 234, 264 that may be associated with the scene 262 include scene timestamps, keywords, location data, lighting data, text descriptions, and / or instructions or parameters for image and / or audio processing.

[0034] In various embodiments, the reference identification application 162 may cause a stored scene 232 from an external data source 230 to be stored in the scene storage area 164. For example, the reference identification application 162 may cause a portion of content ( For example , movies, television shows, songs, etc.) are associated with a first specified time period. In some embodiments, the reference identification application 162 may receive a portion of the content as a stored scene 232; in such a case, the reference identification application 162 may generate a scene 262 to include the stored scene 232 as an included data set. In some embodiments, the reference identification application 162 may cause the stored scene 232 to be stored directly in the scene storage area 164 as a scene 262-2. In such a case, the reference identification application 162 may add metadata 264 for the specified time period to the scene 262-2.

[0035] In various embodiments, the external data source 230 can be a database, a storage unit, and / or a computing device. The external data source 230 stores data that can be analyzed by the reference identification application 162 when the reference identification application 162 is analyzing a scene. In some embodiments, the reference identification application 162 can generate a query to retrieve data from the external data source 230, such as a stored scene 232, metadata 234, and / or other data. For example, the external data source 230 can be an external security system that records multiple video frames. The reference identification application 162 can query the external data source 230 to retrieve one or more video frames stored in the external data source 230. After retrieving one or more frames, the reference identification application 162 can associate the one or more retrieved frames with one or more scenes 261-1, 261-2 stored in the scene storage area 164.

[0036] Figure 3 The present invention shows a method for using various embodiments of the present invention. Figure 1 A reference identification system is used to analyze one or more scenes to identify candidate objects within the scenes. As shown in the figure, a portion of the reference identification application 300 includes a scene timeline 310 and multiple scenes 340 ( For example , 340-1 to 340-6). Each of the scenes 340-1 to 340-6 is associated with a scene timestamp ( For example The reference identification application 162 analyzes one or more of the scenes 340 to identify candidate objects 352 , 354 , 356 .

[0037] In operation, the reference identification application 162 receives a user request and responds by selecting one or more scenes 340 stored in the scene storage area 164 for analysis. In various embodiments, the user request can be a triggering event detected by the reference identification application 162. In some embodiments, the triggering event can be a verbal input. An example of a triggering event is a user speaking an utterance (such as "What's that?") that includes a keyword or key phrase recognized by the reference identification application 162. In some embodiments, the reference identification application 162 can recognize other verbal utterances or sounds generated by the user as triggering events.

[0038] In some embodiments, the triggering event can be a different physical input, such as a physical input to a touch-based mechanism. Examples of physical input to a touch-based mechanism include pressing a button, actuating a switch, or making a gesture on a touch-sensitive screen ( For example , tap, swipe, double-click, etc.). Other physical inputs include user gestures, such as facial gestures ( For example , raising eyebrows, frowning forehead, squinting, blinking, sudden changes in expression, etc.), hand gestures ( For example , pointing hands, etc.), head gestures ( For example , nodding or shaking the head), and / or body gestures ( For example , putting hand in front of mouth, etc.). In some embodiments, the triggering event can be a detected physiological change or a detected physiological event associated with or experienced by the user, such as a sudden change in breathing rate, heart rate, pupil dilation, and / or brain activity.

[0039] In response to detecting the user request, the reference identification application 162 selects one or more of the scenes 340 to analyze the data associated with the selection. In various embodiments, each of the scenes 340 can be stored in the scene storage area 164. In some embodiments, the reference identification application 162 can select one or more scenes 340 by generating a query and retrieving the scenes 340 from the external data source 230.

[0040] In various embodiments, the reference identification application 162 can determine the time at which the user request was initiated by the triggering event, and can respond by retrieving data associated with the determined time. For example, the reference identification application 162 can determine that the triggering event was at time T 0 Based on the time when the user request was initiated, the reference identification application 162 will make a 0 The reference identification application 162 may make a selection 336 to retrieve the 0Associated data, including scene 340 - 5 . In some embodiments, scene 340 - 5 and / or other data (including sensor data 228 and / or metadata 264 ) is associated with a matching timestamp 326 .

[0041] In various embodiments, the reference identification application 162 may also make a selection that has a predefined relationship to the determined time. In some embodiments, the reference identification application 162 may select to 0 For example, the reference identification application 162 may have been configured to select data with a timestamp T i-2 322 associated scene, the timestamp is at the determined time T 0 In such an example, reference identification application 162 will make selection 332 and retrieve scene 340 - 1 and / or other data associated with timestamp 322 .

[0042] In various embodiments, the reference identification application 162 may perform other methods to select one or more scenes 340 based on a user request. For example, a user request may indicate a specific time ( For example , “Who was that person in front of me at 5pm?”). In such a case, the reference identification application 162 may set the designated time to the determined time T 0 , and one or more timestamps 322 , 324 , 326 may be selected based on one or more relationships to a specified time.

[0043] In various embodiments, the reference identification application 162 may maintain a buffer for a constant, predefined number of scenes 340 ( For example , a buffer in the scene storage area 164). In such a case, when the user requests information about an object in the environment, the reference identification application 162 can retrieve and analyze one or more scenes 340 included in the buffer. In some embodiments, the reference identification application 162 can respond to the user request by making two selections of the scenes in the buffer: a first selection 336 associated with the time of the triggering event; and a second selection 332 associated with a predefined time period before the first selection 336. In some embodiments, the time period between the second selection 332 and the first selection 336 can be a constant specified value. The value of the time period between the selections can be manually set by the user. In some embodiments, the time period between the selections can be included in the user request, such as when the request specifies a specific time to be analyzed ( For example , "What was this object 30 seconds ago?").

[0044] For a given scene 340, 350, the reference identification application 162 analyzes data associated with the scene 340, 350 to identify one or more candidate objects within the scene 350. For example, the reference identification application 162 may select the scenes 340-1, 340-5 to generate the synthetic scene 350. The reference identification application 162 may then analyze the sensor data 232, metadata 264, and other data associated with the synthetic scene 350 to segment portions of the synthetic scene 350 and identify candidate objects 352, 354, 356 included in the synthetic scene 350.

[0045] In various embodiments, the reference identification application 162 may perform various audio processing, video processing, imaging processing, and / or other processing techniques on the sensor data 232 associated with the synthetic scene 350 in order to identify one or more candidate objects 352, 354, 356 within the synthetic scene 350. Similarly, in some embodiments, the reference identification application 162 may perform various processing techniques on the metadata 264 associated with the synthetic scene 350. For example, the reference identification application 162 may retrieve information associated with the synthetic scene 350. i-2 The metadata 264 associated with the timestamp 322 and the 0 The metadata 264 associated with the timestamp 326 ; the reference identification application 162 may then analyze the metadata 264 in order to identify candidate objects 352 , 354 , 356 within the synthetic scene 350 .

[0046] In various embodiments, the reference identification application 162 may implement statistical techniques, machine learning (ML) techniques, and / or artificial intelligence (AI) techniques to analyze data associated with the synthetic scene 350 in order to identify candidate objects 352, 354, 356 within the synthetic scene 350. In various embodiments, the reference identification application 162 may implement at least one ML technique to identify one or more candidate objects 352, 354, 356 based on the sensor data 228, metadata 264, and / or other data associated with the synthetic scene 350. Suitable ML and / or AI systems employed in the reference identification application 162 may include, for example, nearest neighbor classifier procedures, Markov chains, deep learning methods, and / or any other technically feasible methods.

[0047] In various embodiments, the reference identification application 162 can analyze the plurality of scenes 340 to identify one or more candidate objects 352, 354, 356 present in at least one scene. For example, the reference identification application 162 can make selections 332, 336 for analysis, where the selections 332, 336 include scenes 340-1, 340-5. The reference identification application 162 can analyze each selection 332, 336, respectively, to identify one or more candidate objects 352, 354, 356.

[0048] In some embodiments, the reference identification application 162 can compare the data in the selections 332, 336 to identify the candidate objects 352, 354, 356 and / or calculate a relevance score for each of the identified candidate objects 352, 354, 356. For example, the reference identification application 162 can analyze the data associated with each of the selections 332, 336 by identifying the differences between the scene 340-1 and the scene 340-5. For example, when comparing the selections 332, 336, the reference identification application 162 can determine that the candidate object 354 does not exist in the newer scene 340-5, while the candidate object 354 previously existed in the older scene 340-1. In some embodiments, the reference identification application 162 can calculate a relevance score associated with the candidate object 354 based on determining that the candidate object 354 no longer exists in the newer scene 340-5. For example, the reference identification application 162 can calculate a relevance score that includes a relevance factor ( For example , “missing object” relevance factor), which reflects a high likelihood that the user is requesting information for candidate object 354.

[0049] Figure 4 The present invention shows a method for using various embodiments of the present invention. Figure 1 The reference identification system 400 is a technology for obtaining and providing information associated with a selected object in a scene. As shown in the figure, the reference identification system 400 includes a reference identification application 162, knowledge agents 130-1 to 130-3, and an I / O device 114.

[0050] In operation, the reference identification application 162 selects one of the candidate objects 352, 354, 356 (referred to herein as the selected object 402) and generates one or more queries 404 based on the selected object 402. The reference identification application 162 selects one or more knowledge agents 130 ( For example, 130-1 to 130-3) to query and send one or more queries 404 to the selected knowledge agent 130. The reference identification application 162 receives one or more responses 410 from the selected knowledge agent 130, wherein the one or more responses 410 include additional information about the selected object 402. The reference identification application 162 analyzes the responses 410 and generates a set of information about the selected object 402 (referred to herein as selected object information 412). The reference identification application 162 provides the selected object information 412 via one or more I / O devices 114.

[0051] In various embodiments, the reference identification application 162 may select a candidate object 354 from one or more candidate objects 352, 354, 356 included in the scene 350 based on the calculated relevance score. The reference identification application 162 may calculate the relevance score for each candidate object 352, 354, 356 included in the synthetic scene 350. When calculating the relevance score (R) for the candidate object 352, the reference identification application 162 calculates one or more relevance factors (rf) associated with the candidate object 352, wherein the relevance score (R) is calculated using the one or more calculated relevance factors. 1 =f{rf 1 ,rf 2 ,…rf n}). The reference identification application 162 analyzes the attributes of the candidate objects 352, 354, 356 and calculates a relevance factor associated with one or more of the analyzed attributes. The reference identification application 162 uses the one or more relevance factors to calculate a relevance score for each of the candidate objects 352, 354, 356. In some embodiments, the reference identification application 162 may rank the candidate objects 352, 354, 356 based on the respective relevance scores. In some embodiments, the reference identification application 162 may select the candidate object 354 based on the relative ranking of each candidate object 352, 354, 356.

[0052] In some embodiments, the attributes of the candidate objects 352 may include objective attributes of the candidate objects 352 such as color, shape, classification, location, etc. In some embodiments, the attributes of the candidate objects 352 may include relative attributes, such as which candidate object 352, 354, 356 is the loudest, largest, brightest, fastest, smells the worst, is the most difficult, is the most dangerous, is the most repeated, is mentioned the most in social media feeds, etc. In some embodiments, the reference identification application 162 may assign quantitative values ​​to one or more attributes of the candidate objects 352 when calculating the relevance factor.

[0053] For example, the reference identification application 162 will segment the synthetic scene 350. The synthetic scene 350 includes three candidate objects: a first candidate object 352, a second candidate object 354, and a third candidate object 356. The reference identification application 162 can analyze the properties of the three candidate objects separately and assign a value to each of the determined properties. In this example, the reference identification application 162 will be configured to assign a relevance factor value ( For example , brightest object = 0.8, largest object = 0.4, moving object = 0.2, etc.). The reference identification application 162 may determine that the first candidate object 352 is the largest object, the dimmest object, and the fastest object. Similarly, the reference identification application 162 may determine that the second object 354 is the brightest object and the slowest object ( For example , stationary). Finally, the reference identification application 162 can determine that the third candidate object 356 is the brightest object and the most repeated object. The reference identification application 162 can calculate a separate relevance factor based on each of the determined attributes, and can calculate a relevance score by applying a weight value to each of the calculated relevance factors and adding the weighted relevance factors. For example, the reference identification application 162 can calculate a relevance score of 0.6 for the first candidate object 352, a relevance score of 0.8 for the second candidate object 354, and a relevance score of 0.3 for the third candidate object 356, respectively. Based on the calculated relevance scores, the reference identification application 162 can select the second candidate object 354 as the selected object 402.

[0054] In some embodiments, in addition to the relevance factors based on the attributes of the candidate objects 352, 354, 356, the reference identification application 162 also calculates other relevance factors associated with the candidate objects 352, 354, 356. For example, other relevance factors include the number of queries for the object that were previously generated, whether the object does not exist in the most recent scene 340-5, whether the object was previously tagged with a keyword, biometric data related to the user ( For example , heart rate, brain activity, skin conductance, blood oxygenation, etc.), qualitative factors, etc. For example, the reference identification application 162 can be configured to associate a specific heart rate range with a specific correlation factor ( For example , when the user's heart rate is between 150 and 180 beats per minute, the reference identification application 162 applies a higher weight value to the relevance factor of the loudest object.) In some embodiments, the reference identification application 162 can assign quantitative values ​​to qualitative factors when calculating the relevance factor.

[0055] In another example, the reference identification application 162 will identify four candidate objects from the selection 332. The selection 332 includes the stored scene 232 and metadata 234 retrieved from the external data source 230, and the sensor data 228. The stored scene 232 includes four candidate objects, including character 1, character 2, character 3, and background object 1. The reference identification application 162 can analyze the metadata 234 associated with the stored scene 232 and determine that character 1 has the most previous queries for the stored scene 232. Similarly, the reference identification application 162 can determine that character 2 is the candidate object most mentioned in the social media information feed. Similarly, the reference identification application 162 can determine that character 3 exists in the stored scene 232, but does not exist in the previous scene. Finally, the reference identification application 162 can determine that background object 1 is the largest object and the brightest object. In this example, the reference identification application 162 will be configured to assign values ​​to each of the relevance factors. The reference identification application 162 may also be configured to analyze biometric data associated with the user to determine how to weight the relevance factors. The reference identification application 162 may calculate separate relevance factors and then calculate relevance scores based on each of the relevance factors. In this example, the reference identification application 162 may calculate relevance scores of 0.9 for character 1, 0.7 for character 2, 0.3 for character 3, and 0.2 for background object 1, respectively. Based on the calculated relevance scores, the reference identification application 162 may select character 1 as the selected object 402 and generate a query 404 to obtain additional information about character 1.

[0056] In another example, the reference identification application 162 may generate a synthetic scene 350 based on the multiple selections 332, 336. When analyzing the synthetic scene 350, the reference identification application 162 may analyze the scenes 340-1 and 340-5, respectively. In such a case, the reference identification application 162 will determine that the candidate object 354 exists in the scene 340-1, but does not exist in the scene 340-5, and apply the transient object relevance factor to the candidate object 354. In this example, the reference identification application 162 may be configured to assign a high value to the transient object relevance factor (e.g., transient object relevance factor = 0.95). Therefore, the reference identification application 162 may calculate a high relevance score for the candidate object 354 based on the transient object relevance factor.

[0057] When the candidate object 354 is selected as the selected object 402, the reference identification application 162 generates one or more queries 404 based on the selected object 402. The reference identification application 162 generates the queries 404 associated with the selected object 402, wherein the responses 410 subsequently received by the reference identification application 162 provide information associated with the selected object 402.

[0058] In various embodiments, the reference identification application 162 selects one or more knowledge agents 130 ( For example , 130-1 to 130-3), the knowledge agent will receive the generated query 404. In some embodiments, the reference identification application 162 selects the knowledge agent 1 130-1 based on the content of the query. For example, the reference identification application 162 will analyze the content of the query 404 to determine whether the knowledge agent 1 130-1 can respond with applicable information. When the reference identification application 162 determines that the knowledge agent 1 130-1 can provide applicable information to the query 404, the reference identification application 162 sends the query 404 to the knowledge agent 1 130-1. In some embodiments, the reference identification application 162 can also select additional knowledge agents 130, such as knowledge agent 2 130-2 and / or knowledge agent 3 130-3, and send the query 404 to each of the knowledge agents 130-2, 130-3. One or more knowledge agents 130 receive the query 404 and respond to the included content by retrieving additional information associated with the content included in the query 404. In some embodiments, the knowledge broker 130 may be a database storing information. In such a case, the query 404 may cause a data set to be extracted from the database. In some embodiments, the knowledge broker 130 may be an application programming interface (API) that may generate additional queries to retrieve information from an external data source. After obtaining additional information associated with the content included in the query 404, the knowledge broker 130 generates a response 410 and sends it to the reference identification application 162, wherein the response includes the retrieved information associated with the selected object 402.

[0059] In various embodiments, the reference identification application 162 can determine whether the user wants additional information about the selected object 402. In such a case, the reference identification application 162 can respond to the user's request for additional information by sending one or more additional queries 404 associated with the selected object 402. In some embodiments, the reference identification application 162 can generate a continuous query 404 based on one or more responses 410 received from the knowledge agent 130. For example, the reference identification application 162 will generate a first query 404 and send it to the knowledge agent 1 130-1. The reference identification application 162 will receive a first response, the content of which includes a definition of the selected object 402 ( For example , "the object is a person"). After receiving the first response, the reference identification application 162 can then generate a second query 404 that causes the knowledge agent 2130-2 to provide more detailed information about the selected object 402 ( For example Similarly, upon receiving the second response 410, the reference identification application 162 may generate a third query 404 that causes the knowledge agent 3 130-3 to provide additional information about the selected object 402 ( For example , “Ann lives in San Jose, California”). In some embodiments, the reference identification application 162 can generate additional queries 404 before receiving feedback from the user.

[0060] The reference identification application 162 analyzes the information included in the response 410 to generate an information set about the selected object 402. The reference identification application 162 transmits a portion of the information set to the user as the selected object information 412 via one or more I / O devices 114. In various embodiments, one or more I / O devices 114 can provide the selected object information 412 to the user via any suitable presentation mode. For example, the user selected setting of the reference identification system 100 will indicate that the preferred presentation mode of the selected object information 412 is an auditory mode. In another example, the user selected setting of the reference identification system 100 will indicate that the preferred presentation mode of the selected object information 412 is a visual mode, including displaying text and / or images on a display included in the mobile device. In addition, the selected object information 412 can be divided into multiple separate parts and can be provided via multiple I / O devices 114. For example, the reference identification application 162 can cause a speaker to transmit a first part of the selected object information 412, and the reference identification application 162 can cause a display to transmit a second part of the selected object information 412.

[0061] In various embodiments, the reference identification application 162 can determine whether the user wants to select a different candidate object 352, 356 included in the synthetic scene 350. In such a case, the reference identification application 162 can select a different selected object 402 by selecting one of the remaining candidate objects 352 in the group of candidate objects 352, 356. After selecting the different selected object 402, the reference identification application 162 can generate a query 404 and send the query to the applicable knowledge agent 130 to obtain additional information related to the different selected object 402.

[0062] Figure 5 is a flow chart of method steps for providing information associated with an object included in a scene according to various embodiments of the present disclosure. Figures 1 to 4 The method steps are described herein with reference to a system, but those skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the various embodiments.

[0063] As shown, the method 500 begins at step 501, where the reference identification application 162 stores sensor data received from the sensor 220. In various embodiments, the reference identification application 162 can receive the sensor data 228 from the sensor 220. The reference identification application 162 generates a scene 262 associated with the sensor data 228 and associates the scene with a timestamp 322. The reference identification application 162 stores the scene 262 in the scene storage area 164. In various embodiments, the reference identification application 162 can also associate metadata 264 with the scene 262 and store the metadata 264 in the scene storage area 164.

[0064] At step 505, the reference identification application 162 determines whether a user request is received. In various embodiments, the user request may be a trigger event that the reference identification application 162 can detect. The trigger event may be a verbal input, a physical input on a touch-based mechanism, a body gesture, etc. When the reference identification application 162 detects a trigger event, the reference identification application 162 determines that a user request is received and proceeds to step 507; otherwise, the reference identification application 162 determines that a user request is not received and returns to step 501.

[0065] At step 507, the reference identification application 162 loads one or more scenes 340 ( example like, 340-1 to 340-6). In response to detecting the user request, the reference identification application 162 loads one or more scenes 340 stored in the scene storage area 164 and makes one or more selections 332, 336 to analyze the data associated with the selections 332, 336. In some embodiments, the reference identification application 162 can load one or more scenes 340 by generating a query and retrieving one or more stored scenes 232 from the external data source 230.

[0066] In various embodiments, the reference identification application 162 may maintain a buffer for a constant, predefined number of scenes 340 in the scene storage area 164. In such a case, when a user requests information about an object in the environment, the reference identification application 162 may retrieve one or more scenes 340 included in the buffer. In some embodiments, the reference identification application 162 may respond to the user request by making two selections of scenes in the buffer: a first selection 336 associated with the time of the triggering event; and a second selection 332 of a predefined time period prior to the first selection 336.

[0067] At step 509, the reference identification application 162 scans the selected scene to identify one or more candidate objects ( For example , candidate objects 352, 354, 356). For a given selection 332, the reference identification application 162 analyzes data associated with the selection 332 and / or the scene 340-1 (including the sensor data 228 and metadata 264 associated with the scene 340-1) to identify one or more candidate objects 352, 354, 356. For example, the reference identification application 162 may analyze the sensor data 232, metadata 264, and other data associated with the scene 340-1 included in the selection 332 to segment portions of the scene 340-1 and identify the candidate objects 352, 354, 356.

[0068] At step 511, the reference identification application 162 selects one of the candidate objects 352, 354, 356 included in the scene 340-1 (the selected object 402). In various embodiments, the reference identification application 162 can calculate a relevance score for each candidate object 352, 354, 356 included in the scene 340-1, and can select the candidate object 354 based on the calculated relevance score. When calculating the relevance score, the reference identification application 162 calculates one or more relevance factors associated with the candidate object 352, wherein the relevance score is calculated using one or more calculated relevance factors. In some embodiments, the reference identification application 162 can rank the candidate objects 352, 354, 356 based on the calculated relevance score, and can select the highest ranked one of the candidate objects 354.

[0069] At step 513, the reference identification application 162 selects a knowledge agent 130 based on the selected object 402. In various embodiments, the reference identification application 162 selects one or more knowledge agents 130 that will receive the query 404 related to the selected object 402. In some embodiments, the reference identification application 162 can analyze the content of the selected object 402 and / or the query 404 to determine whether a particular knowledge agent 130 ( For example , whether knowledge agent 1 130-1) is able to respond with applicable information.

[0070] At step 515, in operation, the reference identification application 162 generates a query 404 based on the selected object 402. The reference identification application 162 sends the query 404 to the knowledge broker 130, and receives a response 410. The response 410 includes additional information about the selected object 402. When generating the response 410, the knowledge broker 130 receives the query 404, analyzes the content included in the query 404, and retrieves additional information based on the content included in the query 404. The knowledge broker 130 generates and sends the response 410, wherein the response 410 includes information associated with the selected object 402.

[0071] At step 517, the reference identification application 162 provides the selected object information 412 to the user. In various embodiments, the reference identification application 162 can analyze the information included in the response 410 to generate an information set about the selected object 402. In some embodiments, the reference identification application 162 can analyze the information set and generate the selected object information 412 as part of the information set. The reference identification application provides the selected object information 412 to the user 101 via one or more I / O devices 114.

[0072] At step 519, the reference identification application 162 determines whether the selected object 402 is relevant to the user. In various embodiments, the reference identification application 162 can receive a response from the user indicating whether the selected object 402 is an object that caused the user to initiate a user request. If the reference identification application 162 determines that the selected object 402 is relevant to the user and is therefore an object that caused the user to initiate a user request, the reference identification application 162 can then proceed to step 521; otherwise, when the reference identification application 162 determines that the selected object 402 is not relevant to the user, the reference identification application 162 can then proceed to step 523.

[0073] At step 521, the reference identification application 162 determines whether the user wants more information about the selected object 402. In various embodiments, the reference identification application 162 can determine whether the user wants additional information about the selected object 402. For example, the reference identification application 162 will receive user input indicating a request for more information. In such a case, the reference identification application 162 can respond to the user request for additional information by returning to step 513, where the reference identification application 162 selects the knowledge agent 130 ( For example , knowledge agent 3 130-3), generates an additional query 404 and sends the additional query 404 to the selected knowledge agent 3 130-3.

[0074] At step 523, the reference identification application 162 determines whether the user wants to select a different candidate object 352, 356 included in the scene 340-1. When the reference identification application 162 determines that the user wants to make a different selection, the reference identification application 162 may select a different selected object 402 by returning to step 511, where the reference identification application 162 selects one of the remaining candidate objects 352 in the group of candidate objects 352, 356 in the scene 340-1. After selecting a different selected object 402, the reference identification application 162 may proceed with steps 513-517 by generating a query 404 to obtain additional information related to the different selected object 402. When the reference identification application 162 determines that the user does not want to make a different selection, the reference identification application 162 may end the method 500.

[0075] In summary, one or more sensors of a reference identification system acquire sensor data associated with one or more objects in an environment. A reference identification application included in the reference identification system stores scenes associated with the sensor data and associated metadata in a scene storage area. When the reference identification application determines that a user has triggered a request for information associated with an object included in a scene, the reference identification application loads and analyzes data associated with one or more of the scenes stored in the scene storage area. The reference identification application selects candidate objects included in one or more scenes and then queries the knowledge agent for information related to the selected objects. The reference identification application then presents information about the selected object to the user. If the user indicates that the selected object is not an object that the user wants to obtain more information about, the reference identification application selects a different object within one or more scenes and obtains information about the different object.

[0076] At least one technical advantage of the disclosed technology is that objects within a given scene can be identified even when the objects no longer exist within the scene. Another advantage is that information about objects identified in a scene can be obtained from one or more data sources and provided to a user without the user having to provide detailed information about the objects.

[0077] 1. In some embodiments, a computer-implemented method for identifying an object in an environment includes: receiving first sensor data associated with the environment via at least one sensor; storing the first sensor data in association with a first scene in a memory; and in response to receiving a user request for information associated with the environment: selecting the first scene based on the user request; accessing the first sensor data associated with the first scene via the memory; and analyzing the first sensor data to identify a first object included in the first scene; and causing information associated with the first object to be output via at least one output device.

[0078] 2. A computer-implemented method as described in clause 1, further comprising: storing second sensor data in the memory in association with a second scene; and in response to receiving the user request for the information associated with the environment: selecting the second scene based on the user request; accessing the second sensor data associated with the second scene via the memory; and analyzing the second sensor data to identify the first object, wherein the first object is included in at least one of the first scene and the second scene.

[0079] 3. The computer-implemented method of clauses 1 to 2, wherein a first scene includes the first object and a second scene does not include the first object.

[0080] 4. A computer-implemented method as described in any of clauses 1 to 3, wherein the first scene is associated with a first timestamp, the second scene is associated with a second timestamp, and the first timestamp and the second timestamp are included in a time period specified in the user request.

[0081] 5. A computer-implemented method as described in any of clauses 1 to 4, further comprising: analyzing the first sensor data to identify a second object included in the first scene; selecting the first object from at least the first object and the second object; and obtaining information associated with the first object.

[0082] 6. A computer-implemented method as described in any of clauses 1 to 5, wherein selecting the first object includes: calculating a first relevance score associated with the first object; calculating a second relevance score associated with the second object; and comparing the first relevance score with the second relevance score, wherein the first relevance score is greater than the second relevance score.

[0083] 7. A computer-implemented method as described in any of clauses 1 to 6, wherein calculating the first relevance score includes: calculating a first relevance factor for the first object; and calculating a second relevance factor for the first object, wherein the first relevance score is based at least on the first relevance factor and the second relevance factor.

[0084] 8. The computer-implemented method of any of clauses 1 to 7, wherein receiving the user request comprises detecting a triggering event initiated by the user.

[0085] 9. A computer-implemented method as described in any one of clauses 1 to 8, further comprising: receiving first external data associated with the first scene via at least one external data storage area; and storing the first external data in the memory in association with the first scene.

[0086] 10. In some embodiments, one or more non-transitory computer-readable media, which include instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps: receiving first sensor data associated with an environment via at least one sensor; storing the first sensor data in association with a first scene in a memory and storing first external data in association with the first scene in a memory, wherein the first external data is received from at least one external data source; and in response to receiving a user request for information associated with the environment: selecting the first scene based on the user request; accessing at least one of the first sensor data and the first external data via the memory; and analyzing at least one of the first sensor data and the first external data to identify a first object included in the first scene; and causing information associated with the first object to be output via at least one output device.

[0087] 11. A non-temporary computer-readable medium as described in clause 10, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: receiving a user request for additional information associated with the first object; obtaining supplemental information associated with the first object; and causing the supplemental information associated with the first object to be output via at least one output device.

[0088] 12. A non-transitory computer-readable medium as described in clause 10 or 11, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: selecting a first knowledge agent based on the first object, wherein the first knowledge agent provides the information associated with the first object.

[0089] 13. A non-transitory computer-readable medium as described in any of clauses 10 to 12, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: selecting a second knowledge agent based on the first object, wherein the second knowledge agent is different from the first knowledge agent and provides the supplemental information associated with the first object.

[0090] 14. A non-transitory computer-readable medium as described in any one of clauses 10 to 13, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: detecting a trigger event initiated by the user, wherein the trigger event initiates the user request, and wherein selecting the first scene based on the user request includes: determining a first timestamp associated with the trigger event; determining that the first scene is associated with the first timestamp; and selecting the first scene.

[0091] 15. A non-transitory computer-readable medium as described in any of clauses 10 to 14, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: in response to receiving the user request for information associated with the environment: accessing first metadata associated with the first scene via the memory; and analyzing the first metadata to identify a first object included in the first scene.

[0092] 16. A non-transitory computer-readable medium as described in any of clauses 10 to 15, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: storing second sensor data in association with a second scene in the memory; and in response to receiving the user request for information associated with the environment: selecting the second scene based on the user request; accessing the second sensor data via the memory; and analyzing the second sensor data to identify the first object, wherein the first object is included in at least one of the first scene and the second scene.

[0093] 17. A non-transitory computer-readable medium as described in any of clauses 10 to 16, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: analyzing the first sensor data to identify a second object included in the first scene; selecting the first object from the first object and the second object; and obtaining information associated with the first object.

[0094] 18. The non-transitory computer-readable medium of claim 17, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: calculating a first relevance score associated with the first object; calculating a second relevance score associated with the second object; and comparing the first relevance score to the second relevance score, wherein the first relevance score is greater than the second relevance score.

[0095] 19. In some embodiments, an object identification system includes: at least one sensor, the at least one sensor being configured to generate first sensor data associated with an environment; a memory, the memory being configured to store the first sensor data in association with a first scene; and a processor, the processor being coupled to the at least one sensor and the memory and being configured to: receive the first sensor data from the at least one sensor; store the first sensor data in the memory; and in response to receiving a user request for information associated with the environment: select the first scene based on the user request; analyze at least one of the first sensor data and first metadata associated with the first scene to identify a first object included in the first scene; and transmit a query associated with the first object to a first knowledge agent; receive a first response from the first knowledge agent, the first response including information associated with the first object; and cause the information associated with the first object to be output via at least one output device.

[0096] 20. An object identification system as described in clause 19, wherein the processor is further configured to: analyze at least one of the first sensor data and the first metadata to identify a second object included in the first scene; calculate a first relevance score associated with the first object; calculate a second relevance score associated with the second object; compare the first relevance score with the second relevance score, wherein the first relevance score is greater than the second relevance score; and select the first object from the first object and the second object.

[0097] Any and all combinations of any claim elements as described in any of the claims and / or any elements described in this application, in any manner, are within the scope of this disclosure and protection.

[0098] The description of various embodiments has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0099] Aspects of the present embodiment may be embodied as a system, method or computer program product. Therefore, aspects of the present disclosure may take the form of a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.) or a combination of software and hardware implementations, which may be generally referred to as "modules" or "systems" herein. In addition, any hardware and / or software technology, process, function, component, engine, module or system described in the present disclosure may be implemented as a circuit or circuit group. In addition, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable media, on which a computer-readable program code is embodied.

[0100] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media would include the following: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus.

[0101] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiments of the present disclosure. It will be understood that each frame in the flowchart and / or block diagram and the frame combination in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine. The instructions enable the function / action specified in one or more frames of the flowchart and / or block diagram to be realized when the processor of the computer or other programmable data processing device is executed. Such processors can be, but are not limited to, general-purpose processors, special-purpose processors, specific application processors or field programmable gate arrays.

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a fragment or a portion of a code, and the code includes one or more executable instructions for implementing the specified one or more logical functions. It should also be noted that in some optional implementations, the functions indicated in the box can appear in an order different from the order indicated in the accompanying drawings. For example, the two boxes shown in succession can actually be executed substantially simultaneously, or the boxes can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each box of the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can be implemented by a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0103] While the foregoing is directed to various embodiments of the present disclosure, other and further embodiments of the present disclosure may be envisaged without departing from the basic scope of the present disclosure, and the scope of the present disclosure is determined by the claims that follow.

Claims

1. A computer-implemented method for identifying an object in an environment, the method comprising: receiving, via at least one sensor, first sensor data associated with the environment; storing the first sensor data in association with a first scene in a memory; as well as In response to receiving a user request for information associated with the environment: selecting the first scene based on the user request; accessing, via the memory, the first sensor data associated with the first scene; as well as analyzing the first sensor data to identify a first object included in the first scene; By analyzing the first object, determining that a first knowledge agent included in a plurality of knowledge agents has information associated with the first object; In response to determining that the first knowledge agent has information associated with the first object, selecting the first knowledge agent from a plurality of knowledge agents and transmitting a query associated with the first object to the first knowledge agent, wherein the first knowledge agent provides information associated with the first object; as well as Information associated with the first object is caused to be output via at least one output device.

2. The computer-implemented method of claim 1 , further comprising: storing second sensor data in association with a second scene in the memory; as well as In response to receiving the user request for the information associated with the environment: selecting the second scene based on the user request; accessing, via the memory, the second sensor data associated with the second scene; as well as The second sensor data is analyzed to identify the first object, wherein the first object is included in at least one of the first scene and the second scene. 3 . The computer-implemented method of claim 2 , wherein the first scene includes the first object and the second scene does not include the first object.

4. The computer-implemented method of claim 2, wherein the first scene is associated with a first timestamp, the second scene is associated with a second timestamp, and the first timestamp and the second timestamp are included in a time period specified in the user request.

5. The computer-implemented method of claim 1 , further comprising: The first sensor data is analyzed to identify a second object included in the first scene.

6. The computer-implemented method of claim 5, further comprising: calculating a first relevance score associated with the first object based on at least one of a position of the first object within the first scene, a size of the first object within the first scene, and a brightness of the first object within the first scene; calculating a second relevance score associated with the second object; as well as In response to determining that the first relevance score is greater than the second relevance score, the first object is selected from the first object and the second object.

7. The computer-implemented method of claim 6, wherein calculating the first relevance score comprises: calculating a first relevance factor of the first object; as well as A second relevance factor for the first object is calculated, wherein a first relevance score is based on at least the first relevance factor and the second relevance factor.

8. The computer-implemented method of claim 1, wherein receiving the user request comprises detecting a triggering event initiated by a user.

9. The computer-implemented method of claim 1, further comprising: receiving, via at least one external data storage area, first external data associated with the first scene; as well as The first external data is stored in the memory in association with the first scene.

10. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps: receiving, via at least one sensor, first sensor data associated with an environment; In storage: storing the first sensor data in association with a first scene, and storing first external data in association with the first scenario, wherein the first external data is received from at least one external data source; as well as In response to receiving a user request for information associated with the environment: selecting the first scene based on the user request; accessing at least one of the first sensor data and the first external data via the memory; as well as analyzing at least one of the first sensor data and the first external data to identify a first object included in the first scene; By analyzing the first object, determining that a first knowledge agent included in a plurality of knowledge agents has information associated with the first object; In response to determining that the first knowledge agent has information associated with the first object, selecting the first knowledge agent from a plurality of knowledge agents and transmitting a query associated with the first object to the first knowledge agent, wherein the first knowledge agent provides information associated with the first object; as well as Information associated with the first object is caused to be output via at least one output device.

11. The non-transitory computer readable medium of claim 10, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: receiving a user request for additional information associated with the first object; obtaining supplemental information associated with the first object; and The supplemental information associated with the first object is caused to be output via at least one output device.

12. The non-transitory computer-readable medium of claim 11, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: selecting a second knowledge agent based on the first object, wherein the second knowledge agent is different from the first knowledge agent and provides the supplemental information associated with the first object.

13. The non-transitory computer-readable medium of claim 10, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: detecting a triggering event initiated by a user, wherein the triggering event initiates the user request, and wherein selecting the first scenario based on the user request comprises: determining a first timestamp associated with the triggering event; Determining that the first scene is associated with the first timestamp; as well as Select the first scenario.

14. The non-transitory computer-readable medium of claim 10, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: in response to receiving the user request for information associated with the environment: accessing, via the memory, first metadata associated with the first scene; and The first metadata is analyzed to identify a first object included in the first scene.

15. The non-transitory computer readable medium of claim 10, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: storing second sensor data in association with a second scene in the memory; and In response to receiving the user request for information associated with the environment: selecting the second scene based on the user request; accessing the second sensor data via the memory; and The second sensor data is analyzed to identify the first object, wherein the first object is included in at least one of the first scene and the second scene.

16. The non-transitory computer readable medium of claim 10, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: The first sensor data is analyzed to identify a second object included in the first scene.

17. The non-transitory computer readable medium of claim 16, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: calculating a first relevance score associated with the first object based on at least one of a position of the first object within the first scene, a size of the first object within the first scene, and a brightness of the first object within the first scene; calculating a second relevance score associated with the second object; and In response to determining that the first relevance score is greater than the second relevance score, the first object is selected from the first object and the second object.

18. An object identification system, comprising: at least one sensor configured to generate first sensor data associated with an environment; a memory configured to store the first sensor data in association with a first scene; as well as a processor coupled to the at least one sensor and the memory and configured to: receiving the first sensor data from the at least one sensor; storing the first sensor data in the memory; as well as In response to receiving a user request for information associated with the environment: selecting the first scene based on the user request; analyzing at least one of the first sensor data and first metadata associated with the first scene to identify a first object included in the first scene; By analyzing the first object, determining that a first knowledge agent included in a plurality of knowledge agents has information associated with the first object; as well as In response to determining that the first knowledge agent has information associated with the first object, selecting the first knowledge agent from a plurality of knowledge agents and transmitting a query associated with the first object to the first knowledge agent; receiving a first response from the first knowledge agent, the first response including information associated with the first object; as well as The information associated with the first object is caused to be output via at least one output device.

19. The object identification system of claim 18, wherein the processor is further configured to: analyzing at least one of the first sensor data and the first metadata to identify a second object included in the first scene; calculating a first relevance score associated with the first object; calculating a second relevance score associated with the second object; comparing the first relevance score to the second relevance score, wherein the first relevance score is greater than the second relevance score; and The first object is selected from among the first object and the second object.

Images