Object perception based on surface detection and surface motion detection

Abstract

Embodiments relate to perceiving surfaces and objects. Trajectories can be generated based on a continuous stream of sensor events such that each trajectory is a parameterized representation of a curved line segment. The trajectories can be utilized to determine a surface. The trajectories can be provided to a modeling engine, which can perform one or more actions based on the trajectories and the surface. In response to a change in the surface, the trajectories can be updated based on the continuous stream of sensor events, and one or more additional actions can be performed based on the updated trajectories and the changed surface. The change in the surface can include one or more surface position changes, orientation changes, motion changes, deformations, etc. A shape corresponding to the surface can be determined based on characteristics of the surface or the trajectories.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is a utility patent application based on U.S. Provisional Patent Application Serial No. 63 / 205,480, filed December 14, 2020, the benefit of the filing date of which is hereby claimed under 35 U.S.C. § 119(e), the contents of which are incorporated by reference in their entirety.

[0002] (Technical field) The present invention relates generally to machine detection or machine vision systems, and particularly, but not exclusively, to object perception based on surface detection and surface motion detection. [Background technology]

[0003] Prior art robotic vision is primarily based on cameras where the input to the detection system is a two-dimensional (2D) array of pixels, with each pixel encoding the amount of light received over an exposure time, or on depth capture technologies (e.g., time-of-flight (ToF) cameras, structured light cameras, LIDAR, RADAR, or stereo cameras, to name a few) that provide three-dimensional (3D) point clouds, where each point in the point cloud can store its location in space relative to the vision system and any of a number of other pieces of data associated with the patch of reflective material from which the point was generated (e.g., brightness, color, relative radial velocity, spectral composition, to name a few). It should be noted that 3D point clouds can be represented in “frames,” similar to frames of images from a camera, meaning they do not have an underlying representation of continuously evolving time.

[0004] To provide useful sensory output that may be used by machine vision applications, such as robotic planning and control systems, these 2D or 3D data must be processed by machine vision algorithms implemented in software or hardware. In some cases, some machine vision systems may employ machine learning techniques to determine properties or characteristics of the world that may be salient to a particular robotic task, such as location, shape orientation, material properties, object classification, object motion, and the relative motion of the robotic system. Often, neither the 2D nor 3D representations utilized by conventional machine vision systems provide inherent support for a continuous surface representation of objects in the environment. Similarly, they often represent a scene using static data captured from sensors, and basic data about the scene may be filtered out before the data is provided to machine vision algorithms for processing. Thus, conventional machine vision systems are disadvantageous because they may rely on such inaccurate or unrepresentative scene data to perform machine vision analysis. It is with respect to these and other problems that the present invention was made.

[0005] Non-limiting and non-exhaustive embodiments of the present invention are described with reference to the following drawings, in which like reference numerals refer to like elements throughout the various views unless otherwise indicated. For a better understanding of the described invention, reference is made to the following detailed description of various embodiments, which should be read in connection with the accompanying drawings. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 illustrates a system environment in which various embodiments may be implemented. [Figure 2] FIG. 1 is a schematic diagram illustrating an embodiment of a client computer. [Figure 3] FIG. 1 is a schematic diagram illustrating an embodiment of a network computer. [Figure 4]FIG. 1 illustrates a logical architecture of a system for perceiving objects based on surface detection and surface movement detection, according to one or more of the various embodiments. [Figure 5] 1 is a logical schematic diagram of a system for perceiving objects based on surface detection and surface movement detection according to one or more of the various embodiments. [Figure 6] FIG. 1 illustrates a logical representation of sensors and sensor output information for perceiving an object based on detecting a surface and detecting movement of the surface, according to one or more of the various embodiments. [Figure 7] FIG. 1 illustrates a logical representation of a scan path for perceiving an object based on surface detection and surface motion detection, according to one or more of the various embodiments. [Figure 8] FIG. 1 illustrates a logical representation of a scanning system that perceives objects based on surface detection and surface movement detection, according to one or more of the various embodiments. [Figure 9] FIG. 1 illustrates a logical representation of a scanning system that perceives objects based on surface detection and surface movement detection, according to one or more of the various embodiments. [Figure 10] FIG. 1 illustrates a logical overview of a system for perceiving objects based on surface detection and surface movement detection, according to one or more of the various embodiments. [Figure 11] FIG. 1 illustrates a logical overview of a system for perceiving objects based on surface detection and surface movement detection, according to one or more of the various embodiments. [Figure 12] 1 is a schematic flow diagram illustrating a process for perceiving an object based on detecting a surface and detecting movement of the surface, according to one or more of the various embodiments. [Figure 13] 1 is a flow diagram illustrating a process for perceiving an object based on detecting a surface and detecting movement of the surface, according to one or more of the various embodiments. [Figure 14] 1 is a flow diagram illustrating a process for perceiving an object based on detecting a surface and detecting movement of the surface, according to one or more of the various embodiments. [Figure 15]1 is a flow diagram illustrating a process for perceiving an object based on detecting a surface and detecting movement of the surface, according to one or more of the various embodiments. [Figure 16] 1 is a flow diagram illustrating a process for perceiving an object based on detecting a surface and detecting movement of the surface, according to one or more of the various embodiments. [Figure 17] 1 illustrates a non-limiting use case of perceiving an object based on surface detection and surface motion detection according to one or more of the various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0007] Various embodiments will now be described in more detail below with reference to the accompanying drawings, which form a part hereof and which show, by way of example, specific exemplary embodiments in which the invention may be practiced. However, embodiments may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the embodiments to those skilled in the art. Among other things, various embodiments may be methods, systems, media, or devices. Accordingly, various embodiments may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Therefore, the following detailed description should not be taken in a limiting sense.

[0008] Throughout the specification and claims, the following terms shall have the meanings expressly associated therewith, unless the context clearly dictates otherwise. As used herein, the phrase "in one embodiment" may refer to the same embodiment, but does not necessarily refer to the same embodiment. Furthermore, as used herein, the phrase "in another embodiment" may refer to different embodiments, but does not necessarily refer to different embodiments. Thus, as described below, various embodiments can be readily combined without departing from the scope or spirit of the invention.

[0009] Additionally, as used herein, the term "or" is an inclusive "or" operator and is equivalent to the term "and / or" unless the context clearly dictates otherwise. The term "based on" is not inclusive and further allows for the basis of additional unstated factors unless the context clearly dictates otherwise. Additionally, throughout this specification, the meanings of "a," "an," and "the" include plural referents. The meaning of "in" includes "in" and "on."

[0010] With respect to the exemplary embodiments, the following terms are also used herein in accordance with their corresponding meanings, unless the context clearly dictates otherwise:

[0011] As used herein, the term "engine" refers to logic embodied in hardware or software instructions that may be written in a programming language, such as C, C++, Objective-C, COBOL, Java™, PHP, Perl, JavaScript, Ruby, VBScript, C#, or other Microsoft .NET™ languages. An engine may be compiled into an executable program or written in an interpreted programming language. A software engine may be called by other engines or by itself. An engine as described herein may refer to one or more logical modules that may be merged into other engines or applications, or may be divided into sub-engines. An engine may be stored on a non-transitory computer-readable medium or computer storage device, and may also be stored on and executed by one or more general-purpose computers, thus creating a special-purpose computer configured to provide the engine.

[0012] As used herein, the term "scanning signal generator" refers to a system or device capable of generating a beam that can be scanned / directed for projection into an environment. For example, the scanning signal generator can be a high-speed laser-based scanning device based on a dual-axis microelectromechanical system (MEMS) configured to scan a laser over a defined area of ​​interest. The characteristics of the scanning signal generator can vary depending on the application or service environment. The scanning signal generator is not strictly limited to lasers or laser MEMS; other types of beam signal generators can be utilized depending on the situation. Critical selection criteria for the scanning signal generator characteristics can include beam width, beam dispersion, beam energy, wavelength, phase, etc. The scanning signal generator can be selected to enable sufficiently precise energy reflection from the scanning surface or scanning object in the scanning environment of interest. The scanning signal generator can be designed to scan at frequencies up to 10 kHz. The scanning signal generator can be controlled in a closed-loop manner by one or more processors that can provide feedback regarding objects in the environment, directing the scanning signal generator to modify the scanning signal generator's amplitude, frequency, phase, etc.

[0013] As used herein, the term "sensor" refers to a device or system capable of detecting reflected energy from a scanning signal generator. A sensor may be thought of as including an array of detector cells that respond to energy reflected from the scanning signal generator. The sensor may provide an output that indicates which detector cells are triggered and when they are triggered. A sensor may be thought of as producing a sensor output that reports cell location and detection time for individual cells, rather than being limited to reporting the state or status of every cell. For example, a sensor may include an event sensor camera, a SPAD array, a SiPM array, etc.

[0014] As used herein, the terms "trajectory" and "surface trajectory" refer to one or more data structures that store or represent a parameterized representation of a curve segment that may correspond to a surface detected by one or more sensors. The trajectory may include one or more attributes / elements that correspond to constants or coefficients of a segment of a one-dimensional analytical curve in three-dimensional space. The trajectory of a surface is determined based on fitting or relating one or more sensor events to a known analytical curve. Sensor events that are inconsistent with the analytical curve may be considered noise or otherwise excluded from the trajectory.

[0015] As used herein, the term "configuration information" refers to information that may include rule-based strategies, pattern matching, scripts (e.g., computer-readable instructions), and the like, provided from a variety of sources, including configuration files, databases, user input, built-in defaults, plug-ins, extensions, and the like, or combinations thereof.

[0016] The following briefly describes embodiments of the invention to provide a basic understanding of some aspects of the invention. This brief description is not intended to be an extensive overview. It is not intended to identify key or critical elements or to delineate or otherwise narrow the scope. Its purpose is merely to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.

[0017] Briefly, various embodiments are directed to surface and object recognition. In one or more of the various embodiments, one or more trajectories can be generated based on a continuous stream of sensor events, such that each trajectory is a parameterized representation of a one-dimensional curve segment in three-dimensional space.

[0018] In one or more of the various embodiments, one or more trajectories can be utilized to determine one or more surfaces.

[0019] In one or more of the various embodiments, the one or more trajectories can be provided to a modeling engine to perform one or more actions based on the one or more trajectories and the one or more surfaces.

[0020] In one or more of the various embodiments, further actions may be performed in response to one or more changes in the one or more surfaces, including updating one or more trajectories based on the continuous stream of sensor events, performing one or more additional actions based on the one or more updated trajectories and the one or more changed surfaces, and the like.

[0021] In one or more of the various embodiments, the one or more changes to the one or more surfaces can include one or more of a change in position, a change in orientation, a change in motion, a deformation, etc. of the one or more surfaces.

[0022] In one or more of the various embodiments, a continuous stream of sensor events can be provided based on one or more sensors such that each sensor event includes one or more of a timestamp, a time of flight, or a position value.

[0023] In one or more of the various embodiments, one or more shapes corresponding to one or more surfaces can be determined based on one or more characteristics of the one or more surfaces and the one or more trajectories.

[0024] In one or more of the various embodiments, each trajectory may further include a B-spline parameterization.

[0025] In one or more of various embodiments, the one or more trajectories can be used to continuously determine one or more changes to one or more of the positions of one or more surfaces, the orientation of one or more surfaces, the deformation of one or more surfaces, the movement of one or more surfaces, etc.

[0026] In one or more of the various embodiments, the modeling engine can be configured to perform further operations including determining one or more objects based on one or more portions of the trajectory that can be associated with one or more portions of the surface.

[0027] In one or more of various embodiments, the modeling engine can be configured to perform further operations including determining one or more characteristics of the one or more objects based on the one or more trajectories, such that the one or more characteristics include one or more of a position, an orientation, a movement, or a deformation of the one or more objects.

[0028] Example Operating Environment 1 illustrates components of one embodiment of an environment in which embodiments of the present invention may be practiced. Not all of the components are required to practice the present invention, and variations in the arrangement and type of components may be made without departing from the spirit or scope of the present invention. As illustrated, system 100 of FIG. 1 includes a local area network (LAN) / wide area network (WAN) 110, a wireless network 108, client computers 102-105, an application server computer 116, a detection system 118, etc.

[0029] At least one embodiment of client computers 102-105 is described in detail below with respect to FIG. 2. In one embodiment, at least some of client computers 102-105 can operate over one or more wired or wireless networks, such as network 108 or 110. In general, client computers 102-105 can include virtually any computer capable of communicating over a network to send and receive information, perform various online activities, offline operations, etc. In one embodiment, one or more of client computers 102-105 can operate within a business or other entity and can be configured to perform a wide variety of services for the business or other entity. For example, client computers 102-105 can be configured to operate as a web server, a firewall, a client application, a media player, a mobile phone, a game console, a desktop computer, etc. However, client computers 102-105 are not limited to these services and can also be utilized in connection with end-user computing in other embodiments, for example. It should be understood that more or fewer client computers (shown in FIG. 1) may be included in the system as described herein, and that embodiments are therefore not constrained by the number or type of client computers utilized.

[0030] Computers that can operate as client computers 102 include computers that typically connect using wired or wireless communication media, such as personal computers, multiprocessor systems, microprocessor-based or programmable electronic devices, and network PCs. In some embodiments, client computers 102-105 can include virtually any portable computer capable of connecting to and receiving information from another computer, such as a laptop computer 103, a mobile computer 104, or a tablet computer 105. However, portable computers are not limited thereto and can also include other portable computers, such as cellular telephones, display pagers, radio frequency (RF) devices, infrared (IR) devices, personal digital assistants (PDAs), handheld computers, wearable computers, and integrated devices that combine one or more of the preceding computers. Thus, client computers 102-105 generally range widely in terms of functionality and features. Furthermore, client computers 102-105 can access a variety of computer applications, including browsers or other web-based applications.

[0031] A web-enabled client computer may include a browser application configured to send requests and receive responses over the web. The browser application may be configured to receive and display graphics, text, multimedia, etc. utilizing virtually any web-based language. In one embodiment, the browser application may display and send messages utilizing JavaScript, Hypertext Markup Language (HTML), Extensible Markup Language (XML), JavaScript Object Notation (JSON), Cascading Style Sheets (CSS), etc., or a combination thereof. In one embodiment, a user of a client computer may utilize the browser application to perform various activities over a network (online). However, other applications may also be used to perform various online activities.

[0032] The client computers 102-105 may also include at least one other client application configured to receive or send content to or from another computer. The client application may include functionality for sending or receiving content, etc. The client application may further provide information to identify itself, including type, function, name, etc. In one embodiment, the client computers 102-105 may uniquely identify themselves through any of a variety of means, including an Internet Protocol (IP) address, a telephone number, a mobile identification number (MIN), an electronic serial number (ESN), a client certificate, or other device identifier. Such information may be provided in one or more network packets transmitted between other client computers, the application server computer 116, the detection system 118, or other computers.

[0033] The client computers 102-105 may further be configured to include a client application that allows an end user to log in to an end-user account that may be managed by another computer, such as the application server computer 116 or the detection system 118. Such an end-user account, in one non-limiting example, may be configured to allow the end user to manage one or more online activities, including project management, software development, systems administration, configuration management, search activities, and social networking activities, browse various websites, and communicate with other users. The client computers may also be configured to allow the user to view reports, interactive user interfaces, or results provided by the detection system 118.

[0034] Wireless network 108 is configured to couple client computers 103-105 and components thereof to network 110. Wireless network 108 can include any of a wide variety of wireless sub-networks, which can further overlay standalone ad-hoc networks and the like, to provide infrastructure-oriented connectivity for client computers 103-105. Such sub-networks can include mesh networks, wireless local area network (WLAN) networks, cellular networks, and the like. In one embodiment, the system of the present invention can include more than one wireless network.

[0035] The wireless network 108 may further include an autonomous system of terminals, gateways, routers, etc., connected by wireless radio links, etc. These connectors may be configured to move freely and randomly and organize themselves arbitrarily, thus rapidly changing the topology of the wireless network 108.

[0036] The wireless network 108 may further utilize multiple access technologies, including second (2G), third (3G), fourth (4G), and fifth (5G) generation radio access for cellular systems, WLAN, wireless router (WR) mesh, etc. Access technologies such as 2G, 3G, 4G, 5G, and future access networks may enable wide-area coverage for mobile computers, such as the client computers 103-105, with various degrees of mobility. In one non-limiting example, the wireless network 108 may enable wireless connectivity via wireless network access such as Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wideband Code Division Multiple Access (WCDMA), High Speed ​​Downlink Packet Access (HSDPA), Long Term Evolution (LTE), etc. Essentially, wireless network 108 can include virtually any wireless communication means capable of moving information between client computers 103-105 and another computer, network, cloud-based network, cloud instance, etc.

[0037] Network 110 is configured to couple network computers to other computers, including application server computer 116, detection system 118, client computer 102, and client computers 103-105, such as via wireless network 108. Network 110 can utilize any form of computer-readable medium for transmitting information from one electronic device to another. Network 110 can also include the Internet, wide area networks (WANs), direct connections, such as via universal serial bus (USB) ports, Ethernet ports, other forms of computer-readable media, or any combination thereof, in addition to local area networks (LANs). In an interconnected set of LANs, including those based on different architectures and protocols, routers act as links between LANs, enabling messages to be sent from one to another. Additionally, while communication links within a LAN typically include twisted-pair or coaxial cable, communication links between networks may utilize other carrier means, including analog telephone lines, full or fractional dedicated digital lines including T1, T2, T3, and T4, or wireless links, including, for example, E-carrier, Integrated Services Digital Network (ISDN), Digital Subscriber Line (DSL), satellite links, or other communication links known to those skilled in the art. Furthermore, communication links may utilize any of a wide variety of digital signaling technologies, including, without limitation, DS-0, DS-1, DS-2, DS-3, DS-4, OC-3, OC-12, OC-48, etc. Furthermore, remote computers and other associated electronic devices may be remotely connected to either a LAN or a WAN via modems and temporary telephone links. In one embodiment, network 110 may be configured to transport Internet Protocol (IP) information.

[0038] Additionally, communication media typically embodies computer-readable instructions, data structures, program modules, or other transport mechanisms and includes any non-transitory or transitory distribution media of information. By way of example, communication media includes wired media such as twisted pair, coaxial cable, fiber optics, wave guides, and other wired media, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0039] Additionally, one embodiment of the application server computer 116 or the detection system 118 is described in more detail below with respect to FIG. 3. While FIG. 1 depicts the application server computer 116 and the detection system 118 as each a single computer, the inventions or embodiments are not so limited. For example, one or more functions of the application server computer 116, the detection system 118, etc., may be distributed across one or more separate network computers. Furthermore, in one or more embodiments, the detection system 118 may be implemented using multiple network computers. Furthermore, in one or more of various embodiments, the application server computer 116, the detection system 118, etc., may be implemented using one or more cloud instances in one or more cloud networks. Thus, these inventions and embodiments should not be construed as limited to a single environment or other configurations, and other architectures are also contemplated.

[0040] Example Client Computer 2 illustrates one embodiment of a client computer 200, which may include more or fewer components than those shown. Client computer 200 may represent, for example, one or more embodiments of the mobile computer or client computer illustrated in FIG.

[0041] Client computer 200 may include a processor 202 in communication with memory 204 via bus 228. Client computer 200 may also include a power supply 230, a network interface 232, an audio interface 256, a display 250, a keypad 252, an illuminator 254, a video interface 242, an input / output interface 238, a haptic interface 264, a global positioning system (GPS) receiver 258, an open-air gesture interface 260, a temperature interface 262, a camera 240, a projector 246, a pointing device interface 266, a processor-readable fixed storage device 234, and a processor-readable removable storage device 236. Client computer 200 may optionally communicate with a base station (not shown) or directly with another computer. Additionally, in one embodiment, although not shown, a gyroscope may be utilized within client computer 200 to measure or maintain the orientation of client computer 200.

[0042] The power supply 230 can provide power to the client computer 200. Power can be provided using a rechargeable or non-rechargeable battery. Power can also be provided by an external power source, such as an AC adapter or a powered docking cradle that supplements or recharges the battery.

[0043] The network interface 232 includes circuitry for coupling the client computer 200 to one or more networks and is further configured for use with one or more communications protocols and technologies, including, but not limited to, protocols and technologies implementing any portion of the OSI Model for Mobile Communications (GSM), CDMA, Time Division Multiple Access (TDMA), UDP, TCP / IP, SMS, MMS, GPRS, WAP, UWB, WiMAX, SIP / RTP, GPRS, EDGE, WCDMA, LTE, UMTS, OFDM, CDMA2000, EV-DO, HSDPA, or a wide variety of other wireless communications protocols. The network interface 232 may also be known as a transceiver, a transceiver device, or a network interface card (NIC).

[0044] Audio interface 256 may be configured to generate and receive audio signals, such as the sound of a human voice. For example, audio interface 256 may be coupled to a speaker and microphone (not shown) to enable telecommunication with others or to generate audio acknowledgments for certain actions. The microphone of audio interface 256 may also be used for input to or control of client computer 200, for example, using voice recognition, detecting touch based on sound, etc.

[0045] Display 250 may be a liquid crystal display (LCD), gas plasma, electronic ink, light emitting diode (LED), organic LED (OLED), or any other type of light reflective or light transmissive display that can be used with a computer. Display 250 may also include a touch interface 244 configured to receive input from an object such as a stylus or a human finger, and may further detect touch or gestures using resistive, capacitive, surface acoustic wave (SAW), infrared, radar, or other technologies.

[0046] Projector 246 may be a remote handheld projector or an integrated projector capable of projecting an image onto a distant wall or any other reflective object such as a remote screen.

[0047] Video interface 242 can be configured to capture video images, such as still images, video segments, infrared video, etc. For example, video interface 242 can be coupled to a digital video camera, a webcam, etc. Video interface 242 can include a lens, an image sensor, and other electronics. The image sensor can include a complementary metal-oxide semiconductor (CMOS) integrated circuit, a charge-coupled device (CCD), or any other integrated circuit for detecting light.

[0048] Keypad 252 may include any input device configured to receive input from a user. For example, keypad 252 may include a push-button numeric dial or a keyboard. Keypad 252 may also include command buttons associated with selecting and sending images.

[0049] Illuminator 254 can provide a status indication or provide light. Illuminator 254 can remain active for a specific period of time or in response to an event message. For example, when illuminator 254 is active, illuminator 254 can backlight the buttons of keypad 252 and remain on while the client computer is powered. Illuminator 254 can also backlight these buttons in various patterns when certain actions are performed, such as dialing another client computer. Illuminator 254 can also illuminate light sources positioned within a transparent or translucent case of the client computer in response to an action.

[0050] Additionally, client computer 200 may also include a hardware security module (HSM) 268 that provides additional tamper-resistant safeguards for generating, storing, or using security / cryptographic information, such as keys, digital certificates, passwords, passphrases, two-factor authentication information, etc. In some embodiments, the hardware security module may be utilized to support one or more standard public key infrastructures (PKIs), and may further be utilized to generate, manage, or store key pairs, etc. In some embodiments, HSM 268 may be a stand-alone computer, while in others, HSM 268 may be configured as a hardware card that can be added to a client computer.

[0051] Client computer 200 may also include an input / output interface 238 for communicating with external peripheral devices or other computers, such as other client computers and network computers. Peripheral devices may include audio headsets, virtual reality headsets, display screen glasses, remote speaker systems, remote speaker and microphone systems, etc. Input / output interface 238 may utilize one or more technologies, such as Universal Serial Bus (USB), infrared, WiFi, WiMAX, Bluetooth™, etc.

[0052] The input / output interface 238 may also include one or more sensors for determining geolocation information (e.g., GPS), monitoring power conditions (e.g., voltage sensors, current sensors, frequency sensors, etc.), monitoring weather (e.g., thermostat, barometer, anemometer, humidity detector, precipitation scale, etc.), etc. The sensors may be one or more hardware sensors that collect or measure data outside of the client computer 200.

[0053] Haptic interface 264 can be configured to provide tactile feedback to the user of the client computer. For example, haptic interface 264 can be utilized to vibrate client computer 200 in a particular manner when another user of the computer is calling. Temperature interface 262 can be used to provide temperature measurement input or temperature change output to the user of client computer 200. Open-air gesture interface 260 can detect physical gestures of the user of client computer 200, for example, by using single or stereo video cameras, radar, and gyroscope sensors within the computer held or worn by the user. Camera 240 can be used to track the physical eye movements of the user of client computer 200.

[0054] The GPS transceiver 258 can determine the physical coordinates of the client computer 200 on the Earth's surface, typically outputting a location as latitude and longitude values. The GPS transceiver 258 can also utilize other global positioning methods, including, but not limited to, triangulation, Assisted GPS (AGPS), Extended Observed Time Difference (E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), Extended Timing Advance (ETA), Base Station Subsystem (BSS), etc., to further determine the physical location of the client computer 200 on the Earth's surface. It should be understood that the GPS transceiver 258 can determine the physical location of the client computer 200 under a variety of circumstances. However, in one or more embodiments, the client computer 200, via other components, can provide other information that can be used to determine the client computer's physical location, including, for example, a Medium Access Control (MAC) address, an IP address, etc.

[0055] In at least one of various embodiments, applications such as operating system 206, other client applications 224, and web browser 226 can be configured to use geolocation information to select one or more localization features, such as time zone, language, currency, calendar formatting, etc. The localization features can be used in file systems, user interfaces, reports, and internal processes or databases. In at least one of various embodiments, the geolocation information used to select the localization information can be provided by GPS 258. Additionally, in some embodiments, the geolocation information can include information provided using one or more geolocation protocols over a network, such as wireless network 108 or network 111.

[0056] Human interface components can be peripheral devices that are physically separate from client computer 200 and enable remote input or output to client computer 200. For example, information routed as described herein through human interface components such as display 250 or keyboard 252 can instead be routed to an appropriate remotely located human interface component via network interface 232. Examples of human interface peripheral components that can be remote include, but are not limited to, audio devices, pointing devices, keypads, displays, cameras, projectors, etc. These peripheral components can communicate through pico-networks such as Bluetooth™, Zigbee™, etc. One non-limiting example of a client computer with such peripheral human interface components is a wearable computer that includes a remote pico-projector with one or more cameras that communicate remotely with a separately located client computer and can detect user gestures toward portions of an image projected by the pico-projector onto a reflective surface such as a wall or the user's hand.

[0057] The client computer may include a web browser application 226 configured to receive and send web pages, web-based messages, graphics, text, multimedia, etc. The browser application of the client computer may utilize virtually any programming language, including Wireless Application Protocol Messages (WAP), etc. In one or more embodiments, the browser application may utilize Handheld Device Markup Language (HDML), Wireless Markup Language (WML), WMLScript, JavaScript, Standard Generalized Markup Language (SGML), Hypertext Markup Language (HTML), Extensible Markup Language (XML), HTML5, etc.

[0058] The memory 204 may include RAM, ROM, or other types of memory. The memory 204 represents an example of a computer-readable storage medium (device) for storage of information such as computer-readable instructions, data structures, program modules, or other data. The memory 204 may store a BIOS 208 that controls the low-level operation of the client computer 200. The memory may also store an operating system 206 that controls the operation of the client computer 200. It will be appreciated that this component may include a general-purpose operating system such as a version of UNIX or Linux®, or a dedicated client computer communications operating system such as the Windows Phone™ or Symbian® operating systems. The operating system may include or be connected to a Java virtual machine module that enables control of the operation of hardware components or the operating system via Java application programs.

[0059] The memory 204 may further include one or more data storages 210 that the client computer 200 may utilize to store, among other things, applications 220 or other data. For example, the data storage 210 may also be utilized to store information describing various functions of the client computer 200. The information may then be provided to another device or computer in any of a variety of ways, including as part of a header during a communication, upon request, etc. The data storage 210 may also be utilized to store social networking information, including address books, buddy lists, aliases, user profile information, etc. The data storage 210 may also include program code, data, algorithms, etc., used by a processor, such as the processor 202, to perform and execute operations. In one embodiment, at least a portion of the data storage 210 may also be stored in another component of the client computer 200, including, but not limited to, a non-transitory processor-readable removable storage device 236, a processor-readable non-removable storage device 234, or external to the client computer.

[0060] Applications 220 may include computer-executable instructions that, when executed by client computer 200, send, receive, or otherwise process instructions and data. Applications 220 may include, for example, other client applications 224, web browsers 226, etc. Client computers may be configured to exchange communications such as queries, searches, messages, notification messages, event messages, sensor events, alerts, performance metrics, log data, API calls, or combinations thereof, with an application server or network monitoring computer.

[0061] Other examples of application programs include calendars, search programs, email client applications, IM applications, SMS applications, Voice over Internet Protocol (VOIP) applications, contact managers, task managers, transcoders, database programs, word processing programs, security applications, spreadsheet programs, games, search programs, and the like.

[0062] Additionally, in one or more embodiments (not shown), client computer 200 may include, instead of a CPU, an embedded logic hardware device such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable array logic (PAL), or a combination thereof. The embedded logic hardware device may directly implement embedded logic to perform operations. Also, in one or more embodiments (not shown), client computer 200 may include, instead of a CPU, one or more hardware microcontrollers. In one or more embodiments, the one or more microcontrollers may directly implement embedded logic to perform operations and may further access the microcontroller's own internal memory and its own external input and output interfaces (e.g., hardware pins or a wireless transceiver) to perform operations such as a system on a chip (SOC).

[0063] Example Network Computer 3 illustrates one embodiment of a network computer 300 that may be included in a system implementing one or more of the various embodiments. Network computer 300 may include more or fewer components than those shown in FIG. 3. However, the illustrated components are sufficient to disclose exemplary embodiments implementing these inventions. Network computer 300 may represent, for example, one embodiment of application server computer 116 or at least one of detection systems 118 of FIG. 1.

[0064] A network computer such as network computer 300 can include a processor 302 that can communicate with memory 304 via bus 328. In some embodiments, processor 302 can be comprised of one or more hardware processors or one or more processor cores. In some cases, one or more of the one or more processors can be special-purpose processors designed to perform one or more specialized operations, such as those described herein. Network computer 300 also includes a power supply 330, a network interface 332, an audio interface 356, a display 350, a keyboard 352, an input / output interface 338, a processor-readable fixed storage device 334, and a processor-readable removable storage device 336. Power supply 330 provides power to network computer 300.

[0065] The network interface 332 is configured to be used with one or more communication protocols and technologies, including, but not limited to, protocols and technologies implementing any portion of the Open Systems Interconnection Model (OSI model), Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), User Datagram Protocol (UDP), Transmission Control Protocol / Internet Protocol (TCP / IP), Short Message Service (SMS), Multimedia Messaging Service (MMS), General Packet Radio Service (GPRS), WAP, Ultra Wideband (UWB), IEEE 802.16 Worldwide Interoperability for Microwave Access (WiMAX), Session Initiation Protocol / Real-time Transport Protocol (SIP / RTP), or a wide variety of other wired and wireless communication protocols. The network interface 332 may also be known as a transceiver, a transceiver device, or a network interface card (NIC). Network computer 300 may optionally communicate with a base station (not shown) or directly with another computer.

[0066] Audio interface 356 is configured to generate and receive audio signals, such as the sound of a human voice. For example, audio interface 356 may be coupled to a speaker and microphone (not shown) to enable telecommunication with others or to generate audio acknowledgments of certain actions. The microphone of audio interface 356 may also be used for input to or control of network computer 300, for example, using voice recognition.

[0067] Display 350 can be a liquid crystal display (LCD), gas plasma, electronic ink, light emitting diode (LED), organic LED (OLED), or any other type of light reflective or light transmissive display that can be used with a computer. In some embodiments, display 350 can be a handheld projector or picoprojector that can project an image onto a wall or other object.

[0068] Network computer 300 may also include an input / output interface 338 for communicating with external devices or computers not shown in Figure 3. Input / output interface 338 may utilize one or more wired or wireless communication technologies, such as USB™, Firewire™, WiFi, WiMAX, Thunderbolt™, infrared, Bluetooth™, Zigbee™, serial port, parallel port, etc.

[0069] Input / output interface 338 may also include one or more sensors for determining geolocation information (e.g., GPS), monitoring power conditions (e.g., voltage sensors, current sensors, frequency sensors, etc.), monitoring weather (e.g., thermostat, barometer, anemometer, humidity detector, precipitation scale, etc.), etc. The sensors may be one or more hardware sensors that collect or measure data outside of network computer 300. Human interface components may be physically separate from network computer 300, allowing for remote input or output to network computer 300. For example, information routed as described herein through human interface components such as display 350 or keyboard 352 may instead be routed via network interface 332 to an appropriate human interface component located anywhere on the network. Human interface components include any component that enables a computer to receive input from or transmit output to a human user of the computer. Thus, pointing devices such as a mouse, stylus, trackball, etc. may communicate and receive user input via pointing device interface 358.

[0070] The GPS transceiver 340 can determine the physical coordinates of the network computer 300 on the Earth's surface, typically outputting a location as latitude and longitude values. The GPS transceiver 340 can also utilize other geopositioning methods, including, but not limited to, triangulation, Assisted GPS (AGPS), Extended Observed Time Difference (E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), Extended Timing Advance (ETA), Base Station Subsystem (BSS), etc., to further determine the physical location of the network computer 300 on the Earth's surface. It should be understood that under various circumstances, the GPS transceiver 340 can determine the physical location of the network computer 300. However, in one or more embodiments, the network computer 300, via other components, can provide other information that can be used to determine the physical location of the client computer, including, for example, a Medium Access Control (MAC) address, an IP address, etc.

[0071] In at least one of various embodiments, applications such as operating system 306, detection engine 322, modeling engine 324, and web services 329 can be configured to utilize geolocation information to select one or more localization features, such as time zone, language, currency, currency formatting, calendar formatting, etc. The localization features can be used in file systems, user interfaces, reports, and internal processes or databases. In at least one of various embodiments, the geolocation information used to select the localization information can be provided by GPS 340. Additionally, in some embodiments, the geolocation information can include information provided using one or more geolocation protocols over a network, such as wireless network 108 or network 111.

[0072] Memory 304 may include random access memory (RAM), read-only memory (ROM), or other types of memory. Memory 304 represents an example of a computer-readable storage medium (device) for storage of information such as computer-readable instructions, data structures, program modules, or other data. Memory 304 stores a basic input / output system (BIOS) 308, which controls the low-level operation of network computer 300. Memory also stores an operating system 306, which controls the operation of network computer 300. It will be understood that this component may include a general-purpose operating system, such as a version of UNIX® or Linux®, or a dedicated operating system, such as Microsoft's Windows® operating system or Apple's macOS® operating system. The operating system may include or interface with one or more virtual machine modules, such as a Java virtual machine module, which enables control of the operation of hardware components or the operating system via Java application programs. Other runtime environments may also be included.

[0073] The memory 304 may further include one or more data storages 310 that may be utilized by the network computer 300 to store, among other things, applications 320 or other data. For example, the data storage 310 may also be utilized to store information describing various functions of the network computer 300. The information may then be provided to another device or computer in any of a variety of ways, including as part of a header during communication, upon request, etc. The data storage 310 may also be utilized to store social networking information, including address books, buddy lists, aliases, user profile information, etc. The data storage 310 may further include program code, data, algorithms, etc., used by a processor, such as the processor 302, to perform and execute operations, such as those described below. In one embodiment, at least a portion of the data storage 310 may be stored in another component of the network computer 300, including, but not limited to, a non-transitory medium within the processor-readable removable storage device 336, the processor-readable non-transitory storage device 334, or any other computer-readable storage device within the network computer 300, or external to the network computer 300. The data storage 310 may include, for example, an evaluation module 314 .

[0074] Applications 320 may include computer-executable instructions that, when executed by network computer 300, send, receive, or otherwise process messages (e.g., SMS, multimedia messaging service (MMS), instant messaging (IM), email, or other messages), audio, video, and / or other communications, enabling telecommunications with other users of other mobile computers. Other examples of application programs include calendars, search programs, email client applications, IM applications, SMS applications, Voice over Internet Protocol (VOIP) applications, contact managers, task managers, transcoders, database programs, word processing programs, security applications, spreadsheet programs, games, search programs, and the like. Applications 320 may include a detection engine 322, a modeling engine 324, web services 329, and the like, which may be configured to perform the operations of embodiments described below. In one or more of various embodiments, one or more of the applications may be implemented as a module or component of another application. Additionally, in one or more of various embodiments, an application may be implemented as an operating system extension, module, plug-in, or the like.

[0075] Furthermore, in one or more of various embodiments, the discovery engine 322, the modeling engine 324, the web services 329, etc. can operate in a cloud-based computing environment. In one or more of various embodiments, these applications, including the management platform, and others, can be executed within virtual machines or virtual servers that can be managed in the cloud-based computing environment. In one or more of various embodiments, applications in this context can flow from one physical network computer to another within the cloud-based environment, depending on performance and scaling issues that are automatically managed by the cloud computing environment. Similarly, in one or more of various embodiments, virtual machines or virtual servers dedicated to the discovery engine 322, the modeling engine 324, the web services 329, etc. can be automatically deployed and deactivated.

[0076] Additionally, in one or more of various embodiments, the detection engine 322, modeling engine 324, web services 329, etc. may be located on virtual servers running in a cloud-based computing environment rather than being tied to one or more specific physical network computers.

[0077] Additionally, network computer 300 may also include hardware security module (HSM) 360, which provides additional tamper-resistant safeguards for generating, storing, or using security / cryptographic information, such as keys, digital certificates, passwords, passphrases, two-factor authentication information, etc. In some embodiments, hardware security module may be utilized to support one or more standard public key infrastructure (PKI) systems, and may further be utilized to generate, manage, or store key pairs, etc. In some embodiments, HSM 360 may be a stand-alone network computer; in other cases, HSM 360 may be configured as hardware that can be installed on a network computer.

[0078] Additionally, in one or more embodiments (not shown), the network computer 300 may include an embedded logic hardware device, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable array logic (PAL), or a combination thereof, instead of a CPU. The embedded logic hardware device may directly execute embedded logic to perform operations. Also, in one or more embodiments (not shown), the network computer may include one or more hardware microcontrollers instead of a CPU. In one or more embodiments, the one or more microcontrollers may directly execute embedded logic specific to the microcontroller to perform operations and may further access the microcontroller's own internal memory and specific external input and output interfaces (e.g., hardware pins or a wireless transceiver) to perform operations such as a system on a chip (SOC).

[0079] Example Logical System Architecture FIG. 4 illustrates a logical architecture of a system 400 for perceiving objects based on surface detection and surface motion detection, according to one or more of the various embodiments.

[0080] In this example, for some embodiments, a detection system, such as system 400, can include one or more servers, such as detection server 402. In some embodiments, the detection server can be configured to include one or more detection engines, such as detection engine 404, one or more modeling engines, such as modeling engine 404, and one or more detection engines, such as detection engine 406.

[0081] Additionally, in some embodiments, the detection system may include one or more signal generators capable of generating at least sensor information based on where energy from the signal generators reflects off the surface. In this example, in some embodiments, signal generator 408 may be considered a laser scanning system. Further, in some embodiments, the detection system may include one or more sensors capable of receiving reflected signal energy. In this example, in some embodiments, the sensors may be considered sensors that can be configured to generate sensor information corresponding to the reflected signal energy. In this example, sensors such as sensor 410, sensor 412, and sensor 414 may be considered, for example, to be CCDs that provide two-dimensional (2D) sensor information based on CCD cells detecting reflected signal energy.

[0082] Thus, in some embodiments, the 2D sensor information from each sensor may be provided to a detection engine, such as detection engine 404. In some embodiments, the detection engine may be configured to synthesize the 2D points provided by the sensors into 3D points, such as based on triangulation.

[0083] Additionally, in some embodiments, the detection engine can be configured to be utilized to direct the signal generator (e.g., scanning laser 408) to follow a particular pattern based on one or more path functions. Thus, in some embodiments, the signal generator can scan a subject area using a known and precise path that can be defined or described using one or more functions corresponding to the curve / path of the scan.

[0084] Thus, in some embodiments, the detection engine can be configured to synthesize information about the object or surface being scanned by the signals generated based on 3D sensor information provided by the sensor and a known scanning curve pattern.

[0085] In some embodiments, the scanning signal generator 408 can be implemented using one or more high-speed laser scanning devices, such as a dual-axis MEMs mirror that scans a laser beam. In some embodiments, the laser wavelength can range from UV to IR. In some embodiments, the scanning signal generator can be designed to scan at frequencies up to 10 kHz. In some embodiments, the scanning signal generator can be controlled in a closed-loop manner using one or more processors that can provide feedback regarding objects in the environment and further direct the scanning signal generator to adapt one or more of the amplitude, frequency, phase, etc. Optionally, in some embodiments, the scanning signal generator can be configured to periodically switch on and off, where the scanner may slow down before changing direction or reversing direction.

[0086] In some embodiments, system 400 may include two or more sensors, such as sensor 410, sensor 412, and sensor 414. In some embodiments, the sensors may include an array of pixels or cells that respond to reflected signal energy. In some embodiments, the sensors may be configured such that some or all of the sensors share a portion of the field of view with each other and the scanning signal generator. Furthermore, in some embodiments, the relative position and pose of each sensor may be known. Also, in some embodiments, each sensor utilizes a synchronized clock. For example, in some embodiments, the sensors may synchronize their time by using one sensor's clock as a master clock or by using an external source that periodically sends a synchronization signal to the sensors. Alternatively, in some embodiments, the sensors may be configured to provide sensor events to the detection engine independently of or synchronized with each other.

[0087] Thus, as the beam from the scanning signal generator beam scans across the scene, the sensors trigger events at cells / pixels based on receiving reflected signal energy (e.g., photons / light from a laser) and observing physical reflections from the scene. Thus, in some embodiments, each sensor event (e.g., sensor event) can be determined based on a cell's location and a timestamp based on where and when the reflected energy is detected at each sensor. Thus, in some embodiments, each sensor independently reports each sensor event as it is detected, rather than collecting information / signals from the entire sensor array before providing the sensor event. This behavior can be distinguished from many conventional pixel arrays or CCDs, which can "raster scan" an entire array of cells before outputting signal data. In contrast, sensors such as sensor 410, sensor 412, and sensor 414 can instantly and continuously report signals (if any) from individual cells. Thus, rather than individual sensor cells sharing a common exposure time, each cell reports its own unique detected event. Thus, in some embodiments, sensors such as sensor 410, sensor 412, sensor 414 may be based on event sensor cameras, SPADs, SiPM arrays, or the like.

[0088] 5 illustrates a logical schematic diagram of a system 500 for perceiving objects based on surface detection and surface movement detection, according to one or more of various embodiments. In some embodiments, a detection engine, such as detection engine 502, can be configured to provide sensor output representing sensor information, such as position, timing, etc. As mentioned above, in some embodiments, a signal generator, such as a scanning laser, can scan an area of ​​interest such that reflections of energy are collected by the sensor. Thus, in some embodiments, information from each sensor can be provided to detection engine 502.

[0089] Additionally, in some embodiments, the detection engine 502 may be provided with a scan path that corresponds to the scan path of the scanning signal generator. Thus, in some embodiments, the detection engine 502 may utilize the scan path to determine the path that the scanning signal generator will traverse to scan an area of ​​interest.

[0090] Thus, in some embodiments, the detection engine 502 can be configured to generate sensor events corresponding to three-dimensional surface locations based on the sensor outputs. For example, if there can be three sensors, the detection engine can use triangulation to compute the location in the area of ​​interest where the scan signal energy was reflected. Those skilled in the art will appreciate that triangulation or other similar techniques can be applied to determine scan locations when the sensor locations are known.

[0091] In some embodiments, the scanning signal generator (e.g., a high-speed scanning laser) can be configured to perform a precision scanning pattern. Thus, in some embodiments, the detection engine 502 can be provided with a specific scan path function. Also, in some embodiments, the detection engine 502 can be configured to determine a specific scan path based on configuration information to take into account local requirements.

[0092] In one or more of the various embodiments, a detection engine, such as detection engine 502, can generate a sequence of surface trajectories that can be based on sensor information synthesized from the scan path and sensor output 504.

[0093] FIG. 6 illustrates a logical representation of sensors and sensor output information for perceiving objects based on detecting surfaces and detecting surface movement, according to one or more of the various embodiments.

[0094] In one or more of various embodiments, the detection engine can be provided with sensor outputs from a variety of sensors. In some embodiments, specific sensor characteristics can vary depending on the particular application, which can be directed to perceiving objects based on surface detection and surface movement detection. In this example, for some embodiments, sensor 602A can be considered to represent a general sensor capable of generating a signal corresponding to a precise location on the sensor where reflected energy from a scanning signal generator can be detected. For example, sensor 602A can be considered to be an array of detector cells reporting the cell location where the energy reflected from the scanning signal generator was detected. In this example, horizontal position 604 and vertical position 606 can be considered to represent locations corresponding to the location on sensor 602 where the reflected signal energy was detected.

[0095] In one or more of the various embodiments, the detection engine can be configured to receive sensor information of one or more detection events from one or more sensors. Accordingly, in some embodiments, the detection engine can be configured to determine additional information about the source of the reflected energy (beam position on the scanning surface) based on triangulation or other methods. In some embodiments, when the detection engine utilizes triangulation or other methods to locate the position of the signal beam in the scanning environment, the combined sensor information can be considered a single sensor event that includes a horizontal (x) position, a vertical position (y), and a time component (t). Additionally, in some embodiments, the sensor event can include other information, such as time-of-flight information, depending on the type or capabilities of the sensor.

[0096] Additionally, as described above, a scanning signal generator (e.g., a scanning laser) can be configured to traverse a precise path / curve (e.g., a scan path). Thus, in some embodiments, the pattern or sequence of cells in the sensor that detect reflected energy will follow a path / curve that is related to the path / curve of the scanning signal generator. Thus, in some embodiments, when the signal generator scans a particular path / curve, the associated path / curve of the active cells of the sensor can be detected. Thus, in this example, in some embodiments, path 608 can represent the sequence of cells in sensor 602B that detected reflected energy from the scanning signal generator.

[0097] In one or more of the various embodiments, the detection engine can be configured to fit the sensor events to a scan path curve. Thus, in one or more of the various embodiments, the detection engine can be configured to predict where the scan events should occur based on the scan path curve to determine information about the position or orientation of the scanned surface or object. Thus, in some embodiments, if the detection engine receives a sensor event that is not associated with a known scan path curve, the detection engine can be configured to perform various actions, such as closing the current trajectory and starting a new trajectory, discarding the sensor event as noise, etc.

[0098] In one or more of various embodiments, the scan path curve can be preconfigured within the limitations or constraints of the scan signal generator and sensor. For example, the scan signal generator can be configured or instructed to scan the scanned environment using various curves, including Lissajous curves, 2D lines, etc. In some cases, the scan path curve can be thought of as a piecewise function that changes direction or shape of different portions of the scan. For example, a 2D line scan path can be configured to change direction when approaching the edge of the scanned environment (e.g., field of view).

[0099] Those skilled in the art will appreciate that when an unobserved surface is scanned, the scanning frequency, scanning path, and sensor response frequency can determine whether the sensor detection path appears as a continuous path. Accordingly, the operating requirements of the scanning signal generator, sensor accuracy, sensor response frequency, etc., can be modified depending on the application of the system. For example, if the scanning environment is relatively poorly characterized and may be static, the sensor may have a low response time because the scanned environment does not change very quickly. Alternatively, for example, if the scanning environment is dynamic or contains many features of interest, the sensor may require high reactivity or precision to accurately capture the path of the reflected signal energy. Furthermore, in some embodiments, the characteristics of the scanning signal generator can be modified depending on the scanning environment. For example, if a laser is used for the scanning signal generator, the energy level, wavelength, phase, beam width, etc., can be tuned to suit the environment.

[0100] In one or more of the various embodiments, the detection engine may be provided with the sensor output as a continuous stream of sensor events or sensor information that identifies a cell location in the sensor cell array and a timestamp corresponding to the time the detection event occurred.

[0101] In this example, for some embodiments, data structure 610 can be considered a data structure representing sensor events based on sensor outputs provided to the detection engine. In this example, column 612 represents a horizontal position of a location in the scanned environment, column 614 represents a vertical position in the scanned environment, and column 616 represents a time of the event. Thus, in some embodiments, the detection engine can be configured to determine which sensor events, if any, should be associated with a trajectory. In some embodiments, the detection engine can be configured to associate the sensor events with an existing trajectory or create a new trajectory. In some embodiments, if the sensor events fit an expected / predicted curve determined based on the scan path curve, the detection engine can be configured to associate the sensor events with an existing trajectory or create a new trajectory. Also, in some cases, for some embodiments, the detection engine can be configured to determine one or more sensor events as noise if the position of the sensor events deviates from the predicted path by more than a defined threshold.

[0102] In one or more of various embodiments, the detection engine can be configured to determine sensor events for each individual sensor rather than being limited to providing computed sensor events based on outputs from multiple sensors. For example, in some embodiments, the detection engine can be configured to provide a data structure similar to data structure 610 for collecting sensor events for individual sensors.

[0103] In some embodiments, the detection engine can be configured to generate a sequence of trajectories corresponding to the reflected energy paths detected by the sensors. In some embodiments, the detection engine can be configured to utilize one or more data structures, such as data structure 618, to represent trajectories determined based on information captured by the sensors. In this example, data structure 610 can be a table-like structure including columns such as column 620 for storing a first x-position, column 622 for storing a second x-position, column 624 for storing a first y-position, column 626 for storing a second y-position, column 628 for storing a start time of the trajectory, and column 630 for storing an end time of the trajectory.

[0104] In this example, row 632 represents information for a first trajectory, and row 634 represents information for another trajectory. As described herein, the detection engine can be configured to utilize one or more rules or heuristics to determine whether one trajectory has ended and another trajectory has begun. In some embodiments, such heuristics can include observing sensor events occurring geometrically or temporally close together. It should be noted that the specific components or elements of the trajectory can vary depending on the parametric representation or type of analytical curve associated with the scan path and the shape or orientation of the scanned surface. Accordingly, one skilled in the art will appreciate that different types of analytical curves or curve representations can result in more or fewer parameters for each trajectory. Accordingly, in some embodiments, the detection engine can be configured to determine specific parameters of the trajectory based on rules, templates, libraries, etc., provided via configuration information to take local conditions or requirements into account.

[0105] In one or more of the various embodiments, the trajectory may be represented using curve parameters rather than a collection of individual points or pixels. Thus, in some embodiments, the detection engine may be configured to utilize one or more numerical methods to continuously fit a sequence of sensor events to a scan path curve.

[0106] Additionally, in some embodiments, the detection engine can be configured to utilize one or more smoothing methods to improve trajectory accuracy or trajectory fitting. For example, in some embodiments, a scan curve may include sensor events triggered by a scanning laser that may not be one cell wide because reflected energy may bounce off neighboring cells or reach two or more cell boundaries. Thus, in some embodiments, the detection engine can be configured to perform online smoothing estimation, e.g., using a smoothed Kalman filter to predict where a scan beam point should be in fractional units of the detector cell position and fractional units of the sensor's fundamental timestamp, to accurately estimate the actual position of the reflected signal beam as it traverses the sensor plane. Also, in some embodiments, the detection engine can be configured to utilize a batch-based optimization routine, such as weighted least squares, to fit smooth curves to successive segments of the scan trajectory, which may correspond to the time the scanning signal generator beam traverses over a continuous surface.

[0107] Additionally, in some embodiments, the scan path can be used to determine whether a trajectory has begun or ended. For example, if the scan path reaches the end of the scan area and changes direction again, the current trajectory can end, in some cases, and a new trajectory can begin to capture information based on the new direction of the scan. Also, in some embodiments, an object or other feature that blocks or obstructs the scanning energy or reflected scanning energy can result in a break in the sensor output, creating a gap or other discontinuity that can trigger the trajectory to close and another trajectory to open following the break or gap. Furthermore, in some embodiments, the detection engine can be configured to have a maximum trajectory length such that a trajectory can be closed if enough sensor events have been collected or enough time has passed since the trajectory began.

[0108] Additionally, in some embodiments, the detection engine may be configured to determine activation of individual sensors. Thus, in some embodiments, the detection engine may be configured to provide a data structure similar to data structure 618 for each sensor.

[0109] 7 illustrates a logical representation of a scan path for perceiving an object based on detecting a surface and detecting surface movement, according to one or more of various embodiments. In some embodiments, the detection engine can be configured to collect sensor output based on reflected scan signal energy. In some embodiments, the detection engine can be configured to interpret sensor output information based in part on the scan path of the scan signal generator.

[0110] In one or more of the various embodiments, the detection engine can be configured to direct the scan signal generator to traverse a particular path defined by one or more curved functions. For example, scan surface 702 represents a surface that is scanned using scan path 710. Thus, in some embodiments, the detection engine can anticipate or predict that reflected energy will follow the scan path. In this example, surface 702 is scanned using a straight linear path that can correspond to a particular 2D line function. In this example, only a portion of the scan pattern is shown.

[0111] Similarly, in some embodiments, surface 704 illustrates how a rapid scan can cover an entire surface. In some embodiments, the coverage of the scan can be varied depending on the scan path, scan frequency, etc., to further accommodate different applications or different environments. For example, some applications may require more precise scanning than other environments depending on different characteristics of the surface or object being scanned.

[0112] In some embodiments, the observed scan path collected by the sensor may deviate from the planned scan path due to objects or surface features that alter the observed or reflected scan path. For example, surface 706 shows a scan path similar to that shown for surface 702, but the scan path is shown distorted at path portion 712. In this example, this illustrates how intervening objects or surface features can alter the reflected scan energy path that can be observed by the sensor. For example, comparing the scan path of surface 702 with the scan path of surface 706 can indicate that an object or surface feature may be present on surface 706 and absent from surface 702.

[0113] Additionally, in some embodiments, the observed deviation of the scan path can correspond to the orientation of the scanned surface relative to the scanning signal generator or sensor. For example, in some embodiments, surface 708 can represent a surface that is rotated relative to surface 702 and the scanning signal generator (not shown). Thus, in some embodiments, the detection engine can be configured to determine the orientation of the surface based on a comparison of the scan path and the actual reflected scan path to determine that the surface can be rotated relative to the signal generator. Thus, in this example, trajectory 714 and the scan path generating trajectories 710A / 710B can be considered similar. However, in this example, the trajectories themselves are different due to differences in the orientation of the scanned surface. Thus, in some embodiments, the modeling engine can be configured to detect or characterize the difference between the scan path and the reflected path detected by the sensor to determine that surface 708 can be rotated relative to the detection system.

[0114] Further, in some embodiments, the detection engine can be configured to collect a sequence of trajectories corresponding to the reflected scanning energy detected by the sensor, as described above. In this example, for some embodiments, location 710A can represent the start of a trajectory, and location 710B can represent the end of the trajectory. Note that a trajectory can represent many sensor events that are correlated to define the trajectory. In this example, it can be assumed that the scanning signal generator is configured to change direction when it reaches the edge of the scanning environment. Thus, in this example, the location where the scanning signal generator changes direction can be treated as a discontinuity that ends one trajectory and begins another.

[0115] In one or more of various embodiments, the detection engine can be configured to orient the scan path so that the curves of the path intersect each other. Thus, in some embodiments, if the scan path intersects itself, the normal of the scanned surface can be calculated by determining the tangent to each of the trajectories of the intersection point. In some embodiments, the detection engine can be configured to compute the normal of the scanned surface to its sine by computing the cross product of the tangents. In some embodiments, the detection engine can be configured to determine the sign of the normal by selecting the direction of the normal that most closely points back toward the sensor system, since this is the only physically possible orientation of the surface that can be viewed by the sensor system. Additionally, in some embodiments, the curvature of the individual trajectories of the scan path across a surface or object can be used to approximate the curvature of a two-dimensional surface. For example, those skilled in the art will understand that intersecting 1D B-splines embedded in 3D space can be used to estimate 2D surface B-splines embedded in 3D space. Furthermore, in some embodiments, if many 1D curves cross the area parameterized by the 2D surface B-splines, the detection engine can be configured to further adjust the 2D surface B-splines to more accurately approximate the object surface.

[0116] 8 illustrates a logical representation of a scanning system 800 that perceives objects based on detecting surfaces and detecting surface movement, according to one or more of various embodiments. As described above, a scanning system such as scanning system 800 can include one or more scanning signal generators, such as scanning signal generator 802, and multiple sensors, such as sensor 804A, sensor 804B, and sensor 804C. It is noted that it is contemplated that one or more detection engines (not shown) or modeling engines (not shown) can be communicatively coupled to scanning signal generator 802 and sensors 804A, 804B, and 804C, etc. Alternatively, in some embodiments, one or more detection engines or modeling engines can be hosted by the same computer, device, or equipment that can provide the scanning signals or sensors of a robot, autonomous vehicle, etc.

[0117] Further, in this example, surface 806A represents a downward view of a surface in a scanning environment. Further, in this example, object 808A represents a downward view of an object interposed between surface 806A and scan signal generator 802. As described herein, scan signal generator 802 can be a scanning laser configured to traverse a defined scan path at a defined scan rate. Similarly, as described herein, sensor 804A, sensor 804B, sensor 804C, etc. can provide a sensor output to a detection engine based on energy reflected from surface 806A and object 808A. Accordingly, in some embodiments, the detection engine can be configured to generate a trajectory corresponding to the detection output and the scan path.

[0118] In this example, for some embodiments, surface 806B represents the same surface as surface 806A as viewed from the front. Similarly, in this example, object 808B represents the same object as object 808A as viewed from the front. Additionally, this example shows various positions that represent the start or end of a trajectory that can be determined from sensor output for some embodiments. Note that boundaries representing object 808B are included here for clarity and should not be confused as representing a trajectory or sensor output.

[0119] In this example, for some embodiments, the detection engine can be configured to determine trajectories based on the sensor outputs. In this example, the trajectories can be determined such that a first trajectory is defined by position 810A to position 812A, a second trajectory is defined by position 812A to position 814A, a third trajectory is defined by position 814A to position 816A, etc. Note that as described herein, trajectories also include a time component, which is omitted here for brevity and clarity.

[0120] Further, in this example, positions 810B, 812B, 814B, and 816B can be considered to represent proximity to the same positions discussed above. However, positions 818 and 820 can correspond to trajectories that do not include a break / gap.

[0121] Thus, in some embodiments, the modeling engine can be configured to determine information such as object shape, object features, object position, object motion, object rotation, surface features, and surface orientation based on evaluation of trajectories that can be determined by the detection engine. For example, here the detection engine can determine a trajectory based on position 810B, position 812B, position 814B, position 816B, etc. Also, for example, the detection engine can determine a trajectory based on position 818 and position 820. In some embodiments, the detection engine can be configured to compare / evaluate a sequence of trajectories to determine information about the scanned environment. In this example, the detection engine can be configured to recognize objects due to changes to the trajectory during scanning. In this example, a scan line resulting in one trajectory appearing adjacent to another scan line resulting in multiple trajectories can indicate the presence of an object with intervening surface features.

[0122] 9 illustrates a logical representation of a scanning system 900 that perceives objects based on detecting surfaces and detecting surface movement, according to one or more of various embodiments. As described above, a scanning system such as scanning system 900 can include one or more scanning signal generators, such as scanning signal generator 902, multiple sensors, such as sensor 904A, sensor 904B, sensor 904C, etc. It is contemplated that one or more detection engines or modeling engines can be communicatively coupled to scanning signal generator 902 and sensors 904A, 904B, 904C, etc. Alternatively, in some embodiments, the one or more detection engines or modeling engines can be hosted by the same computer or device that can provide the scanning signals or sensors of a robot, autonomous vehicle, etc.

[0123] It should be noted that scanning system 900 may be considered similar to scanning system 800 described above. Accordingly, for the sake of brevity and clarity, one or more features or embodiments described with respect to scanning system 800 may be omitted herein.

[0124] In one or more of various embodiments, the scanning system can be configured to scan a scanning environment that does not have a clearly defined background surface. Thus, in some embodiments, the scanning system can be directed to determining or identifying objects or object activity that may occupy space such that the sensor output cannot provide information about the background surface. However, in some embodiments, the detection engine can be configured to evaluate the sensor output associated with one or more object absence information associated with the background surface. Thus, in this example, object 906A represents a downward view of an object that may be scanned by the scanning signal generator 902.

[0125] In one or more of various embodiments, the detection engine can be configured to generate one or more trajectories for an object similar to the way trajectories for a surface can be generated. In this example, for some embodiments, 906B represents the same object as object 906A from the perspective of the scanning signal generator or sensor. Thus, in this example, for some embodiments, two or more locations, such as location 908A and location 910A with start and end times, can be used to determine a trajectory that can be associated with object 906B. Also, in this example, a trajectory can be determined based on an object 906B-absent trajectory that is associated with a background surface.

[0126] Further, in this example, location 908B and location 910B represent the proximity in the field of view of location 908A and location 910A. Thus, in some embodiments, the modeling engine can be configured to identify one or more features associated with object 906A / 906B based on an analysis of the trajectory associated with the scanned object, such as object features, object position, object motion, object rotation, etc.

[0127] Thus, in some embodiments, the detection engine can be configured to determine trajectories of objects in the scanned environment even if the scanned environment does not include a surface background. For example, for some embodiments, the detection engine determines trajectories for objects similar to rotatable objects 906A / B, such as a first trajectory having a start point at location 912 and an endpoint at location 914, a second trajectory having a start point at location 914 and an endpoint at location 916, etc. Thus, in this example, for some embodiments, the modeling engine can be configured to infer that objects 906A / B may be rotatable solids.

[0128] Further, in some embodiments, the modeling engine can be configured to infer various characteristics associated with the object based on a comparison of how the trajectories change over time. As shown here, in this example, time period 924 can represent the start and end times associated with a trajectory having location 908B as a start point and location 910B as an endpoint. Similarly, in this example, time period 926 can represent the start and end times associated with a trajectory having location 912 as a start point and location 914 as an endpoint, time period 928 can represent the start and end times associated with a trajectory having location 914 as a start point and location 916 as an endpoint, time period 930 can represent the start and end times associated with a trajectory having location 918 as a start point and location 920 as an endpoint, and time period 932 can represent the start and end times associated with a trajectory having location 920 as a start point and location 922 as an endpoint.

[0129] Thus, in some embodiments, the modeling engine can be configured to infer from the trajectory that the object 906A / 906B may be a rotating solid by evaluating the trajectory that the object 906A / 906B undergoes over time.

[0130] 10 illustrates a logical overview of a system 1000 for perceiving objects based on detecting surfaces and detecting surface motion, according to one or more of various embodiments. As noted above, in some embodiments, the detection engine can be configured to generate trajectory information based on sensor output collected based on reflected signal energy.

[0131] Thus, in some embodiments, a modeling engine, such as modeling engine 1002, can be configured to receive trajectory information, such as trajectory information 1004. In this example, for simplicity and clarity, trajectory information 1004 is shown using a diagram that can represent the trajectory information. However, in some embodiments, a detection engine can provide the modeling engine parameterized trajectory information provided via one or more data structures, such as data structure 618 shown in FIG. 6 .

[0132] Accordingly, the modeling engine 1002 can be configured to utilize an evaluation model 1006 to evaluate the trajectory information 1004. In some embodiments, the evaluation model can be configured to include one or more heuristics, rules, conditions, machine learning classifiers, etc. that can be utilized to evaluate the scanned environment based on the trajectory information 1004. In some embodiments, the evaluation model may be conventionally trained or tuned to recognize or perceive various objects, shapes, actions, activities, relationships between or within objects, etc. However, in some embodiments, the input data used to train or tune the evaluation model can be in the form of a numerical representation of the trajectory (e.g., a stream updating a parameter representation of analytical curve segments other than information derived from a point cloud, video frame capture, pixel-based edge detection, pixel-based motion detection, pixel-based color / luminance gradients, etc.).

[0133] In some embodiments, if the trajectory information 1004 can be evaluated, the modeling engine can be configured to generate or update one or more scene reports that provide information about the evaluated scene. In some embodiments, the scene reports can include traditional reports, interactive reports, graphical dashboards, charts, plots, etc. Also, in some embodiments, the scene reports can include one or more data structures that include information that can represent various scene features, which can be further provided to one or more machine vision applications, such as machine vision application 1010, that can automatically interpret the scene report.

[0134] Those skilled in the art will appreciate that many machine vision or machine perception applications can utilize the inventions described herein. For example, in one or more of various embodiments, a combination of high-speed surface scanning and low-latency, high-throughput sensors arranged in a configuration that provides direct depth measurements allows the detection engine to scan the entire field of view in 1 millisecond and measure surfaces with a radial accuracy that is 30 meters on the scale of a human nose or 0.5 meters on the scale of the individual threads of an M3 screw. Also, in some embodiments, the detection engine can be configured to incrementally scan the environment. Thus, in some embodiments, the detection engine can be configured to provide improved radial accuracy measurements of surfaces as more observations are collected.

[0135] Also, in some embodiments, the trajectory of the surface representation is built incrementally on a sub-millisecond scale, allowing the detection system to track motion by observing how the apparent distance of the surface of the most recently scanned region changes over a period that may be short enough to be considered a continuous time update. Thus, in some embodiments, the detection engine can be configured to provide a full 6D representation of reality, consisting of oriented 2D surfaces moving through 3D space over time. This native 6D representation can be advantageous because it can provide a rich and accurate basis on which various application-specific perception algorithms can operate.

[0136] In some embodiments, machine learning-based recognition algorithms can also use shape and motion primitives (e.g., trajectories) as inputs instead of 2D color contrast or 3D point arrays. Using trajectories to represent the scanned environment allows for individual and accurate identification of shapes and features present in the scanned environment. In contrast, some conventional machine vision systems may rely on guesswork or statistical approximations regarding whether an object in a location or phantom image can exist due to new contrast, or whether points in space are part of the same surface or how the surface is moving.

[0137] Furthermore, in some embodiments, the detection engine requires fewer data values ​​to represent surface or object features using parameterized trajectories rather than traditional representations using 2D pixels or 3D point clouds, and in some embodiments, surface / object representations using trajectories can be advantageous because they may be natively invariant under many transformations, such as rotation, translation, and lighting changes.

[0138] Thus, in some embodiments, the amount of data collected to train a deep learning recognition algorithm can be even less than with less rich representations such as 2D images or 3D point clouds. As an example, 2D color techniques utilized to recognize pedestrians may require a collection of data with the pedestrian in every possible pose and position for the system, along with all possible lighting conditions and surrounding background textures. In contrast, in some embodiments, the detection engine can be configured to not only locate and recognize the surface shapes of pedestrians in a wide variety of poses, but also to distinguish between the surface shapes of pedestrians in a wide variety of poses to indicate critical characteristics such as the orientation and relative motion of body parts, all within milliseconds of first seeing the pedestrian.

[0139] Additionally, in some embodiments, the detection systems described herein may be applied to other application domains beyond autonomous mobility. For example, in some embodiments, the detection and modeling engines may be utilized by a field fruit picking robot to locate ripe berries amongst dense foliage, discerning fine surface characteristics faster than a human picker or conventional picking machinery, and simultaneously calculating optimal picking positions with sub-millisecond updates as the robotic hand rapidly reaches to pick the berries.

[0140] 11 illustrates a logical schematic diagram of a system 1100 for perceiving an object based on detecting a surface and detecting surface movement, according to one or more of various embodiments. As described above, in some embodiments, a scanning signal generator can scan a surface in a scanning environment. In some cases, conditions of the scanning environment or characteristics of the surface being scanned can result in one or more spurious sensor events (e.g., noise) being generated by one or more sensors. For example, sensor view 1102 represents a portion of sensor events that may be generated during a scan.

[0141] In conventional machine vision applications, one or more 2D filters may be applied to captured video images, point clusters, etc. to attempt to separate noise events from signals of interest. In some cases, conventional 2D image-based filters may be disadvantageous because they may utilize one or more filters (e.g., weighted moving averages, Gaussian filters, etc.) that may rely on statistical evaluation of pixel colors / weights, pixel color / weight gradients, pixel discrimination / clustering, etc. Thus, in some cases, conventional 2D image filtering may be inherently fuzzy and highly dependent on application / environment assumptions. Also, in some cases, conventional noise detection / noise reduction methods may simultaneously misclassify one or more scene events as noise while erroneously missing some noise events.

[0142] In contrast, in some embodiments, the detection engine can be configured to associate sensor events with trajectories based on precise heuristics such as temporal proximity and location that can be used to fit the sensor events to analytical curves that can be predicted based on the scan path. Because the scan path is defined in advance, the detection engine can be configured to predict which sensor events should be included in the same trajectory.

[0143] Furthermore, in some embodiments, if a surface or object feature creates a gap or break in the trajectory, the detection engine can be configured to close the current trajectory and start a new trajectory as soon as it is further recognizable.

[0144] Additionally, in some embodiments, the detection engine can be configured to determine trajectories directly from sensor events having the form (x, y, t) rather than utilizing fuzzy pattern matching or pattern recognition methods. Thus, in some embodiments, the detection engine can be configured to accurately compute distance, direction, etc. rather than relying on fuzzy machine vision methods to distinguish noise from sensor events that would otherwise be on the same trajectory.

[0145] Generalized Actuation 12-17 illustrate generalized operations for perceiving an object based on surface detection and surface movement detection, according to one or more of various embodiments. In one or more of various embodiments, processes 1200, 1300, 1400, 1500, 1600, and 1700 described with respect to FIGS. 12-17 may be implemented or performed by one or more processors in a single network computer (or network monitoring computer), such as network computer 300 of FIG. 3. In other embodiments, these processes, or portions of the processes, may be implemented or performed by multiple network computers, such as network computer 300 of FIG. 3. In still other embodiments, these processes, or portions of the processes, may be implemented or performed by one or more virtualized computers in a cloud-based environment, or the like. However, embodiments are not limited thereto and various combinations of network computers, client computers, and the like may be utilized. Further, in one or more of various embodiments, the processes described with respect to Figures 12-16 may perform operations for perceiving objects based on surface detection and surface movement detection according to architectures such as those described with respect to at least one of the various embodiments or Figures 4-11. Further, in one or more of various embodiments, some or all of the operations performed by processes 1200, 1300, 1400, 1500, 1600, and 1700 may be performed in part by detection engine 322 or modeling engine 324 executing on one or more processors of one or more networked computers.

[0146] FIG. 12 illustrates a general flow diagram of a process 1200 for perceiving an object based on detecting surfaces and detecting surface movement, according to one or more of various embodiments. Following a start flow diagram block, in one or more of various embodiments, at flow diagram block 1202, there are one or more scanning signal generators, one or more sensors, etc. Also, in some embodiments, a specific scanning path can be provided to direct a beam or signal from the scanning signal generator to traverse a specified curve or path throughout the scanned environment. At block 1204, in one or more of various embodiments, the detection engine can be configured to utilize the scanning signal generator to scan a signal beam throughout the environment of interest to collect signal reflections of the sensor signal. At block 1206, in one or more of various embodiments, the detection engine can be configured to provide a scene trajectory based on the sensor output information. At block 1208, in one or more of various embodiments, the detection engine can be configured to provide one or more scene trajectories to a modeling engine. At block 1210, in one or more of various embodiments, the modeling engine can be configured to evaluate a scene in the scanned environment based on the trajectory. As described herein, the modeling engine can be configured to utilize various evaluation models that can be tuned or trained to identify one or more shapes, objects, object activities, etc. based on trajectories. At decision block 1212, in one or more of the various embodiments, if the scan can be terminated, control can be returned to the calling process; otherwise, control can be returned to block 1204.

[0147] FIG. 13 illustrates a flow diagram of a process 1300 for perceiving an object based on detecting surfaces and detecting surface movement, according to one or more of various embodiments. After start block 1302, in one or more of various embodiments, one or more sensors may capture signal reflections at the one or more sensors. At block 1304, in one or more of various embodiments, position and time information based on the sensor outputs may be provided to a detection engine. At block 1306, in one or more of various embodiments, the detection engine may be configured to determine one or more sensor events based on the scanning signal source positions and the sensor positions. At block 1308, in one or more of various embodiments, the detection engine may be configured to determine one or more scene trajectories based on the one or more sensor events and the scanning path of the signal beam. Then, in one or more of various embodiments, control may be returned to the calling process.

[0148] FIG. 14 illustrates a flow diagram of a process 1402 for perceiving an object based on detecting surfaces and detecting surface motion, according to one or more of various embodiments. After start block 1402, in one or more of various embodiments, a detection engine can be configured to scan a scanning environment using a beam from a scanning signal generator. At decision block 1404, in one or more of various embodiments, if a scan line intersects another previously collected scene trajectory, control can flow to block 1406; otherwise, control can return to the calling process. At block 1406, in one or more of various embodiments, the detection engine can be configured to determine a surface normal of the two-dimensional scanned surface based on the intersecting scan line. At block 1408, in one or more of various embodiments, the detection engine can be configured to determine an orientation of the scanned surface based on the computed surface normal. Then, in one or more of various embodiments, control can return to the calling process.

[0149] FIG. 15 illustrates a flow diagram of a process 1502 for perceiving an object based on detecting a surface and detecting surface movement, according to one or more of various embodiments. After start block 1502, in one or more of various embodiments, the detection engine can be configured to collect sensor events based on the output of one or more sensors. As described above, sensor events can be provided based on multiple reports (e.g., triangulation, etc.) from more than one sensor. Also, in some embodiments, sensor events can be provided individually for each sensor. At block 1504, in one or more of various embodiments, the detection engine can be configured to evaluate the one or more sensor events to determine a trajectory start point. At decision block 1506, in one or more of various embodiments, if a trajectory start point can be determined, control can flow to block 1508; otherwise, control can return to block 1504. At block 1508, in one or more of various embodiments, the detection engine can be configured to add the sensor events to a trajectory. In some embodiments, if sensor events can be collected, the sensor events can be associated with an open trajectory. In some embodiments, additional sensor events can be used to adjust or update the trajectory as more information is gathered. At block 1510, in one or more of various embodiments, the detection engine can be configured to fit / predict one or more sensor events to a trajectory based on the scan path. At decision block 1512, in one or more of various embodiments, if a gap or discontinuity is determined, control can flow to block 1514; otherwise, control can flow back to block 1508. At block 1514, in one or more of various embodiments, the detection engine can be configured to close the current trajectory. At block 1516, in one or more of various embodiments, the detection engine can be configured to provide the closed trajectory to the modeling engine. At decision block 1518, in one or more of various embodiments, if the scanning process can be terminated, control can flow back to the calling process; otherwise, control can flow back to block 1504. Then, in one or more of various embodiments, control can flow back to the calling process.

[0150] FIG. 16 illustrates a flow diagram of a process 1602 for perceiving an object based on detecting a surface and detecting surface movement, according to one or more of various embodiments. After start block 1602, in one or more of various embodiments, a detection engine can be configured to collect sensor events based on the output of one or more sensors. At decision block 1604, in one or more of various embodiments, if the sensor events can be included in a trajectory, control can pass to block 1608; otherwise, control can pass to block 1606. At block 1606, in one or more of various embodiments, the detection engine can be configured to filter out or discard the sensor events as noise. Then, in one or more of various embodiments, control can return to the calling process. At block 1608, in one or more of various embodiments, the detection engine can be configured to associate the sensor events with a trajectory. Then, in one or more of various embodiments, control can return to the calling process.

[0151] It will be understood that each illustrated block of each flowchart, and combinations of illustrated blocks in each flowchart, can be implemented by computer program instructions. These program instructions can be provided to a processor to create a machine, whereby the instructions, executed by the processor, generate means for performing the operations indicated in each flowchart block or blocks. The computer program instructions can be executed by the processor to cause a series of operational steps performed by the processor to create a computer-implemented process, whereby the instructions executed by the processor provide steps for performing the operations indicated in each flowchart block or blocks. The computer program instructions can also cause at least some of the operational steps indicated in each flowchart block to be performed in parallel. Furthermore, some of the steps can also be performed across more than one processor, such as may occur in a multiprocessor computer system. In addition, one or more blocks or combinations of blocks in each flowchart can also be executed concurrently with other blocks or combinations of blocks, or in a sequence different from that shown, without departing from the scope or spirit of the present invention.

[0152] Thus, each illustrated block of each flowchart supports a combination of means for performing the specified operations, a combination of steps for performing the specified operations, and program instruction means for performing the specified operations. It will also be understood that each illustrated block of each flowchart and combination of illustrated blocks of each flowchart can be implemented by a dedicated hardware-based system that performs the specified operations or steps, or a combination of dedicated hardware and computer instructions. The foregoing examples should not be construed as limiting or exhaustive, but rather as illustrative use cases illustrating the implementation of at least one of various embodiments of the present invention.

[0153] Additionally, in one or more embodiments (not shown), the logic of the example flow charts may be implemented using embedded logic hardware, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable array logic (PAL), or the like, or a combination thereof, instead of a CPU. The embedded logic hardware device may directly execute its own embedded logic to perform operations. In one or more embodiments, a microcontroller may be configured to directly execute its own embedded logic to perform operations and access its own internal memory and its own external input and output interfaces (e.g., hardware pins or a wireless transceiver) to perform operations such as a system-on-a-chip (SOC).

[0154] Example Use Cases In one or more of the various embodiments, the detection system can be used to perceive arbitrarily complex environments, depending on the application of the detection system. For brevity and clarity, much of the above description discloses inventions that utilize simple surfaces, objects, or scenes to enable object perception based on surface detection and surface movement detection. However, those skilled in the art will appreciate that some or all of these inventions can be utilized to perceive complex environments containing fewer or more complex objects or surfaces, which may be moving, experiencing deformations, surface changes, or shape changes that can be perceived as they occur in real time. As described with respect to the example shown in FIG. 9 (a rotating solid object), the detection engine can dynamically generate new trajectories or update existing trajectories based on how objects or surfaces in the scanned environment move or change. As described above, these updated / additional trajectories can be provided to the modeling engine in a form of a numerical representation, such as a vector, array, or matrix, where each element corresponds to a parameter value that is part of a parameter representation of a segment of the analytical curve. Accordingly, the modeling engine can be configured to utilize one or more evaluation models to identify one or more features associated with the detected surfaces based on the trajectories provided to the evaluation models. The particular feature of interest or the action taken in response to determining a particular feature may vary depending on the application.

[0155] In this example, environment 1700 represents a scene including a human hand. In this example, the contour lines may correspond to a trajectory determined from the scene. Thus, consistent with this example, in some embodiments, the modeling engine may be configured to utilize an evaluation model that has been tuned or trained to determine various characteristics of a complex object, such as a human hand, based on the trajectory. In this example, such characteristics may include shape, size, finger position, hand position, rotation, speed of movement, distance from other surfaces or other objects, etc.

Claims

1. 1. A method for perceiving surfaces and objects using one or more processors configured to execute instructions, comprising: The instruction: generating one or more trajectories based on a continuous stream of sensor events, each said trajectory being a parameterized representation of a one-dimensional curve segment in three-dimensional space; determining the one or more surfaces using the one or more trajectories; and providing the one or more trajectories to a modeling engine, the modeling engine further performing a further operation of performing one or more actions based on the one or more trajectories and the one or more surfaces; and performing an operation including performing the further action in response to the one or more changes to the one or more surfaces; updating the one or more trajectories based on the continuous stream of sensor events; performing one or more additional actions based on the one or more updated trajectories and the one or more changed surfaces; A method comprising:

2. The method of claim 1 , wherein the one or more changes to the one or more surfaces include one or more of a change in position, a change in orientation, a change in motion, or a deformation of the one or more surfaces.

3. The method of claim 1 , further comprising providing the continuous stream of sensor events based on one or more sensors, each sensor event including one or more of a timestamp, a time of flight, or a position value.

4. The method of claim 1 , further comprising determining one or more shapes corresponding to the one or more surfaces based on one or more characteristics of the one or more surfaces and the one or more trajectories.

5. The method of claim 1 , wherein each of the trajectories further comprises a B-spline parameterization.

6. 10. The method of claim 1, further comprising utilizing the one or more trajectories to continuously determine one or more changes to one or more of the position of the one or more surfaces, the orientation of the one or more surfaces, the deformation of the one or more surfaces, or the movement of the one or more surfaces.

7. The method of claim 1 , wherein performing the further action further comprises determining one or more objects based on portions of the one or more trajectories associated with portions of the one or more surfaces.

8. 7. The method of claim 6, wherein performing the further action further comprises determining one or more characteristics of the one or more objects based on the one or more trajectories, the one or more characteristics comprising one or more of a position, an orientation, a movement, or a deformation of the one or more objects.

9. 1. A system for perceiving surfaces and objects, comprising: a memory for storing at least instructions; one or more processors configured to execute instructions; a network computer including: The instruction: generating one or more trajectories based on a continuous stream of sensor events, each said trajectory being a parameterized representation of a one-dimensional curve segment in three-dimensional space; determining the one or more surfaces using the one or more trajectories; and providing the one or more trajectories to a modeling engine, the modeling engine further performing a further operation of performing one or more actions based on the one or more trajectories and the one or more surfaces; and performing an operation including performing the further action in response to the one or more changes to the one or more surfaces; updating the one or more trajectories based on the continuous stream of sensor events; performing one or more additional actions based on the one or more updated trajectories and the one or more changed surfaces; Including, The system further comprises: a memory for storing at least instructions; one or more processors configured to execute instructions; one or more client computers, The instructions perform actions including providing one or more portions of the sensor event.

10. The system of claim 9 , wherein the one or more changes to the one or more surfaces include one or more of a change in position, a change in orientation, a change in motion, or a deformation of the one or more surfaces.

11. One or more processors of the network computer are configured to execute instructions, the instructions comprising: performing operations further including providing a continuous stream of the sensor events based on one or more sensors; The system of claim 9 , wherein each sensor event includes one or more of a timestamp, a time of flight, or a position value.

12. 10. The system of claim 9, wherein one or more processors of the networked computer are configured to execute instructions, the instructions performing operations further including determining one or more shapes corresponding to the one or more surfaces based on one or more characteristics of the one or more surfaces and the one or more trajectories.

13. The system of claim 9 , wherein each of the trajectories further comprises a B-spline parameterization.

14. 10. The system of claim 9, wherein one or more processors of the networked computer are configured to execute instructions that perform operations further including utilizing the one or more trajectories to continuously determine one or more changes to one or more of the positions of the one or more surfaces, the orientations of the one or more surfaces, the deformations of the one or more surfaces, or the movements of the one or more surfaces.

15. The system of claim 9 , wherein performing the further action further comprises determining one or more objects based on a portion of the one or more trajectories associated with a portion of the one or more surfaces.

16. 16. The system of claim 15, wherein performing the further action further comprises determining one or more characteristics of the one or more objects based on the one or more trajectories, the one or more characteristics comprising one or more of a position, an orientation, a movement, or a deformation of the one or more objects.

17. A processor-readable non-transitory storage medium containing instructions for perceiving surfaces and objects, wherein execution of said instructions by one or more processors on one or more networked computers comprises: generating one or more trajectories based on a continuous stream of sensor events, each said trajectory being a parameterized representation of a one-dimensional curve segment in three-dimensional space; determining the one or more surfaces using the one or more trajectories; and providing the one or more trajectories to a modeling engine, the modeling engine further performing a further operation of performing one or more actions based on the one or more trajectories and the one or more surfaces; and performing an operation including performing the further action in response to the one or more changes to the one or more surfaces; updating the one or more trajectories based on the continuous stream of sensor events; performing one or more additional actions based on the one or more updated trajectories and the one or more changed surfaces; a processor-readable non-transitory storage medium,

18. The medium of claim 17 , wherein the one or more changes to the one or more surfaces include one or more of a change in position, a change in orientation, a change in motion, or a deformation of the one or more surfaces.

19. The medium of claim 17 , further comprising providing the continuous stream of sensor events based on one or more sensors, wherein each sensor event includes one or more of a timestamp, a time of flight, or a position value.

20. The medium of claim 17 , further comprising determining one or more shapes corresponding to the one or more surfaces based on one or more characteristics of the one or more surfaces and the one or more tracks.

21. The medium of claim 17 , wherein each of the trajectories further comprises a B-spline parameterization.

22. 20. The medium of claim 17, further comprising utilizing the one or more trajectories to continuously determine one or more changes to one or more of the position of the one or more surfaces, the orientation of the one or more surfaces, the deformation of the one or more surfaces, or the movement of the one or more surfaces.

23. The medium of claim 17 , wherein performing the further action further comprises determining one or more objects based on a portion of the one or more trajectories associated with a portion of the one or more surfaces.

24. 24. The medium of claim 23, wherein performing the further action further comprises determining one or more characteristics of the one or more objects based on the one or more trajectories, the one or more characteristics comprising one or more of a position, an orientation, a movement, or a deformation of the one or more objects.

25. 1. A networked computer for perceiving surfaces and objects, comprising: a memory for storing at least instructions; one or more processors configured to execute instructions; Equipped with The instruction: generating one or more trajectories based on a continuous stream of sensor events, each said trajectory being a parameterized representation of a one-dimensional curve segment in three-dimensional space; determining the one or more surfaces using the one or more trajectories; and providing the one or more trajectories to a modeling engine, the modeling engine further performing a further operation of performing one or more actions based on the one or more trajectories and the one or more surfaces; and performing an operation including performing the further action in response to the one or more changes to the one or more surfaces; updating the one or more trajectories based on the continuous stream of sensor events; performing one or more additional actions based on the one or more updated trajectories and the one or more changed surfaces; Network computers, including:

26. 26. The network computer of claim 25, wherein the one or more changes to the one or more surfaces include one or more of a change in position, a change in orientation, a change in motion, or a deformation of the one or more surfaces.

27. the one or more processors are configured to execute instructions; The instruction: performing operations further including providing a continuous stream of the sensor events based on one or more sensors; 26. The network computer of claim 25, wherein each sensor event includes one or more of a timestamp, a time of flight, or a position value.

28. the one or more processors are configured to execute instructions; 26. The network computer of claim 25, wherein the instructions perform operations further including determining one or more shapes corresponding to the one or more surfaces based on one or more characteristics of the one or more surfaces and the one or more trajectories.

29. 26. The network computer of claim 25, wherein each said trajectory further comprises a B-spline parameterization.

30. 26. The network computer of claim 25, wherein the one or more processors are configured to execute instructions that perform operations further including utilizing the one or more trajectories to continuously determine one or more changes to one or more of: position of the one or more surfaces, orientation of the one or more surfaces, deformation of the one or more surfaces, or movement of the one or more surfaces.

31. 26. The network computer of claim 25, wherein performing the further action further comprises determining one or more objects based on a portion of the one or more trajectories associated with a portion of the one or more surfaces.

32. 32. The network computer of claim 31 , wherein performing the further action further comprises determining one or more characteristics of the one or more objects based on the one or more trajectories, the one or more characteristics comprising one or more of a position, an orientation, a movement, or a deformation of the one or more objects.

Images