Perception of objects based on surface detection and detection of surface movement

Abstract

The present invention provides methods, systems, storage media, and network computers for perceiving surfaces and objects. [Solution] The method generates trajectories based on a continuous stream of sensor events such that each trajectory is a parameterized representation of a curved segment, and uses the trajectories to determine the surface. The trajectories are provided to a modeling engine, which performs one or more actions based on the trajectories and the surface. In response to surface changes, the trajectories are updated based on a continuous stream of sensor events, and one or more additional actions are performed based on the updated trajectories and the changed surface. Surface changes include changes in the position, orientation, motion, deformation, etc., of one or more surfaces. The shape corresponding to the surface is determined based on the characteristics of the surface or trajectory.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This application is a utility patent application based on U.S. Provisional Patent Application Serial No. 63 / 205,480, filed on December 14, 2020. The benefit of this filing date is claimed herein under 35 U.S.C. § 119(e), and the content of the provisional application is hereby incorporated by reference in its entirety.

[0002] (Technical Field) The present invention generally relates to machine - detection or machine - vision systems, and more particularly, but not limited to, object perception based on surface detection and detection of surface movement.

Background Art

[0003] Conventional robotic vision is mainly based on a camera where the input to the detection system is a two - dimensional (2D) array of pixels that encode the amount of light received by each pixel over an exposure time, or on depth - capture techniques (e.g., to name a few, time - of - flight (ToF) cameras, structured - light cameras, LIDAR, RADAR, or stereo cameras) that provide a three - dimensional (3D) point cloud. Here, each point in the point cloud can store the position of each point in space relative to the vision system, and can further store any of a plurality of other data (e.g., to name a few, luminance, color, relative line - of - sight velocity, spectral composition) associated with the patch of reflective material from which the point was generated. It should be noted that 3D point clouds can be represented as "frames" similar to frames of images from a camera, meaning that they do not have a fundamental representation of continuously evolving time.

[0004] To provide useful perceptual outputs that may be used by machine vision applications such as robot planning and control systems, these 2D or 3D data need to be processed by machine vision algorithms implemented in software or hardware. In some cases, some machine vision systems can utilize machine learning to determine world properties or features that may be prominent in specific robotic tasks, such as position, shape orientation, material properties, object classification, object motion, and relative motion of the robot system. Often, neither the 2D nor 3D representations used by conventional machine vision systems provide intrinsic / innate support for continuous surface representations of objects in the environment. Similarly, they often represent scenes using static data captured from sensors, and fundamental data about the scene may be filtered out before the data is provided to the machine vision algorithms for processing. Thus, conventional machine vision systems are disadvantageous because they may rely on such inaccurate or non-representative scene data to perform machine vision analysis. That is, the present invention has been made in relation to these challenges and other problems.

[0005] Embodiments of the present invention that are neither limiting nor exhaustive will be described with reference to the following drawings. In the drawings, unless otherwise indicated, the same reference numerals indicate the same elements throughout the various figures. For a better understanding of the described invention, refer to the following detailed descriptions of various embodiments, which should be read in conjunction with the accompanying drawings. [Brief explanation of the drawing]

[0006] [Figure 1] This figure shows a system environment in which various embodiments can be implemented. [Figure 2] This is a schematic diagram showing an embodiment of a client computer. [Figure 3] This is a schematic diagram showing an embodiment of a network computer. [Figure 4]This figure shows the logical architecture of a system for perceiving an object based on surface detection and surface motion detection, according to one or more of various embodiments. [Figure 5] This is a logical schematic diagram of a system that perceives an object based on the detection of its surface and the detection of its surface movement, according to one or more of its various embodiments. [Figure 6] This figure shows a sensor that perceives an object based on surface detection and surface motion detection, and a logical representation of sensor output information, according to one or more of various embodiments. [Figure 7] This figure shows a logical representation of a scanning path for perceiving an object based on surface detection and surface motion detection, according to one or more of various embodiments. [Figure 8] This figure shows a logical representation of a scanning system that perceives an object based on surface detection and surface motion detection, according to one or more of its various embodiments. [Figure 9] This figure shows a logical representation of a scanning system that perceives an object based on surface detection and surface motion detection, according to one or more of its various embodiments. [Figure 10] This figure shows a logical schematic of a system for perceiving an object based on the detection of its surface and the detection of its surface movement, according to one or more of its various embodiments. [Figure 11] This figure shows a logical schematic of a system for perceiving an object based on the detection of its surface and the detection of its surface movement, according to one or more of its various embodiments. [Figure 12] This is a schematic flowchart illustrating a process for perceiving an object based on surface detection and surface motion detection, according to one or more of various embodiments. [Figure 13] This flowchart illustrates a process for perceiving an object based on surface detection and surface movement detection, according to one or more of various embodiments. [Figure 14] This flowchart illustrates a process for perceiving an object based on surface detection and surface movement detection, according to one or more of various embodiments. [Figure 15]This flowchart illustrates a process for perceiving an object based on surface detection and surface movement detection, according to one or more of various embodiments. [Figure 16] This flowchart illustrates a process for perceiving an object based on surface detection and surface movement detection, according to one or more of various embodiments. [Figure 17] This figure shows a non-limiting use case in which an object is perceived based on the detection of its surface and the detection of its surface movement, according to one or more of various embodiments. [Modes for carrying out the invention]

[0007] Next, various embodiments will be described in more detail below with reference to the accompanying drawings, which form part of specific exemplary embodiments that can carry out the present invention and are illustrated by illustration. However, embodiments can be carried out in many different forms, and embodiments should not be construed as being limited to those embodiments shown herein. On the contrary, these embodiments are provided so as to make this disclosure detailed and complete, and will further convey the scope of embodiments to those skilled in the art. In particular, various embodiments can be methods, systems, media, or devices. Thus, various embodiments can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments that combine software and hardware aspects. Accordingly, the following detailed description should not be taken as limiting.

[0008] Throughout the specification and claims, the following terms have the meanings expressly associated herein unless otherwise explicitly indicated in this context. The expression “in one embodiment” as used herein may refer to the same embodiment, but not necessarily the same embodiment. Furthermore, the expression “in another embodiment” as used herein may refer to a different embodiment, but not necessarily a different embodiment. Thus, various embodiments can be readily combined without departing from the scope or spirit of the invention, as described below.

[0009] In addition, the term "or" as used herein is an inclusive "or" operator and is equivalent to the term "and / or" unless otherwise explicitly indicated in this context. The term "based on" is not inclusive and, unless further explicitly indicated in this context, may be based on additional factors not described. In addition, throughout this specification, the meanings of "a," "an," and "the" include multiple anaphora. The meaning of "in" includes "in" and "on."

[0010] With respect to exemplary embodiments, the following terms are also used herein in accordance with their corresponding meanings unless otherwise explicitly indicated in this context.

[0011] As used herein, the term "engine" refers to logic implemented by hardware or software instructions that can be written in programming languages ​​such as C, C++, Objective-C, COBOL, Java®, PHP, Perl, JavaScript, Ruby, VBScript, C#, and other Microsoft .NET® languages. An engine is either compiled into an executable program or written in an interpretable programming language. A software engine can be called from other engines or from itself. The engines described herein refer to one or more logical modules that can be merged into other engines or applications, or they can be divided into sub-engines. An engine can be stored in non-temporary computer-readable media or computer storage devices and further stored in one or more general-purpose computers and executed thereon, thus creating a dedicated computer configured to provide the engine.

[0012] As used herein, the term “scanning signal generator” refers to a system or device capable of generating a beam that can be scanned / directed for projection into an environment. For example, a scanning signal generator may be a high-speed laser-based scanning device based on a dual-axis microelectromechanical system (MEMS) configured to scan a laser over a defined area of ​​interest. The characteristics of a scanning signal generator can vary depending on the application or service environment. A scanning signal generator is not strictly limited to a laser or laser MEMS, and other types of beam signal generators may be used depending on the situation. Critical selection criteria for the characteristics of a scanning signal generator may include beam width, beam spectroscopy, beam energy, wavelength, and phase. These can be selected so that the scanning signal generator allows for sufficiently precise energy reflection from the scanning surface or object in the scanning environment of interest. A scanning signal generator can be designed to scan frequencies up to 10 skHz. A scanning signal generator can be controlled in a closed-loop manner by one or more processors that can provide feedback on the object in the environment, instructing the scanning signal generator to modify its amplitude, frequency, phase, etc.

[0013] As used herein, the term “sensor” refers to a device or system capable of detecting reflected energy from a scanning signal generator. A sensor can be thought of as comprising an array of detector cells that respond to energy reflected from the scanning signal generator. A sensor can provide an output indicating which detector cell is triggered and for what time. A sensor can be thought of as producing a sensor output that reports the cell location and detection time for individual cells, rather than being limited to reporting the state or status of any cell. For example, a sensor may include an event sensor camera, a SPAD array, a SiPM array, and the like.

[0014] As used herein, the terms "trajectory" and "surface trajectory" refer to one or more data structures that store or represent a parametric representation of a curve segment corresponding to a surface detected by one or more sensors. A trajectory can include one or more attributes / elements corresponding to constants or coefficients of a segment of a one-dimensional analysis curve in three-dimensional space. The trajectory of a surface is determined based on fitting or associating one or more sensor events to a known analysis curve. Sensor events that conflict with the analysis curve may be considered noise or otherwise excluded from the trajectory.

[0015] As used herein, the term "configuration information" refers to information that can include rule-based policies, pattern matching, scripts (e.g., computer-readable instructions), etc., provided from various sources, including configuration files, databases, user inputs, built-in defaults, plugins, extensions, etc., or combinations thereof.

[0016] The following briefly describes embodiments of the present invention to provide a basic understanding of some aspects of the present invention. This brief description is not intended to be a broad overview. It is not intended to identify key or important elements or to delineate or otherwise narrow the scope. Its purpose is merely to present some concepts in a simplified form as a prelude to the more detailed description that follows.

[0017] Briefly stated, various embodiments are directed to the recognition of surfaces and objects. In one or more of the various embodiments, one or more trajectories can be generated based on a continuous stream of sensor events such that each trajectory is a parametric representation of a one-dimensional curve segment in three-dimensional space.

[0018] In one or more of the various embodiments, one or more trajectories can be utilized to determine one or more surfaces.

[0019] In one or more of the various embodiments, one or more orbitals can be provided to a modeling engine to perform one or more operations based on one or more orbitals and one or more surfaces.

[0020] In one or more of the various embodiments, in response to one or more changes on one or more surfaces, further actions can be performed, including updating one or more trajectories based on a continuous stream of sensor events, and performing one or more additional actions based on one or more updated trajectories and one or more changed surfaces.

[0021] In one or more of the various embodiments, one or more changes to one or more surfaces may include one or more changes in the position, orientation, movement, deformation, etc., of the one or more surfaces.

[0022] In one or more of the various embodiments, a continuous stream of sensor events can be provided based on one or more sensors such that each sensor event includes one or more of the following: a timestamp, time of flight, or position value.

[0023] In one or more of the various embodiments, one or more shapes corresponding to one or more surfaces can be determined based on one or more characteristics of the one or more surfaces and one or more orbitals.

[0024] In one or more of the various embodiments, each orbit may further include a parameterized representation of a B-spline.

[0025] In one or more of the various embodiments, one or more trajectories can be used to continuously determine one or more changes to one or more surfaces, such as the position of one or more surfaces, the orientation of one or more surfaces, the deformation of one or more surfaces, or the movement of one or more surfaces.

[0026] In one or more of the various embodiments, the modeling engine may be configured to perform further operations including determining one or more objects based on one or more segments of trajectories that can be associated with one or more segments of surfaces.

[0027] In one or more of the various embodiments, the modeling engine may be configured to perform further operations including determining one or more features of one or more objects based on one or more trajectories such that one or more features include one or more of the position, orientation, movement, or deformation of one or more objects.

[0028] Example operating environment Figure 1 shows the components of one embodiment of an environment in which embodiments of the present invention can be carried out. Not all components are necessary to carry out the present invention, and variations in the arrangement and type of components can be made without departing from the spirit or scope of the present invention. As shown in the figure, the system 100 in Figure 1 includes a local area network (LAN) / wide area network (WAN)-(network) 110, a wireless network 108, client computers 102-105, an application server computer 116, a detection system 118, and the like.

[0029] At least one embodiment of client computers 102-105 is described in detail below with reference to Figure 2. In one embodiment, at least a portion of client computers 102-105 can operate through one or more wired or wireless networks, such as network 108 or 110. Generally, client computers 102-105 can virtually include any computer that can communicate through the network for various online activities, offline operations, etc., such as sending and receiving information. In one embodiment, one or more of client computers 102-105 can operate within a business or other entity and be configured to perform a wide variety of services for the business or other entity. For example, client computers 102-105 can be configured to operate as a web server, firewall, client application, media player, mobile phone, game console, desktop computer, etc. However, client computers 102-105 are not limited to these services and can also be used for end-user computing in other embodiments, for example. It should be understood that a large or small number of client computers (as shown in Figure 1) may be included in the system as described herein, and that the embodiments are therefore not limited by the number or type of client computers used.

[0030] A computer that can operate as client computer 102 may include personal computers, multiprocessor systems, microprocessor-based or programmable electronic devices, and computers that are typically connected using wired or wireless communication media such as network PCs. In some embodiments, client computers 102-105 may virtually include any portable computer that can connect to and receive information from another computer, such as a laptop computer 103, a mobile computer 104, or a tablet computer 105. However, portable computers are not limited to these and may also include other portable computers such as cellular phones, display pagers, radio frequency (RF) devices, infrared (IR) devices, personal digital assistants (PDAs), handheld computers, wearable computers, and integrated devices that combine one or more of the aforementioned computers. Thus, client computers 102-105 generally have a wide range of functions and features. Furthermore, client computers 102-105 may also be able to access a variety of computer applications, including browsers or other web-based applications.

[0031] A web-enabled client computer may include a browser application configured to send requests and receive responses over the web. The browser application can be configured to receive and display graphics, text, multimedia, etc., virtually utilizing any web-based language. In one embodiment, the browser application may display and send messages using JavaScript, Hypertext Markup Language (HTML), Extensible Markup Language (XML), JavaScript Object Notation (JSON), Cascading Style Sheets (CSS), or a combination thereof. In one embodiment, a user of the client computer may utilize the browser application to perform various activities over the network (online). However, they may also use a different application to perform various online activities.

[0032] Client computers 102-105 may also include at least one other client application configured to receive or send content to or from another computer. A client application may include the ability to send or receive content, etc. A client application may further provide information that identifies itself, including type, function, name, etc. In one embodiment, client computers 102-105 may uniquely identify itself through any of a wide variety of means, including an Internet Protocol (IP) address, telephone number, mobile identification number (MIN), electronic serial number (ESN), client certificate, or other device identifier. Such information may be provided in one or more network packets transmitted between other client computers, application server computer 116, discovery system 118, or other computers.

[0033] Client computers 102-105 can further be configured to include a client application that allows end users to log in to end user accounts that can be managed by other computers, such as the application server computer 116 and the discovery system 118. Such end user accounts can be configured in one, not limited to one, embodiment to allow end users to manage one or more online activities, including project management, software development, system administration, configuration management, search activities, and social networking activities, browse various websites, and communicate with other users. The client computers can also be configured to allow users to view reports, interactive user interfaces, or results provided by the discovery system 118.

[0034] Wireless network 108 is configured to connect client computers 103-105 and components of the client computers to network 110. Wireless network 108 includes any of a wide variety of wireless subnetworks that can be further overlaid on standalone ad-hoc networks, etc., to provide infrastructure-oriented connectivity for client computers 103-105. Such subnetworks may include mesh networks, wireless LAN (WLAN) networks, cellular networks, etc. In one embodiment, the system of the present invention may include more than one wireless network.

[0035] The wireless network 108 may further include autonomous systems such as terminals, gateways, and routers connected by wireless radio links or the like. These connectors can be configured to move freely and randomly and to organize themselves arbitrarily, thus allowing for rapid changes to the topology of the wireless network 108.

[0036] The wireless network 108 can further utilize multiple access technologies, including 2G, 3G, 4G, and 5G generation radio access for cellular systems, WLAN, and wireless router (WR) mesh. Access technologies such as 2G, 3G, 4G, 5G, and future access networks can enable wide reception ranges for mobile computers such as client computers 103-105 with varying degrees of mobility. In one, non-limiting embodiment, the wireless network 108 can enable wireless connectivity via wireless network access such as Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wideband Code Division Multiple Access (WCDMA), High-Speed ​​Downlink Packet Access (HSDPA), and Long Term Evolution (LTE). Basically, the wireless network 108 can virtually include any wireless communication means that can move information between client computers 103-105 and other computers, networks, cloud-based networks, cloud instances, etc.

[0037] Network 110 is configured to connect network computers to other computers, including application server computer 116, discovery system 118, client computer 102, and client computers 103-105, via a wireless network 108 or the like. Network 110 can utilize any form of computer-readable medium for transmitting information from one electronic device to another. Network 110 may also include the Internet, wide area network (WAN), and direct connections in addition to local area networks (LANs) via Universal Serial Bus (USB) ports, Ethernet ports, other forms of computer-readable medium, or any combination thereof. In a set of LAN interconnections, including those based on different architectures and protocols, routers act as links between LANs, enabling the transmission of messages from one to another. In addition, communication links within a LAN generally include twisted-pair wires or coaxial cables, and communication links between networks can utilize other carrier means, including analog telephone lines, full or fractional dedicated digital lines including T1, T2, T3, and T4, or wireless links including, for example, E-carriers, Integrated Digital Networks (ISDN), Digital Subscriber Lines (DSL), satellite links, or other communication links known to those skilled in the art. Furthermore, communication links can utilize any of the wide variety of digital signaling technologies, including, for example, DS-0, DS-1, DS-2, DS-3, DS-4, OC-3, OC-12, OC-48, etc. Furthermore, remote computers and other related electronic devices can be remotely connected to either the LAN or WAN via modems and temporary telephone links. In one embodiment, network 110 can be configured to transmit Internet Protocol (IP) information.

[0038] In addition, communication media generally include any non-temporary or temporary information distribution medium that implements computer-readable instructions, data structures, program modules, or other means of transfer. Examples include wired media such as twisted pair, coaxial cable, optical fiber, waveguide, and other wired media, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0039] Furthermore, one embodiment of the application server computer 116 or the detection system 118 will be described in detail below with reference to Figure 3. Figure 1 shows the application server computer 116 and the detection system 118 as single computers, but the invention or embodiments are not limited thereto. For example, one or more functions of the application server computer 116, the detection system 118, etc., can be distributed across one or more separate network computers. Furthermore, in one or more embodiments, the detection system 118 can be implemented using multiple network computers. Moreover, in one or more of the various embodiments, the application server computer 116, the detection system 118, etc., can be implemented using one or more cloud instances in one or more cloud networks. Therefore, these inventions and embodiments should not be interpreted as being limited to a single environment and other configurations, and other architectures are also envisioned.

[0040] Example client computer Figure 2 shows one embodiment of the client computer 200 that may include more or fewer components than those shown. The client computer 200 may represent, for example, one or more embodiments of the mobile computer or client computer shown in Figure 1.

[0041] The client computer 200 may include a processor 202 that communicates with memory 204 via bus 228. The client computer 200 may also include a power supply 230, a network interface 232, an audio interface 256, a display 250, a keypad 252, an illuminator 254, a video interface 242, an input / output interface 238, a haptic interface 264, a Global Positioning System (GPS) receiver 258, an open-air gesture interface 260, a temperature interface 262, a camera 240, a projector 246, a pointing device interface 266, a processor-readable fixed storage device 234, and a processor-readable removable storage device 236. The client computer 200 may optionally communicate directly with a base station (not shown) or another computer. In one further embodiment, although not shown, a gyroscope may be used in the client computer 200 to measure or maintain the orientation of the client computer 200.

[0042] Power supply 230 can provide power to client computer 200. Power can be provided using a rechargeable or non-rechargeable battery. Power can also be provided by an external power source such as an AC adapter or power docking cradle that supplements or recharges the battery.

[0043] The network interface 232 includes circuitry connecting client computers 200 to one or more networks and is further configured to be used with one or more communication protocols and technologies, including, but not limited to, protocols and technologies that implement any part of the OSI model for mobile communications (GSM), CDMA, Time Division Multiple Access (TDMA), UDP, TCP / IP, SMS, MMS, GPRS, WAP, UWB, WiMAX, SIP / RTP, GPRS, EDGE, WCDMA, LTE, UMTS, OFDM, CDMA2000, EV-DO, HSDPA, or any of a wide variety of other wireless communication protocols. The network interface 232 may also be known as a transceiver, transceiver device, or network interface card (NIC).

[0044] The voice interface 256 can be configured to generate and receive audio signals, such as the sound of a human voice. For example, the voice interface 256 can be coupled to a speaker and microphone (not shown) to enable telecommunication with another person or to generate an audio acknowledgment for a certain action. The microphone of the voice interface 256 can also be used for input to or control of a client computer 200, for example, for use in speech recognition or sound-based touch detection.

[0045] The display 250 may be a liquid crystal display (LCD), gas plasma, electronic ink, light-emitting diode (LED), organic LED (OLED), or any other type of light-reflecting or light-transmitting display that can be used with a computer. The display 250 may also include a touch interface 244 configured to receive input from an object such as a stylus or a human finger, and may further detect touches or gestures using resistive, capacitive, surface acoustic wave (SAW), infrared, radar, or other technologies.

[0046] The projector 246 can be a remote handheld projector or an integrated projector capable of projecting images onto any other reflective surface, such as a distant wall or remote screen.

[0047] The video interface 242 can be configured to capture still images, video segments, infrared video, and other video images. For example, the video interface 242 can be coupled to a digital video camera, a webcam, and the like. The video interface 242 may include a lens, an image sensor, and other electronic components. The image sensor may include a complementary metal-oxide-semiconductor (CMOS) integrated circuit, a charge-coupled device (CCD), or any other integrated circuit for detecting light.

[0048] The keypad 252 may include any input device configured to receive input from the user. For example, the keypad 252 may include a push-button numeric dial or a keyboard. The keypad 252 may also include command buttons associated with selecting and sending images.

[0049] The illuminator 254 can provide status indications or light. The illuminator 254 can remain active for a specific period or in response to an event message. For example, when the illuminator 254 is active, it can backlight the buttons on the keypad 252 and remain on while the client computer is powered. The illuminator 254 can also backlight these buttons in various patterns when certain actions are performed, such as dialing another client computer. The illuminator 254 can also illuminate a light source located within the transparent or translucent case of the client computer in response to its operation.

[0050] Furthermore, the client computer 200 may also include a hardware security module (HSM) 268 that provides additional tamper-proof safeguards for generating, storing, or using security / cryptographic information such as keys, digital certificates, passwords, passphrases, and two-factor authentication information. In some embodiments, the hardware security module can be used to support one or more standard public key infrastructures (PKIs) and can further be used to generate, manage, or store key pairs, etc. In some embodiments, the HSM 268 may be a standalone computer, while in other embodiments, the HSM 268 may be configured as a hardware card that can be added to the client computer.

[0051] The client computer 200 may also include an input / output interface 238 for communicating with external peripheral devices or other computers, such as other client computers and network computers. Peripheral devices may include audio headsets, virtual reality headsets, display screen glass, remote speaker systems, and remote speaker and microphone systems. The input / output interface 238 may utilize one or more technologies, such as Universal Serial Bus (USB), infrared, WiFi, WiMAX, and Bluetooth®.

[0052] The input / output interface 238 may also include one or more sensors for determining geolocation information (e.g., GPS), monitoring power status (e.g., voltage sensor, current sensor, frequency sensor, etc.), and monitoring weather (e.g., thermostat, barometer, anemometer, humidity detector, precipitation scale, etc.). The sensors may be one or more hardware sensors that collect or measure data outside of the client computer 200.

[0053] The haptic interface 264 can be configured to provide tactile feedback to the user of the client computer. For example, the haptic interface 264 can be used to vibrate the client computer 200 in a specific way when another user of the computer is calling it. The temperature interface 262 can be used to provide the user of the client computer 200 with a temperature measurement input or a temperature change output. The open-air gesture interface 260 can detect the physical gestures of the user of the client computer 200, for example, by using a single or stereo video camera, radar, or gyroscope sensor inside the computer that is held or worn by the user. The camera 240 can be used to track the physical eye movements of the user of the client computer 200.

[0054] The GPS transceiver 258 can determine the physical coordinates of the client computer 200 on the Earth's surface, typically outputting the position as latitude and longitude values. The GPS transceiver 258 can also further determine the physical location of the client computer 200 on the Earth's surface using other Earth positioning methods, including, but not limited to, triangulation, assisted GPS (AGPS), extended observation time difference (E-OTD), cell identifier (CI), service area identifier (SAI), extended timing advance (ETA), and base station subsystem (BSS). It should be understood that the GPS transceiver 258 can determine the physical location of the client computer 200 under various circumstances. However, in one or more embodiments, the client computer 200 may provide other information, via other components, that can be used to determine the physical location of the client computer, such as a media access control (MAC) address or IP address.

[0055] In at least one of the various embodiments, applications such as the operating system 206, other client applications 224, and a web browser 226 can be configured to utilize geolocation information to select one or more localization features, such as time zone, language, currency, and calendar formatting. These localization features can be used in the file system, user interface, reports, and internal processing or databases. In at least one of the various embodiments, the geolocation information used to select the localization information can be provided by GPS 258. In some embodiments, the geolocation information may also include information provided using one or more geolocation protocols over a network, such as wireless network 108 or network 111.

[0056] Human interface components can be peripheral devices physically separated from the client computer 200, enabling remote input or output to the client computer 200. For example, information routed as described herein via a human interface component such as a display 250 or a keyboard 252 can instead be routed via a network interface 232 to a suitable remotely located human interface component. Examples of human interface peripheral components that can be remote include, but are not limited to, voice devices, pointing devices, keypads, displays, cameras, and projectors. These peripheral components can communicate via pico networks such as Bluetooth® and Zigbee®. One non-limited embodiment of a client computer with such peripheral human interface components is a wearable computer that includes a remote pico projector with one or more cameras communicating remotely with a separately located client computer, and can detect user gestures directed towards a portion of the image projected by the pico projector onto a reflective surface such as a wall or the user's hand.

[0057] A client computer may include a web browser application 226 configured to receive and send web pages, web-based messages, graphics, text, multimedia, etc. The browser application on the client computer may virtually utilize any programming language, including Wireless Application Protocol Messages (WAP). In one or more embodiments, the browser application may utilize Handheld Device Markup Language (HDML), Wireless Markup Language (WML), WMLScript, JavaScript, Standard Generalized Markup Language (SGML), Hypertext Markup Language (HTML), Extensible Markup Language (XML), HTML5, etc.

[0058] Memory 204 may include RAM, ROM, or other types of memory. Memory 204 is an example of a computer-readable storage medium (device) for storing information such as computer-readable instructions, data structures, program modules, or other data. Memory 204 may store the BIOS 208, which controls the low-level operation of the client computer 200. Memory may also store the operating system 206, which controls the operation of the client computer 200. It will be understood that this component may include a general-purpose operating system such as one version of UNIX or Linux®, or a dedicated client computer communications operating system such as WindowsPhone® or Symbian®. The operating system may include, or be connected to, a Java virtual machine module that enables control of hardware components or the operation of the operating system via a Java application program.

[0059] The memory 204 may further include one or more data storages 210 that can be used by the client computer 200 to store applications 220 or other data. For example, the data storage 210 may also be used to store information describing various functions of the client computer 200. This information can then be provided to another device or computer based on any of a wide variety of methods, including being sent as part of a header during communication, being sent on request, etc. The data storage 210 may also be used to store social networking information, including address books, buddy lists, aliases, user profile information, etc. The data storage 210 may further include program code, data, algorithms, etc., used by a processor, such as the processor 202, to perform and execute operations. In one embodiment, at least a portion of the data storage 210 may also be stored in another component of the client computer 200, or outside the client computer, including, but not limited to, a non-temporary processor-readable removable storage device 236, a processor-readable fixed storage device 234.

[0060] Application 220 may include computer-executable instructions that, when executed by a client computer 200, send, receive, or otherwise process commands and data. Application 220 may include, for example, other client applications 224, a web browser 226, etc. The client computer may be configured to exchange communications with the application server or network monitoring computer, such as queries, searches, messages, notification messages, event messages, sensor events, alerts, performance metrics, log data, API calls, or a combination thereof.

[0061] Other examples of application programs include calendars, search programs, email client applications, IM applications, SMS applications, Voice over Internet Protocol (VOIP) applications, contact managers, task managers, transcoders, database programs, word processing programs, security applications, spreadsheet programs, games, and search programs.

[0062] Additionally, in one or more embodiments (not shown), the client computer 200 may include, instead of a CPU, an embedded logic hardware device such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable array logic (PAL), or a combination thereof. The embedded logic hardware device can directly implement the embedded logic and perform operations. Also, in one or more embodiments (not shown), the client computer 200 may include one or more hardware microcontrollers instead of a CPU. In one or more embodiments, one or more microcontrollers can directly implement the microcontroller-specific embedded logic and perform operations, and can also access microcontroller-specific internal memory and specific external input and output interfaces (e.g., hardware pins or wireless transceivers) to perform operations such as a system-on-a-chip (SOC).

[0063] Example network computer Figure 3 shows one embodiment of a network computer 300 that may be included in a system implementing one or more of the various embodiments. The network computer 300 may include more or fewer components than those shown in Figure 3. However, the illustrated components are sufficient to disclose exemplary embodiments for implementing these inventions. The network computer 300 may represent, for example, one embodiment of the application server computer 116 or the detection system 118 in Figure 1.

[0064] A network computer, such as network computer 300, may include a processor 302 that can communicate with memory 304 via bus 328. In some embodiments, the processor 302 may consist of one or more hardware processors or one or more processor cores. In some cases, one or more of the one or more processors may be dedicated processors designed to perform one or more special operations, such as those described herein. Network computer 300 also includes a power supply 330, a network interface 332, an audio interface 356, a display 350, a keyboard 352, an input / output interface 338, a processor-readable fixed storage device 334, and a processor-readable removable storage device 336. The power supply 330 provides power to network computer 300.

[0065] Network interface 332 is configured to be used by one or more communication protocols and technologies, including, but not limited to, protocols and technologies that implement any part of any of the following: Open System Interconnection Model (OSI model), Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), User Datagram Protocol (UDP), Transmit Control Protocol / Internet Protocol (TCP / IP), Short Message Service (SMS), Multimedia Messaging Service (MMS), General Packet Radio Service (GPRS), WAP, Ultra Wideband (UWB), IEEE 802.16 Worldwide Interoperability for Microwave Access (WiMAX), Session Initiation Protocol / Real-Time Transfer Protocol (SIP / RTP), or a wide variety of other wired and wireless communication protocols. Network interface 332 may be known as a transceiver, transceiver device, or network interface card (NIC). The network computer 300 can optionally communicate with a base station (not shown) or communicate directly with another computer.

[0066] The voice interface 356 is configured to generate and receive audio signals, such as the sound of a human voice. For example, the voice interface 356 can be coupled to a speaker and microphone (not shown) to enable telecommunication with another person or to generate an audio acknowledgment of a certain action. The microphone of the voice interface 356 can also be used, for example, for input to or control of a network computer 300 using speech recognition.

[0067] The display 350 can be a liquid crystal display (LCD), gas plasma, electronic ink, light-emitting diode (LED), organic LED (OLED), or any other type of light-reflecting or light-transmitting display that can be used with a computer. In some embodiments, the display 350 can be a handheld projector or pico projector that can project an image onto a wall or other object.

[0068] The network computer 300 may also include an input / output interface 338 for communicating with external devices or computers not shown in Figure 3. The input / output interface 338 can utilize one or more wired or wireless communication technologies such as USB®, Firewire®, WiFi, WiMAX, Thunderbolt®, infrared, Bluetooth®, Zigbee®, serial port, or parallel port.

[0069] The input / output interface 338 may also include one or more sensors for determining geolocation information (e.g., GPS), monitoring power status (e.g., voltage sensor, current sensor, frequency sensor, etc.), monitoring weather (e.g., thermostat, barometer, anemometer, humidity detector, precipitation scale, etc.). The sensors may be one or more hardware sensors that collect or measure data outside of the network computer 300. Human interface components may be physically separated from the network computer 300 and enable remote input or output to the network computer 300. For example, information routed as described herein via a human interface component such as a display 350 or a keyboard 352 may instead be routed via the network interface 332 to an appropriate human interface component located anywhere on the network. Human interface components include any component that enables the computer to receive input or send output from the computer's human user. Thus, pointing devices such as a mouse, stylus, or trackball can communicate and receive user input via the pointing device interface 358.

[0070] The GPS transceiver 340 can determine the physical coordinates of the network computer 300 on the ground, which typically outputs its position as latitude and longitude values. The GPS transceiver 340 can also further determine the physical location of the network computer 300 on the ground by utilizing other geopositioning means, including, but not limited to, triangulation, assisted GPS (AGPS), extended observation time difference (E-OTD), cell identifier (CI), service area identifier (SAI), extended timing advance (ETA), and base station subsystem (BSS). It should be understood that the GPS transceiver 340 can determine the physical location of the network computer 300 under various circumstances. However, in one or more embodiments, the network computer 300 may provide other information, via other components, that can be used to determine the physical location of client computers, such as, for example, a media access control (MAC) address or an IP address.

[0071] In at least one of the various embodiments, applications such as the operating system 306, the detection engine 322, the modeling engine 324, and the web service 329 can be configured to utilize geolocation information to select one or more localization features, such as time zone, language, currency, currency formatting, and calendar formatting. These localization features can be used in the file system, user interface, reports, and internal processing or databases. In at least one of the various embodiments, the geolocation information used to select the localization information can be provided by the GPS 340. Furthermore, in some embodiments, the geolocation information may include information provided using one or more geolocation protocols over a network, such as the wireless network 108 or network 111.

[0072] Memory 304 may include random access memory (RAM), read-only memory (ROM), or other types of memory. Memory 304 illustrates an embodiment of a computer-readable storage medium (device) for storing information such as computer-readable instructions, data structures, program modules, or other data. Memory 304 stores a basic input / output system (BIOS) 308 that controls the low-level operation of the network computer 300. Memory also stores an operating system 306 that controls the operation of the network computer 300. It will be understood that this component may include a general-purpose operating system such as UNIX® or a version of Linux®, or a dedicated operating system such as Microsoft's Windows® operating system or Apple's macOS® operating system. The operating system may include or be connected to one or more virtual machine modules, such as a Java virtual machine module that enables control of hardware components or the operation of the operating system via a Java application program. Similarly, it may also include other runtime environments.

[0073] The memory 304 may further include one or more data storages 310 available to the network computer 300 for storing applications 320 or other data. For example, the data storage 310 may also be used to store information describing various functions of the network computer 300. This information can then be provided to another device or computer based on any of a wide variety of methods, including being sent as part of a header during communication, being sent on request, etc. The data storage 310 may also be used to store social networking information, including address books, buddy lists, aliases, user profile information, etc. The data storage 310 may further include program code, data, algorithms, etc., used by a processor such as the processor 302, and may perform and execute operations such as those described below. In one embodiment, at least a portion of the data storage 310 may be stored in another component of the network computer 300, including, but not limited to, a non-temporary medium inside a processor-readable removable storage device 336, a processor-readable fixed storage device 334, or any other computer-readable storage device within the network computer 300, or outside the network computer 300. The data storage 310 may include, for example, an evaluation module 314.

[0074] Application 320, when executed by network computer 300, includes computer executable instructions to send, receive, or otherwise process messages (e.g., SMS, Multimedia Messaging Service (MMS), Instant Message (IM), email, or other messages), voice, and video, and can further enable telecommunication with another user on another mobile computer. Other embodiments of the application program include calendars, search programs, email client applications, IM applications, SMS applications, Voice over Internet Protocol (VOIP) applications, contact managers, task managers, transcoders, database programs, word processing programs, security applications, spreadsheet programs, games, and search programs. Application 320 may include a detection engine 322, a modeling engine 324, a web service 329, etc., which can be configured to perform the operations of the embodiments described below. In one or more of the various embodiments, one or more of the applications may be implemented as modules or components of another application. Furthermore, in one or more of the various embodiments, the applications may be implemented as extensions, modules, plug-ins, etc., of an operating system.

[0075] Furthermore, in one or more of the various embodiments, the detection engine 322, the modeling engine 324, the web service 329, etc., can operate in a cloud-based computing environment. In one or more of the various embodiments, these applications and others, including the management platform, can run within a virtual machine or virtual server that can be managed in a cloud-based computing environment. In one or more of the various embodiments, the applications in this context can flow from one physical network computer to another in the cloud-based environment depending on performance and scaling issues that are automatically managed by the cloud computing environment. Similarly, in one or more of the various embodiments, a virtual machine or virtual server dedicated to the detection engine 322, the modeling engine 324, the web service 329, etc., can be automatically deployed and deactivated.

[0076] Furthermore, in one or more of the various embodiments, the detection engine 322, the modeling engine 324, the web service 329, etc., can be located on virtual servers running in a cloud-based computing environment, rather than being tied to one or more specific physical network computers.

[0077] Furthermore, the network computer 300 may also include a hardware security module (HSM) 360 that provides additional tamper-proof safeguards for generating, storing, or using security / cryptographic information such as keys, digital certificates, passwords, passphrases, and two-factor authentication information. In some embodiments, the hardware security module can be used to support one or more standard public key infrastructures (PKIs), and can also be used to generate, manage, or store key pairs, etc. In some embodiments, the HSM 360 may be a standalone network computer, and in other cases, the HSM 360 may be configured as hardware that can be installed in the network computer.

[0078] Additionally, in one or more embodiments (not shown), the network computer 300 may include an embedded logic hardware device in place of the CPU, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable array logic (PAL), or a combination thereof. The embedded logic hardware device can directly execute the logic embedded to perform the operation. Also, in one or more embodiments (not shown), the network computer may include one or more hardware microcontrollers in place of the CPU. In one or more embodiments, one or more microcontrollers can directly execute embedded logic specific to the microcontroller to perform the operation, and can also access microcontroller-specific internal memory and specific external input and output interfaces (e.g., hardware pins or wireless transceivers) to perform operations such as a system-on-a-chip (SOC).

[0079] Example of a logical system architecture Figure 4 shows the logical architecture of a system 400 that perceives an object based on surface detection and surface motion detection, according to one or more of the various embodiments.

[0080] In this embodiment, for some embodiments, the detection system, such as system 400, may include one or more servers, such as detection server 402. In some embodiments, the detection server may be configured to include one or more detection engines, such as detection engine 404, one or more modeling engines, such as modeling engine 404, and one or more detection engines, such as detection engine 406.

[0081] Furthermore, in some embodiments, the detection system may include one or more signal generators capable of generating sensor information based on the location where energy from a signal generator is reflected from a surface. In this embodiment, in some embodiments, the signal generator 408 can be considered as a laser scanning system. Moreover, in some embodiments, the detection system may include one or more sensors capable of receiving reflected signal energy. In this embodiment, in some embodiments, the sensors can be considered as sensors that can be configured to generate sensor information corresponding to the reflected signal energy. In this embodiment, sensors such as sensor 410, sensor 412, and sensor 414 can be considered as CCDs that provide two-dimensional (2D) sensor information based on a CCD cell that detects the reflected signal energy.

[0082] Therefore, in some embodiments, 2D sensor information from each sensor can be provided to a detection engine such as a detection engine 404. In some embodiments, the detection engine can be configured to synthesize the 2D points provided by the sensors into 3D points based on triangulation or the like.

[0083] Furthermore, in some embodiments, the detection engine can be configured to direct the signal generator (e.g., scanning laser 408) to follow a specific pattern based on one or more path functions. Thus, in some embodiments, the signal generator can scan the subject area using known and precise paths that can be defined or described using one or more functions corresponding to the scanning curve / path.

[0084] Therefore, in some embodiments, the detection engine can be configured to synthesize information about an object or surface being scanned by a signal generated based on 3D sensor information provided by the sensor and a known scanning curve pattern.

[0085] In some embodiments, the scanning signal generator 408 can be implemented using one or more high-speed laser scanning devices, such as a dual-axis MEM mirror, to scan the laser beam. In some embodiments, the laser wavelength can be a wide range from UV to IR. In some embodiments, the scanning signal generator can be designed to scan frequencies up to 10 skHz. In some embodiments, the scanning signal generator can be controlled in a closed-loop manner using one or more processors that can provide feedback about objects in the environment and further instruct the scanning signal generator to adapt to one or more of the following: amplitude, frequency, phase, etc. In some cases, in some embodiments, the scanning signal generator can be configured to periodically switch on and off if the scanner may decelerate before changing direction or reversing direction.

[0086] In some embodiments, the system 400 may include two or more sensors, such as sensor 410, sensor 412, and sensor 414. In some embodiments, the sensors may include an array of pixels or cells that respond to reflected signal energy. In some embodiments, the sensors may be configured such that some or all of the sensors share a portion of the field of view with each other and with the scanning signal generator. Furthermore, in some embodiments, the relative position and pose of each sensor may be known. Also, in some embodiments, each sensor utilizes a synchronization clock. For example, in some embodiments, the sensors can be synchronized in time by using the clock of one sensor as a master clock or by using an external source that periodically sends a synchronization signal to the sensors. Alternatively, in some embodiments, the sensors may be configured to provide sensor events to the detection engine independently of or synchronously with each other.

[0087] Therefore, as the beam from the scanning signal generator scans the entire scene, the sensor triggers events in cells / pixels based on receiving reflected signal energy (e.g., photons / light from the laser) and further observing the physical reflections of the scene. Thus, in some embodiments, each sensor event (e.g., sensor event) can be determined based on the cell's position and timestamp, which are based on where and when the reflected energy is detected by each sensor. Thus, in some embodiments, each sensor independently reports each sensor event when detected, rather than collecting information / signals from the entire sensor array before providing sensor events. This behavior can be considered distinct from many conventional pixel arrays or CCDs that can "raster scan" the entire array of cells before outputting signal data. In contrast, sensors such as sensor 410, sensor 412, and sensor 414 can report signals (if any) instantly and continuously from individual cells. Thus, the cells of individual sensors report their own unique detected events rather than sharing a combined exposure time. Therefore, in some embodiments, sensors such as sensor 410, sensor 412, and sensor 414 can be based on an event sensor camera, SPAD, SiPM array, etc.

[0088] Figure 5 shows a logical schematic diagram of a system 500 that perceives an object based on surface detection and detection of surface motion, according to one or more of various embodiments. In some embodiments, a detection engine, such as a detection engine 502, can be configured to provide a sensor output representing sensor information such as position and timing. As described above, in some embodiments, a signal generator, such as a scanning laser, can scan an area of ​​interest so that reflected energy can be collected by the sensor. Thus, in some embodiments, information from each sensor can be provided to the detection engine 502.

[0089] Furthermore, in some embodiments, the detection engine 502 can be provided with a scanning path corresponding to the scanning path of the scanning signal generator. Thus, in some embodiments, the detection engine 502 can utilize the scanning path to determine the path that the scanning signal generator will traverse to scan the area of ​​interest.

[0090] Accordingly, in some embodiments, the detection engine 502 can be configured to generate sensor events corresponding to three-dimensional surface positions based on the sensor output. For example, if three sensors are present, the detection engine can use triangulation to computer-calculate the position in the area of ​​interest to which the scanning signal energy is reflected. Those skilled in the art will understand that triangulation or other similar techniques can be applied to determine the scanning position when the sensor positions are known.

[0091] In some embodiments, the scanning signal generator (e.g., a high-speed scanning laser) can be configured to execute a precise scanning pattern. Therefore, in some embodiments, the detection engine 502 can be provided with a specific scanning path function. Furthermore, in some embodiments, the detection engine 502 can be configured to determine a specific scanning path based on configuration information to take into account local conditions of local requirements.

[0092] In one or more of the various embodiments, a detection engine, such as the detection engine 502, can generate a sequence of surface trajectories that can be based on sensor information synthesized from the scanning path and the sensor output 504.

[0093] Figure 6 shows a sensor that perceives an object based on surface detection and surface motion detection, and a logical representation of sensor output information, according to one or more of various embodiments.

[0094] In one or more of the various embodiments, the detection engine can be provided with sensor outputs from various sensors. In some embodiments, specific sensor characteristics can be modified depending on a particular application that can be directed to perceiving an object based on the detection of a surface and the detection of surface motion. In this embodiment, for some embodiments, sensor 602A can be thought of as representing a general sensor that can generate a signal corresponding to a precise position on the sensor that can detect reflected energy from a scanning signal generator. For example, sensor 602A can be thought of as an array of detector cells that report the cell position where the energy reflected from the scanning signal generator was detected. In this embodiment, the horizontal position 604 and the vertical position 606 can be thought of as representing positions corresponding to the position on sensor 602 where the reflected signal energy was detected.

[0095] In one or more of the various embodiments, the detection engine can be configured to receive sensor information for one or more detection events from one or more sensors. Therefore, in some embodiments, the detection engine can be configured to determine additional information regarding the source of reflected energy (beam position on the scanning surface) based on triangulation or other methods. In some embodiments, when the detection engine utilizes triangulation or other methods to position the signal beam in the scanning environment, the combined sensor information can be considered a single sensor event including horizontal (x) position, vertical (y) position, and time component (t). Furthermore, in some embodiments, the sensor event may include other information, such as time-of-flight information, depending on the type or capability of the sensor.

[0096] Furthermore, as described above, a scanning signal generator (e.g., a scanning laser) can be configured to traverse a precise path / curve (e.g., a scanning path). Thus, in some embodiments, the pattern or sequence of cells in a sensor that detects reflected energy will follow a path / curve associated with the path / curve of the scanning signal generator. Thus, in some embodiments, when the signal generator scans a particular path / curve, the associated path / curve of the working cells of the sensor can be detected. Thus, in this embodiment, in some embodiments, path 608 can represent the sequence of cells in sensor 602B that detect reflected energy from the scanning signal generator.

[0097] In one or more of the various embodiments, the detection engine can be configured to fit sensor events to a scanning path curve. Accordingly, in one or more of the various embodiments, the detection engine can be configured to predict where a scanning event should occur based on the scanning path curve in order to determine information about the position or orientation of the scanning surface or object. Accordingly, in some embodiments, when the detection engine receives a sensor event that is not associated with a known scanning path curve, the detection engine can be configured to perform various actions, such as closing the current trajectory and starting a new trajectory, or discarding the sensor event as noise.

[0098] In one or more of the various embodiments, the scan path curve can be pre-configured within limits or constraints of the scan signal generator and sensor. For example, the scan signal generator can be configured or instructed to scan the scanning environment using various curves, including Lissajous curves, 2D lines, etc. In some cases, the scan path curve can be thought of as a piecewise function that modifies the direction or shape of different parts of the scan. For example, a 2D line scan path can be configured to change direction when approaching the edge of the scanning environment (e.g., field of view).

[0099] Those skilled in the art will understand that when an unobserved surface is scanned, the scanning frequency, scanning path, and sensor response frequency can determine whether the sensor detection path appears as a continuous path. Therefore, operating requirements such as the scanning signal generator, sensor accuracy, and sensor response frequency can be modified according to the system's application. For example, if the scanning environment may be relatively low-key and static, the sensor may have a low response time because the scanned environment does not change very quickly. Alternatively, if the scanning environment is dynamic or contains many features of interest, the sensor may require a high degree of responsiveness or accuracy to accurately capture the path of the reflected signal energy. Furthermore, in some embodiments, the characteristics of the scanning signal generator can be modified according to the scanning environment. For example, if a laser is used as the scanning signal generator, the energy level, wavelength, phase, beam width, etc., can be tuned to suit the environment.

[0100] In one or more of the various embodiments, the detection engine may provide sensor outputs as a continuous stream of sensor events or sensor information that identify the cell location in the sensor cell array, and as a timestamp corresponding to the time the detection event occurred.

[0101] In this embodiment, for some embodiments, the data structure 610 can be considered as a data structure representing sensor events based on sensor outputs provided to the detection engine. In this embodiment, column 612 represents the horizontal position of the location in the scanning environment, column 614 represents the vertical position in the scanning environment, and column 616 represents the time of the event. Thus, in some embodiments, the detection engine can be configured to determine sensor events (if any) that must be associated with a trajectory. In some embodiments, the detection engine can be configured to associate sensor events with existing trajectories or to create new trajectories. In some embodiments, if a sensor event fits an expected / predicted curve determined based on the scanning path curve, the detection engine can be configured to associate the sensor event with an existing trajectory or to create a new trajectory. Also, in some cases, for some embodiments, the detection engine can be configured to determine one or more sensor events as noise if the position of a sensor event deviates from the predicted path by a defined threshold.

[0102] In one or more of the various embodiments, the detection engine can be configured to determine sensor events for each individual sensor, rather than being limited to providing computer-calculated sensor events based on outputs from multiple sensors. For example, in some embodiments, the detection engine can be configured to provide a data structure similar to data structure 610 for collecting sensor events for individual sensors.

[0103] In some embodiments, the detection engine can be configured to generate a sequence of trajectories corresponding to reflected energy paths detected by a sensor. In some embodiments, the detection engine can be configured to utilize one or more data structures, such as data structure 618, to represent trajectories determined based on information acquired by the sensor. In this embodiment, data structure 610 may be a table-like structure including columns such as column 620 for storing a first x position, column 622 for storing a second x position, column 624 for storing a first y position, column 626 for storing a second y position, column 628 for storing the start time of the trajectory, and column 630 for storing the end time of the trajectory.

[0104] In this embodiment, row 632 represents information for a first trajectory, and row 634 further represents information for another trajectory. As described herein, the detection engine can be configured to utilize one or more rules or heuristics to determine whether one trajectory has ended and another trajectory has begun. In some embodiments, such heuristics may include observing geometrically close or temporally close occurrence sensor events. It should be noted that certain components or elements of the trajectory can be varied depending on the parametric representation or type of analysis curve of the analysis curve associated with the shape or orientation of the scanning path and scanning surface. Thus, those skilled in the art will understand that various types of analysis curves or curve representations may result in more or fewer parameters for each trajectory. Accordingly, in some embodiments, the detection engine can be configured to determine certain parameters of the trajectory based on rules, templates, libraries, etc., provided via configuration information to take local circumstances or local requirements into account.

[0105] In one or more of the various embodiments, the trajectory can be represented using curve parameters rather than a collection of individual points or pixels. Accordingly, in some embodiments, the detection engine can be configured to utilize one or more numerical methods to continuously fit a sequence of sensor events to a scan path curve.

[0106] Furthermore, in some embodiments, the detection engine can be configured to utilize one or more smoothing methods to improve trajectory accuracy or trajectory fitting. For example, in some embodiments, the scanning curve may include sensor events triggered by the scanning laser, which may not be a single cell width because the reflected energy may bounce off nearby cells or reach the boundaries of two or more cells. Therefore, in some embodiments, the detection engine can be configured to perform online smoothing estimation to accurately estimate the actual position of the reflected signal beam when traversing the sensor plane, for example, by using a smoothing Kalman filter to predict where the scanning beampoint should be in fractional units of the detector cell position and fractional units of the sensor's basic timestamp. Also, in some embodiments, the detection engine can be configured to utilize batch-based optimization routines, such as weighted least squares, to fit the smoothing curve to a continuous segment of the scanning trajectory that can correspond to the time the scanning signal generator beam has scanned over a continuous surface.

[0107] Furthermore, in some embodiments, the scanning path can be used to determine whether a trajectory has started or ended. For example, if the scanning path reaches the edge of the scanning area and changes direction further, the current trajectory may be terminated and a new trajectory may be started simultaneously to begin acquiring information based on the new scanning direction. Also, in some embodiments, an object or other feature blocking or obstructing scanning energy or reflected scanning energy may result in a break in the sensor output, which can result in a gap or other discontinuity that can trigger the trajectory to be closed and another trajectory to be opened following the break or gap. Moreover, in some embodiments, the detection engine can be configured to have a maximum trajectory length so that the trajectory can be closed when sufficient sensor events have been collected or when sufficient time has elapsed since the start of the trajectory.

[0108] Furthermore, in some embodiments, the detection engine can be configured to determine the activation of individual sensors. Accordingly, in some embodiments, the detection engine can be configured to provide a data structure similar to the data structure 618 of each sensor.

[0109] Figure 7 shows a logical representation of a scanning path for perceiving an object based on surface detection and detection of surface motion, according to one or more of various embodiments. In some embodiments, the detection engine can be configured to collect sensor output based on reflected scanning signal energy. In some embodiments, the detection engine can be configured to interpret sensor output information based in part on the scanning path of the scanning signal generator.

[0110] In one or more of the various embodiments, the detection engine can be configured to instruct the scanning signal generator to traverse a specific path defined by one or more curve functions. For example, scanning surface 702 represents a surface scanned using scanning path 710. Thus, in some embodiments, the detection engine can expect or predict that reflected energy follows the scanning path. In this embodiment, surface 702 is scanned using a linear path which can correspond to a specific 2D line function. In this embodiment, only a portion of the scanning pattern is shown.

[0111] Similarly, in some embodiments, surface 704 demonstrates how rapid scanning can cover the entire surface. In some embodiments, scanning coverage can be varied depending on the scanning path, scanning frequency, etc., and can be further adapted to various applications or environments. For example, some applications may require more precise scanning than other environments, depending on the various characteristics of the surface or object being scanned.

[0112] In some embodiments, the observed scanning path collected by the sensor may deviate from the planned scanning path due to an object or surface feature that alters the observed or reflected scanning path. For example, surface 706 shows a scanning path similar to that shown for surface 702, but the scanning path is shown as deformed in path portion 712. In this embodiment, this represents how an intervening object or surface feature may alter the reflected scanning energy path that can be observed by the sensor. For example, by comparing the scanning path of surface 702 with the scanning path of surface 706, it can be indicated that an object or surface feature may be present on surface 706 but not on surface 702.

[0113] Furthermore, in some embodiments, the observed deviation of the scanning path can correspond to the orientation of the scanned surface relative to the scanning signal generator or sensor. For example, in some embodiments, surface 708 may represent a surface rotated relative to surface 702 and a scanning signal generator (not shown). Thus, in some embodiments, the detection engine can be configured to determine the orientation of the surface based on a comparison of the scanning path and the actual reflected scanning path in order to determine that the surface can be rotated relative to the signal generator. Thus, in this embodiment, the scanning paths that generate trajectory 714 and trajectories 710A / 710B can be considered similar. However, in this embodiment, the trajectories themselves are different due to differences in the orientation of the scanned surface. Thus, in some embodiments, the modeling engine can be configured to detect or characterize the difference between the scanning path and the reflected path detected by the sensor in order to determine that surface 708 can be rotated relative to the detection system.

[0114] Furthermore, in some embodiments, the detection engine can be configured to collect a sequence of trajectories corresponding to the reflected scanning energy detected by the sensor, as described above. In this embodiment, for some embodiments, position 710A may represent the start of the trajectory, and position 710B may represent the end of the trajectory. Note that the trajectory may represent many sensor events that relate to each other in order to determine the trajectory. In this embodiment, it can be assumed that the scanning signal generator is configured to change direction when it reaches the edge of the scanning environment. Thus, in this embodiment, the position where the scanning signal generator changes direction can be treated as a discontinuity that ends one trajectory and starts yet another.

[0115] In one or more of the various embodiments, the detection engine can be configured to orient the scanning path so that the curves of the paths intersect each other. Thus, in some embodiments, if the scanning path itself intersects, the normal of the scanned surface can be calculated by determining the tangents to each of the trajectories at the intersection points. In some embodiments, the detection engine can be configured to computer calculate the normal of the surface scanned up to its sine by computer calculation of the cross product of the tangents. In some embodiments, the detection engine can be configured to determine the sine of the normal by selecting the direction of the normal that is closest to the direction returning to the sensor system, since this is the only physically possible orientation of the surface that can be examined by the sensor system. In addition, in some embodiments, the curvature of a two-dimensional surface can be approximated using the curvature of the individual trajectories of the scanning path of the surface or the entire object. For example, a person skilled in the art will understand that a 2D surface B-spline embedded in 3D space can be estimated using an intersecting 1D B-spline embedded in 3D space. Furthermore, in some embodiments, if many 1D curves traverse an area parameterized by a 2D surface B-spline, the detection engine can be configured to further adjust the 2D surface B-spline to more accurately approximate the object surface.

[0116] Figure 8 shows a logical representation of a scanning system 800 that perceives an object based on surface detection and detection of surface motion, according to one or more of various embodiments. As described above, a scanning system such as the scanning system 800 may include one or more scanning signal generators, such as the scanning signal generator 802, and multiple sensors, such as sensors 804A, 804B, and 804C. Note that one or more detection engines (not shown) or modeling engines (not shown) can be assumed to be communicatively coupled to the scanning signal generator 802 and sensors 804A, 804B, and 804C. Alternatively, in some embodiments, one or more detection engines or modeling engines may be hosted by the same computer, device, or equipment, or by sensors such as a robot or autonomous vehicle, that can provide scanning signals.

[0117] Furthermore, in this embodiment, surface 806A represents a downward-facing view of the surface in the scanning environment. Furthermore, in this embodiment, object 808A represents a downward-facing view of an object interposed between surface 806A and the scanning signal generator 802. As described herein, the scanning signal generator 802 may be a scanning laser configured to traverse a defined scanning path with a defined scanning rate. Similarly, as described herein, sensors 804A, 804B, 804C, etc., may provide sensor outputs to the detection engine based on the energy reflected from surface 806A and object 808A. Thus, in some embodiments, the detection engine may be configured to generate trajectories corresponding to the detection outputs and scanning paths.

[0118] In this embodiment, for some embodiments, surface 806B represents the same surface as surface 806A as seen from the front. Similarly, in this embodiment, object 808B represents the same object as object 808A as seen from the front. Furthermore, in this embodiment, for some embodiments, various positions representing the start or end point of a trajectory, which can be determined from the sensor output, are shown. For clarity, boundary lines indicating object 808B are included here, and it should be noted that these should not be confused with representing a trajectory or sensor output.

[0119] In this embodiment, for some embodiments, the detection engine can be configured to determine the trajectory based on the sensor output. In this embodiment, the trajectory can be determined such that the first trajectory is determined from position 810A to position 812A, the second trajectory from position 812A to position 814A, the third trajectory from position 814A to position 816A, and so on. Note that, as described herein, the trajectory also includes a time component, which is omitted here for brevity and clarity.

[0120] Furthermore, in this embodiment, positions 810B, 812B, 814B, and 816B can be considered to represent proximity to the same points of the above-mentioned locations. However, positions 818 and 820 can correspond to trajectories that do not include breaks / gaps.

[0121] Accordingly, in some embodiments, the modeling engine can be configured to determine information such as the shape of the object, the features of the object, the position of the object, the movement of the object, the rotation of the object, the surface features, and the orientation of the surface, based on the evaluation of the trajectory that can be determined by the detection engine. For example, the detection engine can determine the trajectory based on positions 810B, 812B, 814B, 816B, etc. Alternatively, for example, the detection engine can determine the trajectory based on positions 818 and 820. In some embodiments, the detection engine can be configured to compare / evaluate a sequence of trajectories to determine information about the scanning environment. In this embodiment, the detection engine can be configured to recognize an object by a change in the trajectory during scanning. In this embodiment, one trajectory appearing adjacent to another resulting scanning line can indicate the presence of an intervening object with surface features.

[0122] Figure 9 shows a logical representation of a scanning system 900 that perceives an object based on surface detection and detection of surface motion, according to one or more of various embodiments. As described above, a scanning system such as the scanning system 900 may include one or more scanning signal generators, such as a scanning signal generator 902, and multiple sensors, such as sensors 904A, 904B, and 904C. It is assumed that one or more detection engines or modeling engines can be communicatively coupled to the scanning signal generator 902 and sensors 904A, 904B, and 904C. Alternatively, in some embodiments, one or more detection engines or modeling engines may be hosted by the same computer or device that can provide scanning signals, or by sensors such as those in a robot or autonomous vehicle.

[0123] It should be noted that the scanning system 900 is considered to be similar to the scanning system 800 described above. Therefore, for the sake of brevity and clarity, one or more features or embodiments described for the scanning system 800 can be omitted here.

[0124] In one or more of the various embodiments, the scanning system can be configured to scan a scanning environment in which the background surface is not clearly defined. Therefore, in some embodiments, the scanning system can be directed to determine or identify an object or object activity that may be occupying space in such a way that the sensor output cannot provide information about the background surface. However, in some embodiments, the detection engine can be configured to evaluate the sensor output associated with one or more object absence pieces of information associated with the background surface. Thus, in this embodiment, object 906A represents a downward-facing view of an object that can be scanned by the scanning signal generator 902.

[0125] In one or more of the various embodiments, the detection engine can be configured to generate one or more trajectories for an object, similar to a method that can generate trajectories for a surface. In this embodiment, for some embodiments, 906B represents the same object as object 906A as seen from the viewpoint of the scanning signal generator or sensor. Therefore, in this embodiment, for some embodiments, two or more positions, such as positions 908A and 910A with start and end times, can be used to determine a trajectory that can be associated with object 906B. In this embodiment, the trajectory can also be determined based on a trajectory of object 906B absence associated with the background surface.

[0126] Furthermore, in this embodiment, positions 908B and 910B represent the proximity of positions 908A and 910A in the field of view. Thus, in some embodiments, the modeling engine can be configured to identify one or more features associated with objects 906A / 906B based on an analysis of trajectories associated with the scanned object, such as object features, object position, object movement, and object rotation.

[0127] Therefore, in some embodiments, the detection engine can be configured to determine the trajectory of an object in the scanning environment even if the scanning environment does not include a surface background. For example, in some embodiments, the detection engine determines trajectories for an object similar to the rotatable object 906A / B, such as a first trajectory with a start point at position 912 and an endpoint at position 914, and a second trajectory with a start point at position 914 and an endpoint at position 916. Therefore, in this embodiment, the modeling engine can be configured to infer, for some embodiments, that the object 906A / B can be a rotatable solid.

[0128] Furthermore, in some embodiments, the modeling engine can be configured to infer various features associated with the object based on a comparison of how the trajectory changes due to time overruns. As shown here, in this embodiment, time period 924 can represent the start and end times associated with a trajectory having position 908B as the start point and position 910B as the endpoint. Similarly, in this embodiment, time period 926 can represent the start and end times associated with a trajectory having position 912 as the start point and position 914 as the endpoint, time period 928 can represent the start and end times associated with a trajectory having position 914 as the start point and position 916 as the endpoint, time period 930 can represent the start and end times associated with a trajectory having position 918 as the start point and position 920 as the endpoint, and time period 932 can represent the start and end times associated with a trajectory having position 920 as the start point and position 922 as the endpoint.

[0129] Therefore, in some embodiments, the modeling engine can be configured to infer from the above-described trajectories that the object 906A / 906B can be made into a rotating solid by evaluating the trajectories that occur over time.

[0130] Figure 10 shows a logical schematic diagram of a system 1000 that perceives an object based on surface detection and surface motion detection, according to one or more of various embodiments. As described above, in some embodiments, the detection engine can be configured to generate trajectory information based on sensor outputs collected based on reflected signal energy.

[0131] Therefore, in some embodiments, a modeling engine such as the modeling engine 1002 can be configured to receive trajectory information such as trajectory information 1004. In this embodiment, for simplicity and clarity, trajectory information 1004 is shown using a diagram that can represent trajectory information. However, in some embodiments, the detection engine can provide modeling engine parameterized trajectory information provided through one or more data structures such as the data structure 618 shown in Figure 6.

[0132] Accordingly, the modeling engine 1002 can be configured to utilize the evaluation model 1006 to evaluate the trajectory information 1004. In some embodiments, the evaluation model can be configured to include one or more heuristics, rules, conditions, machine learning classifiers, etc., which can be used to evaluate the environment scanned based on the trajectory information 1004. In some embodiments, the evaluation model may have been conventionally trained or tuned to recognize or perceive various objects, shapes, movements, activities, relationships between or within objects, etc. However, in some embodiments, the input data used to train or tune the evaluation model may be in the form of a numerical representation of the trajectory (e.g., a stream updating parameter representations of analysis curve segments other than information derived from a point cloud, video frame captures, pixel-based edge detection, pixel-based motion detection, pixel-based color / luminance gradients, etc.).

[0133] In some embodiments, if the trajectory information 1004 can be evaluated, the modeling engine may be configured to generate or update one or more scene reports that provide information about the scene being evaluated. In some embodiments, the scene reports may include conventional reports, interactive reports, graphical dashboards, charts, plots, etc. Also in some embodiments, the scene reports may include one or more data structures that can represent various scene features which can be further provided to one or more machine vision applications, such as machine vision application 1010, which can automatically interpret the scene reports.

[0134] Those skilled in the art will understand that many machine vision or machine perception applications can utilize the inventions described herein. For example, in one or more of the various embodiments, a combination of a high-speed surface scan and a low-latency, high-throughput sensor, arranged in a configuration to directly provide depth measurements, allows the detection engine to scan the entire field of view in 1 millisecond and measure the surface with radial accuracy having a scale of a 30-meter human nose or a 0.5-meter individual thread of an M3 screw. Also, in some embodiments, the detection engine can be configured to scan the environment incrementally. Thus, in some embodiments, the detection engine can be configured to provide improved radial accuracy measurements of the surface when many observations are collected.

[0135] Furthermore, in some embodiments, the trajectory of the surface representation is progressively constructed on a sub-millisecond scale, allowing the detection system to track motion by observing how the apparent distance of the surface in the most recently scanned area changes over a period that may be short enough to be considered a continuous time update. Thus, in some embodiments, the detection engine can be configured to provide a full 6D representation of reality consisting of 2D surfaces with orientations moving through 3D space over time. This native 6D representation can be advantageous because it provides a expressive and accurate foundation for various application-specific perception algorithms to operate on.

[0136] Furthermore, in some embodiments, machine learning-based recognition algorithms can use shape and motion primitives (e.g., trajectories) as input instead of 2D color contrast or 3D point arrays. By using trajectories to represent the scanning environment, shapes and features present in the scanned environment can be individually and accurately identified. In contrast, some conventional machine vision systems may rely on guesswork or statistical approximations regarding whether an object can exist in position or illusion due to new contrasts, or whether points in space are part of the same surface or how the surface is moving.

[0137] Furthermore, in some embodiments, the detection engine requires fewer data values ​​to represent surface or object features using parameterized trajectories rather than conventional representations using 2D pixels or 3D point clouds. Also, in some embodiments, trajectory-based representations of surfaces / objects can be advantageous because they may be natively invariant under many deformations such as rotation, transformation, and changes in illumination.

[0138] Therefore, in some embodiments, the amount of data collected to train a deep learning recognition algorithm can be even less than that of less rich representations such as 2D images or 3D point clouds. For example, 2D color technology used to recognize pedestrians may require a collection of data with pedestrians in all possible poses and positions relative to the system, along with all possible lighting conditions and ambient background textures. In contrast, in some embodiments, the detection engine can be configured to identify the surface shape of pedestrians in a wide variety of poses, not only to position and recognize the surface shape of pedestrians in a wide variety of poses, all of which is done within milliseconds of first seeing the pedestrian, but also to reveal critical characteristics such as the orientation of body parts and relative motion.

[0139] Furthermore, in some embodiments, the detection systems described herein can be applied to other application domains beyond autonomous mobility. For example, in some embodiments, the detection engine and modeling engine can be used by a robot that searches for ripe fruit among dense foliage to pick fruit in a field, and can identify surface characteristics in detail faster than a human picker or conventional picking machine, and simultaneously calculate the optimal capture position with sub-millisecond updates while the robot hand quickly reaches to pick the berries.

[0140] Figure 11 shows a logical schematic diagram of a system 1100 that perceives an object based on the detection of a surface and the detection of surface motion, according to one or more of various embodiments. As described above, in some embodiments, the scanning signal generator can scan a surface in the scanning environment. In some cases, the conditions of the scanning environment or the characteristics of the surface being scanned may result in one or more false sensor events (e.g., noise) generated by one or more sensors. For example, sensor view 1102 represents a portion of the sensor events that may be generated during scanning.

[0141] In conventional machine vision applications, one or more 2D filters can be applied to captured video images, point clusters, etc., to attempt to separate noise events from the signal of interest. In some cases, conventional 2D image-based filters can be disadvantageous because they may utilize one or more filters (e.g., weighted moving average, Gaussian filters) that rely on statistical evaluation of pixel color / weighting, pixel color / weighting gradients, pixel distinction / clustering, etc. Therefore, in some cases, conventional 2D image filtering can be inherently fuzzy and highly dependent on application / environmental assumptions. Furthermore, in some cases, conventional noise detection / reduction methods may misclassify one or more scene events as noise while simultaneously overlooking some noise events.

[0142] In contrast, in some embodiments, the detection engine can be configured to associate sensor events with trajectories based on precise heuristics, such as position, which can be used to fit sensor events to an analysis curve that can be predicted based on temporal proximity and scanning path. Since the scanning path is predetermined, the detection engine can be configured to predict which sensor events should be included in the same trajectory.

[0143] Furthermore, in some embodiments, if surface or object features generate a gap or break in the trajectory, the detection engine can be configured to close the current trajectory and start a new one as soon as further recognition is possible.

[0144] Furthermore, in some embodiments, the detection engine can be configured to directly determine the trajectory from a sensor event having a specific form (x, y, t) rather than utilizing fuzzy pattern matching or pattern recognition methods. Thus, in some embodiments, the detection engine can be configured to accurately calculate distance, direction, etc., by computer, rather than relying on fuzzy machine vision methods to distinguish noise from sensor events that should be on the same trajectory.

[0145] Generalized operation Figure 12-17 illustrates a generalized operation for perceiving an object based on surface detection and surface motion detection, according to one or more of the various embodiments. In one or more of the various embodiments, processes 1200, 1300, 1400, 1500, 1600, and 1700 described with respect to Figure 12-17 can be implemented or executed by one or more processors in a single network computer (or network monitoring computer), such as the network computer 300 in Figure 3. In other embodiments, these processes, or parts of the processes, can be implemented or executed by multiple network computers, such as the network computer 300 in Figure 3. In yet another embodiment, these processes, or parts of the processes, can be implemented or executed by one or more virtualized computers in a cloud-based environment, etc. However, the embodiments are not limited thereto, and various combinations of network computers, client computers, etc., can be used. Furthermore, in one or more of the various embodiments, the processes described with respect to Figures 12-16 can perform object perception operations based on surface detection and surface motion detection according to at least one of the various embodiments or an architecture such as that described with respect to Figure 4-11. Furthermore, in one or more of the various embodiments, some or all of the operations performed by processes 1200, 1300, 1400, 1500, 1600, and 1700 can be partially performed by a detection engine 322 or a modeling engine 324 running on one or more processors of one or more network computers.

[0146] Figure 12 shows a schematic flowchart of a process 1200 for perceiving an object based on surface detection and detection of surface motion, according to one or more of the various embodiments. Following the start flowchart block, flowchart block 1202 includes one or more scanning signal generators, one or more sensors, etc., according to one or more of the various embodiments. In some embodiments, a specific scanning path can be provided to direct a beam or signal from the scanning signal generator to traverse a specified curve or path across the entire scanning environment. In block 1204, according to one or more of the various embodiments, the detection engine can be configured to utilize the scanning signal generator to scan a signal beam across the environment of interest to collect signal reflections of the sensor's signal. In block 1206, according to one or more of the various embodiments, the detection engine can be configured to provide a scene trajectory based on sensor output information. In block 1208, according to one or more of the various embodiments, the detection engine can be configured to provide one or more scene trajectories to the modeling engine. In block 1210, according to one or more of the various embodiments, the modeling engine can be configured to evaluate the scene in the environment being scanned based on the trajectory. As described herein, the modeling engine can be configured to utilize various evaluation models that can be tuned or trained to identify one or more shapes, objects, object activities, etc., based on their trajectories. In the decision block 1212, in one or more of the various embodiments, control can be returned to the call process if the scan can be terminated, otherwise control can be returned to block 1204.

[0147] Figure 13 shows a flowchart of a process 1300 for perceiving an object based on surface detection and detection of surface motion, according to one or more of the various embodiments. After the start block 1302, in one or more of the various embodiments, one or more sensors can capture signal reflections from one or more sensors. In block 1304, in one or more of the various embodiments, position and time information based on sensor outputs can be provided to the detection engine. In block 1306, in one or more of the various embodiments, the detection engine can be configured to determine one or more sensor events based on the scanning signal source position and the sensor position. In block 1308, in one or more of the various embodiments, the detection engine can be configured to determine one or more scene trajectories based on one or more sensor events and the scanning path of the signal beam. Next, in one or more of the various embodiments, control can be returned to the call process.

[0148] Figure 14 shows a flowchart of a process 1402 for perceiving an object based on surface detection and surface motion detection, according to one or more of the various embodiments. After the start block 1402, in one or more of the various embodiments, the detection engine can be configured to scan the scanning environment using a beam from a scanning signal generator. In the determination block 1404, in one or more of the various embodiments, if the scanning line intersects another previously collected scene trajectory, control can be passed to block 1406; otherwise, control can be returned to the call process. In block 1406, in one or more of the various embodiments, the detection engine can be configured to determine the surface normal of the two-dimensional scanning surface based on the intersecting scanning line. In block 1408, in one or more of the various embodiments, the detection engine can be configured to determine the orientation of the scanning surface based on the computer-calculated surface normal. Next, in one or more of the various embodiments, control can be returned to the call process.

[0149] Figure 15 shows a flowchart of a process 1502 for perceiving an object based on surface detection and surface motion detection, according to one or more of the various embodiments. After the start block 1502, in one or more of the various embodiments, the detection engine can be configured to collect sensor events based on the output of one or more sensors. As described above, sensor events can be provided based on multiple reports from more than one sensor (e.g., triangulation). Also, in some embodiments, sensor events can be provided individually to each sensor. In block 1504, in one or more of the various embodiments, the detection engine can be configured to evaluate one or more sensor events to determine a trajectory start point. In the determination block 1506, in one or more of the various embodiments, if a trajectory start point can be determined, control can be passed to block 1508; otherwise, control can be returned to block 1504. In block 1508, in one or more of the various embodiments, the detection engine can be configured to add the sensor events to the trajectory. In some embodiments, if sensor events can be collected, the sensor events can be associated with an open trajectory. In some embodiments, additional sensor events can be used to adjust or update the trajectory, allowing for the collection of even more information. In block 1510, in one or more of the various embodiments, the detection engine can be configured to fit / predict one or more sensor events to the trajectory based on the scan path. In the determination block 1512, in one or more of the various embodiments, control can be passed to block 1514 if a gap or discontinuity is determined, or otherwise, control can be returned to 1508. In block 1514, in one or more of the various embodiments, the detection engine can be configured to close the current trajectory. In block 1516, in one or more of the various embodiments, the detection engine can be configured to provide the closed trajectory to the modeling engine. In the determination block 1518, in one or more of the various embodiments, control can be returned to the calling process if the scanning process can be terminated, or otherwise, control can be returned to block 1504. Then, in one or more of the various embodiments, control can be returned to the calling process.

[0150] Figure 16 shows a flowchart of a process 1602 for perceiving an object based on surface detection and surface motion detection, according to one or more of the various embodiments. After the start block 1602, in one or more of the various embodiments, the detection engine can be configured to collect sensor events based on the output of one or more sensors. In the decision block 1604, in one or more of the various embodiments, control can be passed to block 1608 if the sensor events can be included in the trajectory, otherwise control can be passed to block 1606. In block 1606, in one or more of the various embodiments, the detection engine can be configured to exclude or discard the sensor events as noise. Next, in one or more of the various embodiments, control can be returned to the call process. In block 1608, in one or more of the various embodiments, the detection engine can be configured to associate the sensor events with the trajectory. Next, in one or more of the various embodiments, control can be returned to the call process.

[0151] It will be understood that each block illustrated in each flowchart, and each combination of blocks illustrated in each flowchart, can be implemented by computer program instructions. These program instructions can be provided to a processor to create a machine, and the instructions executed by the processor will generate means to perform the actions indicated in each flowchart block or combination of blocks. Computer program instructions can be executed by a processor to create a computer implementation process in a series of actuation steps, and the instructions executed by the processor will provide steps to perform the actions indicated in each flowchart block or combination of blocks. Computer program instructions can also be made to execute at least some of the actuation steps shown in each flowchart block in parallel. Furthermore, some of the steps can also be executed across more than one processor, which may occur in a multiprocessor computer system. In addition, one or more blocks or combinations of blocks in each flowchart can also be executed simultaneously by other blocks or combinations of blocks, or in a sequence different from those shown without departing from the scope or spirit of the invention.

[0152] Accordingly, each illustrated block in each flowchart supports a combination of means for performing a specified operation, a combination of steps for performing a specified operation, and a means for performing a specified operation. It will also be understood that each illustrated block in each flowchart, and any combination of illustrated blocks in each flowchart, can be implemented by a dedicated hardware-based system or a combination of dedicated hardware and computer instructions for performing a specified operation or step. The embodiments described above should not be interpreted as limiting or exhaustive, but rather as illustrative use cases illustrating at least one implementation of various embodiments of the present invention.

[0153] Furthermore, in one or more embodiments (not shown), the logic of the illustrative flowchart can be executed using embedded logic hardware such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable array logic (PAL), or a combination thereof, instead of a CPU. Embedded logic hardware devices can directly execute their own embedded logic to perform operations. In one or more embodiments, a microcontroller can be configured to directly execute its own embedded logic to perform operations and further access its own internal memory and its own external input and output interfaces (e.g., hardware pins or wireless transceivers) to perform operations such as those of a systemon chip (SOC).

[0154] Example Use Cases In one or more of the various embodiments, the detection system can be used to arbitrarily perceive complex environments depending on the application of the detection system. For brevity and clarity, much of the above description discloses inventions that enable the perception of objects based on surface detection and detection of surface motion, using simple surfaces, objects, or scenes. However, those skilled in the art will understand that some or all of these inventions can be used to perceive complex environments, including few or many complex objects or surfaces, which can be perceived as moving, experiencing deformation, surface changes, or shape changes that occur in real time. As described with respect to the embodiment shown in Figure 9 (rotating solid object), the detection engine can dynamically generate new trajectories or update existing trajectories based on how the object or surface of the scanning environment moves or changes. As described above, these updated / additional trajectories can be provided to the modeling engine in morphological numerical representations such as vectors, arrays, or matrices, where each element corresponds to a parameter value that is part of the parameter representation of a segment of the analysis curve. Thus, the modeling engine can be configured to identify one or more features associated with the detected surface based on the trajectories provided to the evaluation models, using one or more evaluation models. The actions taken in response to the determination of specific features of interest or specific features can be varied depending on the application.

[0155] In this embodiment, environment 1700 represents a scene including a human hand. In this embodiment, the contour lines can correspond to trajectories determined from the scene. Accordingly, in accordance with this embodiment, in some embodiments, the modeling engine can be configured to utilize an evaluation model that has been tuned or trained to determine various features of a complex object, such as a human hand, based on its trajectory. In this embodiment, such features may include shape, size, finger positions, hand position, rotation, speed of movement, distance from other surfaces or other objects, etc.

Claims

1. A method for perceiving surfaces and objects using one or more processors configured to execute instructions, The aforementioned instruction is, The method involves generating one or more trajectories based on a continuous stream of sensor events, wherein each trajectory is a parameterized representation of a one-dimensional curve segment in three-dimensional space. Determining the one or more surfaces using the one or more orbitals, The modeling engine is provided with one or more orbitals, and the modeling engine further performs one or more operations based on the one or more orbitals and the one or more surfaces. Perform an action that includes, In response to the one or more changes on the one or more surfaces, the further operation is performed. Updating the one or more trajectories based on the continuous stream of sensor events, Performing one or more additional operations based on the one or more updated trajectories and the one or more changed surfaces, Methods that include...

2. The method according to claim 1, wherein the one or more changes to the one or more surfaces include one or more of the following: a change in position, a change in orientation, a change in movement, or deformation of the one or more surfaces.

3. The method according to claim 1, further comprising providing a continuous stream of sensor events based on one or more sensors, wherein each sensor event includes one or more of a timestamp, time of flight, or position value.

4. The method according to claim 1, further comprising determining one or more shapes corresponding to the one or more surfaces based on one or more characteristics of the one or more surfaces and the one or more orbitals.

5. The method according to claim 1, wherein each of the aforementioned orbitals further includes a parameterized representation of a B-spline.

6. The method according to claim 1, further comprising using the one or more orbitals to continuously determine one or more changes to one or more of the positions of the one or more surfaces, the orientations of the one or more surfaces, the deformations of the one or more surfaces, or the movements of the one or more surfaces.

7. The method according to claim 1, further comprising performing the aforementioned further operation to determine one or more objects based on the portion of the one or more trajectories associated with the portion of the one or more surfaces.

8. The method according to claim 6, wherein performing the further operations further includes determining one or more features of the one or more objects based on the one or more trajectories, the one or more features including one or more of the position, orientation, movement or deformation of the one or more objects.

9. A system for perceiving surfaces and objects, At least memory to store instructions, One or more processors configured to execute instructions, Includes a network computer, The aforementioned instruction is, The method involves generating one or more trajectories based on a continuous stream of sensor events, wherein each trajectory is a parameterized representation of a one-dimensional curve segment in three-dimensional space. Determining the one or more surfaces using the one or more orbitals, The modeling engine is provided with one or more orbitals, and the modeling engine further performs one or more operations based on the one or more orbitals and the one or more surfaces. Perform an action that includes, In response to the one or more changes on the one or more surfaces, the further operation is performed. Updating the one or more trajectories based on the continuous stream of sensor events, Performing one or more additional operations based on the one or more updated trajectories and the one or more changed surfaces, Includes, The aforementioned system further, At least memory to store instructions, One or more processors configured to execute instructions, It includes one or more client computers, The command is a system that performs an operation which includes providing one or more parts of the sensor event.

10. The system according to claim 9, wherein the one or more changes to the one or more surfaces include one or more of the following: a change in position, a change in orientation, a change in movement, or deformation of the one or more surfaces.

11. One or more processors of the network computer are configured to execute instructions, and these instructions are: The operation further includes providing a continuous stream of sensor events based on one or more sensors, The system according to claim 9, wherein each of the sensor events includes one or more of a timestamp, time of flight, or position value.

12. The system according to claim 9, wherein one or more processors of the network computer are configured to execute instructions, the instructions further include performing an operation that determines one or more shapes corresponding to one or more surfaces based on one or more characteristics of the one or more surfaces and the one or more trajectories.

13. The system according to claim 9, wherein each of the aforementioned orbits further includes a parameterized representation of a B-spline.

14. The system according to claim 9, wherein one or more processors of the network computer are configured to execute instructions, the instructions further include performing an operation that uses the one or more trajectories to successively determine one or more changes to one or more of the positions of the one or more surfaces, the orientations of the one or more surfaces, the deformations of the one or more surfaces, or the movements of the one or more surfaces.

15. The system according to claim 9, further comprising performing the aforementioned further operations to determine one or more objects based on the portion of the one or more trajectories associated with the portion of the one or more surfaces.

16. The system according to claim 15, wherein performing the further operations further includes determining one or more features of the one or more objects based on the one or more trajectories, the one or more features including one or more of the position, orientation, movement or deformation of the one or more objects.

17. A processor-readable non-temporary storage medium containing instructions for perceiving surfaces and objects, wherein the execution of said instructions by one or more processors on one or more network computers is: The method involves generating one or more trajectories based on a continuous stream of sensor events, wherein each trajectory is a parameterized representation of a one-dimensional curve segment in three-dimensional space. Determining the one or more surfaces using the one or more orbitals, The modeling engine is provided with one or more orbitals, and the modeling engine further performs one or more operations based on the one or more orbitals and the one or more surfaces. Perform an action that includes, In response to the one or more changes on the one or more surfaces, the further operation is performed. Updating the one or more trajectories based on the continuous stream of sensor events, Performing one or more additional operations based on the one or more updated trajectories and the one or more changed surfaces, Processor-readable non-temporary storage media, including [specific data / information].

18. The medium according to claim 17, wherein the one or more changes to the one or more surfaces include one or more of the following: a change in position, a change in orientation, a change in movement, or deformation of the one or more surfaces.

19. The medium according to claim 17, further comprising providing a continuous stream of sensor events based on one or more sensors, wherein each sensor event includes one or more of a timestamp, time of flight, or position value.

20. The medium according to claim 17, further comprising determining one or more shapes corresponding to the one or more surfaces based on one or more characteristics of the one or more surfaces and the one or more orbitals.

21. The medium according to claim 17, wherein each of the aforementioned orbitals further includes a parameterized representation of a B-spline.

22. The medium according to claim 17, further comprising using the one or more orbits to continuously determine one or more changes to one or more of the positions of the one or more surfaces, the orientations of the one or more surfaces, the deformations of the one or more surfaces, or the movements of the one or more surfaces.

23. The medium according to claim 17, further comprising performing the aforementioned further operation to determine one or more objects based on the portion of the one or more trajectories associated with the portion of the one or more surfaces.

24. The medium according to claim 23, wherein performing the further operation further includes determining one or more features of the one or more objects based on the one or more trajectories, the one or more features including one or more of the position, orientation, movement or deformation of the one or more objects.

25. A network computer that perceives surfaces and objects, At least memory to store instructions, One or more processors configured to execute instructions, Equipped with, The aforementioned instruction is, The method involves generating one or more trajectories based on a continuous stream of sensor events, wherein each trajectory is a parameterized representation of a one-dimensional curve segment in three-dimensional space. Determining the one or more surfaces using the one or more orbitals, The modeling engine is provided with one or more orbitals, and the modeling engine further performs one or more operations based on the one or more orbitals and the one or more surfaces. Perform an action that includes, In response to the one or more changes on the one or more surfaces, the further operation is performed. Updating the one or more trajectories based on the continuous stream of sensor events, Performing one or more additional operations based on the one or more updated trajectories and the one or more changed surfaces, Network computers, including

26. The network computer according to claim 25, wherein the one or more changes to the one or more surfaces include one or more of the following: a change in position, a change in orientation, a change in movement, or deformation of the one or more surfaces.

27. The one or more processors are configured to execute instructions, The aforementioned instruction is, The operation further includes providing a continuous stream of sensor events based on one or more sensors, The network computer according to claim 25, wherein each of the aforementioned sensor events includes one or more of a timestamp, time of flight, or position value.

28. The one or more processors are configured to execute instructions, The network computer according to claim 25, wherein the instruction further performs an operation that includes determining one or more shapes corresponding to the one or more surfaces based on one or more characteristics of the one or more surfaces and the one or more trajectories.

29. The network computer according to claim 25, wherein each of the aforementioned orbits further includes a parameterized representation of a B-spline.

30. The network computer according to claim 25, wherein the one or more processors are configured to execute instructions, the instructions further include performing operations that use the one or more trajectories to successively determine one or more changes to one or more of the positions of the one or more surfaces, the orientations of the one or more surfaces, the deformations of the one or more surfaces, or the movements of the one or more surfaces.

31. The network computer according to claim 25, further comprising performing the aforementioned further operations to determine one or more objects based on the portion of the one or more trajectories associated with the portion of the one or more surfaces.

32. The network computer according to claim 31, wherein performing the further operations further includes determining one or more features of the one or more objects based on the one or more trajectories, the one or more features including one or more of the position, orientation, movement or deformation of the one or more objects.

Images