Method and system for scene understanding in a multi-sensor platform

Abstract

A system and method of understanding a scene by at least one processor may include an iterative process. At least one iteration of the iterative process may include acquiring, from one or more scene sensors mounted on a mobile platform, one or more respective scene data streams, representing a scene surrounding the mobile platform; analyzing at least one of the one or more scene data streams, to determine a movement scenario data element, representing a dynamic relation between the mobile platform and its surroundings; based on the movement scenario data element, modifying a data transfer rate of at least one of the one or more scene data streams; and constructing a model of the scene, based on the one or more scene data streams of the modified data transfer rate.

Description

FOR-P-008-PCTMETHOD AND SYSTEM FOR SCENE UNDERSTANDING IN A MULTISENSOR PLATFORMCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority of U.S. Provisional Patent Applications No. 63 / 682,894, titled "METHOD AND SYSTEM FOR SCENE UNDERSTANDING IN A MULTI-SENSOR PLATFORM", filed 14 August 2024, the contents of which is all incorporated herein by reference in its entirety.FIELD OF THE INVENTION

[0002] The present invention relates generally to analysis of images in real-time, or near real-time. More specifically, the present invention relates to methods and systems for understanding a scene in a multi-sensor platform.BACKGROUND OF THE INVENTION

[0003] In recent years, advancements in assistive driving and autonomous vehicle technology have revolutionized the automotive industry, promising safer and more efficient transportation solutions. Essential to the operation of such technologies is their ability to perceive and interpret the surrounding environment accurately, in real-time. Sensors (e.g., cameras) mounted on these vehicles play a pivotal role in capturing visual data, enabling various perception tasks such as object detection, lane tracking, and pedestrian recognition.

[0004] However, despite the widespread adoption of camera-based perception systems, challenges persist in efficiently analyzing the vast amount of image data generated by autonomous vehicles.

[0005] To address challenges in understanding complex and dynamic driving environments, and improve the analysis of images obtained from sensors (e.g., cameras) mounted on platforms such as autonomous vehicles, there is a growing need for innovative solutions that can enhance the efficiency, accuracy, and robustness of image perception systems.

[0006] The present invention seeks to fulfill this need by introducing a novel approach to image analysis specifically tailored for vehicle-mounted cameras, enabling mobile platforms such as autonomous vehicles and Advanced Driver Assistance Systems (ADAS) to achieveFOR-P-008-PCT higher levels of accuracy, reliability, and adaptability in interpreting visual information from their surroundings.SUMMARY OF THE INVENTION

[0007] Embodiments of the invention provide a method and system for dynamic computational resource allocation, for enhanced environmental perception in an autonomous, mobile system. Embodiments of the invention may serve for a wide variety of applications, including autonomous vehicle navigation, 3D environmental mapping, advanced robotics and drone navigation, virtual and augmented reality.

[0008] Embodiments of the invention provide a novel, adaptive strategy for managing and optimizing computational resources in multi-view imaging systems. As elaborated herein, embodiments of the invention may dynamically, and separately alter each sensor’s processing capacity, influenced by factors like vehicle orientation, scene intricacy, and varying driving conditions. Embodiments of the invention may fluidly adjust, and control processing aspects, such as resolution, field of view, frame rate, and accuracy of reconstruction models. Embodiments of the invention may also be utilized to intelligently toggle assisting driving features, including for example object detection, object tracking, scene segmentation, and lane-keeping.

[0009] Embodiments of the invention may include a method of understanding a scene by at least one processor, in an iterative process. In at least one iteration of the iterative process, the at least one processor may acquire, from one or more scene sensors mounted on a mobile platform, one or more respective scene data streams, representing a scene surrounding the mobile platform. The at least one processor may analyze at least one of the one or more scene data streams, to determine a movement scenario data element, representing a dynamic relation between the mobile platform and its surroundings. Based on the movement scenario data element, the at least one processor may modify a data transfer rate of at least one of the one or more scene data streams, construct a model of the scene, based on the one or more scene data streams of the modified data transfer rate.

[0010] According to some embodiments, the at least one processor may be associated with, and may communicate with a controller of the mobile platform. Subject to this communication, the controller of the mobile platform may control at least one motor or actuator of the mobile platform, so as to conduct the mobile platform based on the scene model.FOR-P-008-PCT[Oi l] According to some embodiments, the one or more scene sensors may include, for example, one or more cameras having timewise-overlapping, or physically-overlapping Field Of View (FOV).

[0012] Additionally, or alternatively, the one or more scene sensors may include, for example: a single imaging device, one or more single imaging devices, having a timewise overlapping FOV, two or more imaging devices, configured to have a physically- overlapping FOV, one or more LIDAR sensors, and one or more radar sensors.

[0013] According to some embodiments, the at least one processor may construct a scene model that may include a three dimensional (3D) model of the scene, based on the one or more scene data streams of the modified data rate.

[0014] According to some embodiments, the at least one processor may apply a first machine-learning (ML) based model on the movement scenario data element to predict a first requirement for function of at least one of the one or more scene sensors. Based on the first requirement, the at least one processor may determine an acquisition parameter of at least one scene sensor, and control the scene sensor based on the determined acquisition parameter, to modify the data transfer rate of at least one respective scene data stream.

[0015] The acquisition parameter of the one or more scene sensors may include, for example: a frame rate of a scene sensor of the one or more scene sensors, a resolution of the scene sensor, a grey-level quantization of the scene sensor, a quantization of a color channel of the scene sensor, an FOV of the scene sensor, a definition of a Region Of Interest (ROI) within the FOV of the scene sensor, and a range of the scene sensor.

[0016] Additionally, or alternatively, in at least one iteration, based on the movement scenario data element, the at least one processor may control allocation of at least one computational resource, for analysis of the at least one scene data stream in a subsequent iteration.

[0017] According to some embodiments, the at least one processor may control allocation of at least one computational resource by: applying a second ML-based model on the movement scenario data element to predict a second requirement for analysis of at least one scene data stream; and based on the second requirement, modifying allocation of at least one computational resource of the at least one processor, for analyzing the at least one scene data stream.FOR-P-008-PCT

[0018] According to some embodiments, the at least one processor may analyze a scene data stream by: applying an object recognition algorithm on the scene data stream, to identify one or more objects in the scene; and predicting a relative motion vector between the one or more identified objects and the mobile platform in a subsequent iteration. The movement scenario data element may, for example, include: (i) the identification of the one or more object and (ii) the predicted relative motion vector.

[0019] According to some embodiments, the at least one processor may predict the relative motion vector by: analyzing the scene data streams to assess a trajectory of the identified object in relation to the mobile platform; obtaining a platform location data element, representing at least one of: a position, an orientation, a velocity, an acceleration and a trajectory of the mobile platform; and calculating the relative motion vector based on (i) the trajectory of the recognized object and (ii) the platform location data element.

[0020] Additionally, or alternatively, the at least one processor may analyze a scene data stream by: for each of the one or more identified objects, calculating a relevance score based on: (i) a type of the identified object and (ii) the predicted relative motion vector; and adjusting the movement scenario data element to include the relevance score.

[0021] According to some embodiments, the at least one processor may obtain at least one environmental data element, representing an environment-related condition at a location of the mobile platform; and adjust the relevance score based on the at least one environmental data element.

[0022] Additionally, or alternatively, the at least one processor may obtain at least one diagnostic data element, indicating a potential issue affecting a performance of the mobile platform; and adjust the relevance score based on the at least one diagnostic data element.

[0023] Additionally, or alternatively, the at least one processor may obtain at least one platform attribute data element representing at least one of (i) a dimension of the mobile platform and (ii) a navigational requirement of the mobile platform; and adjust the relevance score based on the at least one platform attribute data element.

[0024] According to some embodiments, the at least one processor may analyze the scene data stream to determine a scene complexity data element, representing a quantity and a distribution of the one or more identified objects. The at least one processor may adjust the movement scenario data element to include the scene complexity data element; and modifyFOR-P-008-PCT the data transfer rate of at least one of the one or more scene data streams based on the adjusted movement scenario data element.

[0025] Additionally, or alternatively, the at least one processor may dynamically (e.g., changing, over time) control allocation of at least one computing resource, to efficiently analyze the one or more scene data streams, corresponding to the modified data rate.

[0026] According to some embodiments, the at least one processor may apply a third ML- based model on the platform location data element to produce a predicted scene complexity value. The at least one processor may modify the data transfer rate of at least one of the one or more scene data streams further based on the predicted scene complexity value.

[0027] According to some embodiments, at least two of the first, second, and third ML- based models may be implemented by a single, unified ML model entity.

[0028] Additionally, or alternatively, the object recognition algorithm, and at least one of the first, second, and third ML-based models may be implemented by a single, unified ML model entity.

[0029] Embodiments of the invention may include a system for understanding a scene. Embodiments of the system may include a non-transitory memory device, wherein modules of instruction code may be stored, and at least one processor associated with the memory device, and configured to execute the modules of instruction code. Upon execution of said modules of instruction code, the at least one processor may be configured to execute an iterative process. Each iteration of the iterative process may include, for example: acquiring, from one or more scene sensors mounted on a mobile platform, one or more respective scene data streams, representing a scene surrounding the mobile platform; analyzing at least one of the one or more scene data streams, to determine a movement scenario data element, representing a dynamic relation between the mobile platform and its surroundings; based on the movement scenario data element, modifying a data transfer rate of at least one of the one or more scene data streams; and constructing a model of the scene, based on the one or more scene data streams of the modified data transfer rate.

[0030] Embodiments of the invention may include a mobile platform, that may include at least one motor or actuator, a controller configured to control the at least one motor or actuator, one or more scene sensors, and at least one processor. According to some embodiments, the at least one processor may be configured to perform an iterative process, where at least one iteration of the iterative process may include: acquiring, from the one orFOR-P-008-PCT more scene sensors, one or more respective scene data streams, representing a scene surrounding the mobile platform; analyzing at least one of the one or more scene data streams, to determine a movement scenario data element, representing a dynamic relation between the mobile platform and its surroundings; based on the movement scenario data element, modifying a data transfer rate of at least one of the one or more scene data streams; constructing a model of the scene, based on the one or more scene data streams of the modified data transfer rate; and communicating said controller, to control the at least one motor or actuator, so as to conduct the mobile platform based on the scene model.BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The subject matter regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of the specification. The invention, however, both as to organization and method of operation, together with objects, features, and advantages thereof, may best be understood by reference to the following detailed description when read with the accompanying drawings in which:

[0032] Fig. l is a block diagram, depicting a computing device which may be included in a system for scene understanding in a multi-sensor platform, according to some embodiments;

[0033] Fig. 2 is a block diagram, depicting a simplified overview of a system for scene understanding in the multi-sensor platform, according to some embodiments;

[0034] Fig. 3 is a block diagram, depicting a portion of the system for scene understanding, according to some embodiments;

[0035] Fig. 4 is a block diagram, depicting another portion of the system for scene understanding, according to some embodiments; and

[0036] Fig. 5 is a flow diagram, depicting a method of scene understanding in a multisensor platform, according to some embodiments of the invention.

[0037] It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements.FOR-P-008-PCTDETAILED DESCRIPTION OF THE PRESENT INVENTION

[0038] One skilled in the art will realize the invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The foregoing embodiments are therefore to be considered in all respects illustrative rather than limiting of the invention described herein. Scope of the invention is thus indicated by the appended claims, rather than by the foregoing description, and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.

[0039] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the present invention. Some features or elements described with respect to one embodiment may be combined with features or elements described with respect to other embodiments. For the sake of clarity, discussion of same or similar features or elements may not be repeated.

[0040] Although embodiments of the invention are not limited in this regard, discussions utilizing terms such as, for example, “processing,” “computing,” “calculating,” “determining,” “establishing”, “analyzing”, “checking”, or the like, may refer to operation(s) and / or process(es) of a computer, a computing platform, a computing system, or other electronic computing device, that manipulates and / or transforms data represented as physical (e.g., electronic) quantities within the computer’s registers and / or memories into other data similarly represented as physical quantities within the computer’s registers and / or memories or other information non-transitory storage medium that may store instructions to perform operations and / or processes.

[0041] Although embodiments of the invention are not limited in this regard, the terms “plurality” and “a plurality” as used herein may include, for example, “multiple” or “two or more”. The terms “plurality” or “a plurality” may be used throughout the specification to describe two or more components, devices, elements, units, parameters, or the like. The term “set” when used herein may include one or more items.

[0042] Unless explicitly stated, the method embodiments described herein are not constrained to a particular order or sequence. Additionally, some of theFOR-P-008-PCT described method embodiments or elements thereof can occur or be performed simultaneously, at the same point in time, or concurrently.

[0043] Reference is now made to Fig. 1, which is a block diagram depicting a computing device, which may be included within an embodiment of a system for understanding a scene in a multiple-sensor platform, according to some embodiments.

[0044] Computing device 1 may include a processor or controller 2 that may be, for example, a central processing unit (CPU) processor, a chip or any suitable computing or computational device, an operating system 3, a memory 4, executable code 5, a storage system 6, input devices 7 and output devices 8. Processor 2 (or one or more controllers or processors, possibly across multiple units or devices) may be configured to carry out methods described herein, and / or to execute or act as the various modules, units, etc. More than one computing device 1 may be included in, and one or more computing devices 1 may act as the components of, a system according to embodiments of the invention.

[0045] Operating system 3 may be or may include any code segment (e.g., one similar to executable code 5 described herein) designed and / or configured to perform tasks involving coordination, scheduling, arbitration, supervising, controlling or otherwise managing operation of computing device 1, for example, scheduling execution of software programs or tasks or enabling software programs or other modules or units to communicate. Operating system 3 may be a commercial operating system. It will be noted that an operating system 3 may be an optional component, e.g., in some embodiments, a system may include a computing device that does not require or include an operating system 3.

[0046] Memory 4 may be or may include, for example, a Random-Access Memory (RAM), a read only memory (ROM), a Dynamic RAM (DRAM), a Synchronous DRAM (SD-RAM), a double data rate (DDR) memory chip, a Flash memory, a volatile memory, a non-volatile memory, a cache memory, a buffer, a short term memory unit, a long term memory unit, or other suitable memory units or storage units. Memory 4 may be or may include a plurality of possibly different memory units. Memory 4 may be a computer or processor non-transitory readable medium, or a computer non-transitory storage medium, e.g., a RAM. In one embodiment, a non-transitory storage medium such as memory 4, a hard disk drive, another storage device, etc. may store instructions or code which when executed by a processor may cause the processor to carry out methods as described herein.FOR-P-008-PCT

[0047] Executable code 5 may be any executable code, e.g., an application, a program, a process, task, or script. Executable code 5 may be executed by processor or controller 2 possibly under control of operating system 3. For example, executable code 5 may be an application that may understand a scene, as further described herein. Although, for the sake of clarity, a single item of executable code 5 is shown in Fig. 1, a system according to some embodiments of the invention may include a plurality of executable code segments similar to executable code 5 that may be loaded into memory 4 and cause processor 2 to carry out methods described herein.

[0048] Storage system 6 may be or may include, for example, a flash memory as known in the art, a memory that is internal to, or embedded in, a micro controller or chip as known in the art, a hard disk drive, a CD-Recordable (CD-R) drive, a Blu-ray disk (BD), a universal serial bus (USB) device or other suitable removable and / or fixed storage unit. Data pertaining to a surrounding of the platform may be stored in storage system 6 and may be loaded from storage system 6 into memory 4 where it may be processed by processor or controller 2. In some embodiments, some of the components shown in Fig. 1 may be omitted. For example, memory 4 may be a non-volatile memory having the storage capacity of storage system 6. Accordingly, although shown as a separate component, storage system 6 may be embedded or included in memory 4.

[0049] Input devices 7 may be or may include any suitable input devices, components, or systems, e.g., a detachable keyboard or keypad, a mouse and the like. Output devices 8 may include one or more (possibly detachable) displays or monitors, speakers and / or any other suitable output devices. Any applicable input / output (VO) devices may be connected to Computing device 1 as shown by blocks 7 and 8. For example, a wired or wireless network interface card (NIC), a universal serial bus (USB) device or external hard drive may be included in input devices 7 and / or output devices 8. It will be recognized that any suitable number of input devices 7 and output device 8 may be operatively connected to Computing device 1 as shown by blocks 7 and 8.

[0050] A system according to some embodiments of the invention may include components such as, but not limited to, a plurality of central processing units (CPU) or any other suitable multi-purpose or specific processors or controllers (e.g., similar to element 2), a plurality of input units, a plurality of output units, a plurality of memory units, and a plurality of storage units.FOR-P-008-PCT

[0051] The term neural network (NN) or artificial neural network (ANN), e.g., a neural network implementing a machine learning (ML) or artificial intelligence (Al) function, may be used herein to refer to an information processing paradigm that may include nodes, referred to as neurons, organized into layers, with links between the neurons. The links may transfer signals between neurons and may be associated with weights. A NN may be configured or trained for a specific task, e.g., pattern recognition or classification. Training a NN for the specific task may involve adjusting these weights based on examples. Each neuron of an intermediate or last layer may receive an input signal, e.g., a weighted sum of output signals from other neurons, and may process the input signal using a linear or nonlinear function (e.g., an activation function). The results of the input and intermediate layers may be transferred to other neurons and the results of the output layer may be provided as the output of the NN. Typically, the neurons and links within a NN are represented by mathematical constructs, such as activation functions and matrices of data elements and weights. At least one processor (e.g., processor 2 of Fig. 1) such as one or more CPUs or graphics processing units (GPUs), or a dedicated hardware device may perform the relevant calculations.

[0052] Reference is now made to Fig. 2, which depicts a simplified overview of a system 10 for understanding a scene in a multiple-sensor platform 11. Multiple-sensor platform 11 may include, for example a manual, autonomous, or semi-autonomous vehicle, a drone, a robot and the like, and may be associated with an assistive driving system. According to some embodiments of the invention, system 10 may be implemented as a software module, a hardware module, or any combination thereof. For example, system 10 may be or may include a computing device such as element 1 of Fig. 1, and may be adapted to execute one or more modules of executable code (e.g., element 5 of Fig. 1) to understand a scene surrounding platform 11, as further described herein.

[0053] As shown in Fig. 2, arrows may represent flow of one or more data elements to and from system 10 and / or among modules or elements of system 10. Some arrows have been omitted for the purpose of clarity.

[0054] Reference is also made to Fig. 3, which is a block diagram, depicting a portion of system 10 (e.g., same as system 10 of Fig. 2) for scene understanding, according to some embodiments.FOR-P-008-PCT

[0055] As shown in Figs. 2 and 3, system 10 may be associated with (e.g., communicatively connected to) a plurality of sensors 20, that may be mounted on, or related to platform 11. Additionally, or alternatively, system 10 may include one or more of the plurality of sensors 20.

[0056] Sensors 20 may include, for example, one or more (e.g., a plurality) of scene sensors 21, adapted to produce data representing a scene surrounding platform 11.

[0057] Scene sensors 21 may include, for example a single imaging device (e.g., a camera), one or more single imaging devices (e.g., cameras), having a timewise overlapping (e.g., through motion) Field Of View (FOV), two or more (e.g., a pair of ) cameras, configured to have a physically-overlapping FOV such as a stereo camera device, one or more LIDAR sensors, one or more radar sensors, and the like.

[0058] Additionally, or alternatively, sensors 20 may include at least one location sensor 23, configured to obtain at least one location data element 23D, representing location of mobile platform 11. For example, location sensor 23 may be a Global Positioning Satellite (GPS) receiver, and location data element 23D may include an indication regarding location (e.g., latitude and longitude) of mobile platform 11.

[0059] Additionally, or alternatively, sensors 20 may include at least one environmental sensor 25, configured to obtain at least one environmental data element 25D, representing an environment-related condition at a location of mobile platform 11. For example, environmental sensor 25 may be a rain sensor that is assembled on mobile platform 11, and environmental data element 25D may thereby include an indication regarding precipitation at a location of mobile platform 11.

[0060] Additionally, or alternatively, system 10 may be communicatively connected (e.g., via wireless communication such as a cellular data network) to at least one external computing device 50, such as a server computer. System 10 may obtain environmental data element 25D from server computer 50. In such embodiments, environmental data element 25D may environmental related data (e.g., temperature, wind, humidity, precipitation, etc.) pertaining to a current time and location of mobile platform 11, and / or pertaining to projected time and location (e.g., a weather forecast of temperature, wind, humidity, precipitation, etc.) for a future time and / or area surrounding mobile platform 11.FOR-P-008-PCT

[0061] Additionally, or alternatively, sensors 20 may include at least one conduction sensor 27, configured to obtain at least one conduction data element 27D, representing conduction of mobile platform 11.

[0062] For example, conduction data element 27D may include data representing a speed of mobile platform 11, a bearing of mobile platform 11, a steering (e.g., angle of a steering wheel) of mobile platform 11, an inclination of mobile platform 11, information regarding a motor or actuator (e.g., revolutions per minute (RPM)) of mobile platform 11, information regarding a breaking mechanism of mobile platform 11, and the like.

[0063] Additionally, or alternatively, sensors 20 may include, or be communicatively connected to at least one diagnostic sensor 29, configured to obtain at least one diagnostic data element 29D, representing diagnostic information regarding functionality of one or more elements of mobile platform 11.

[0064] For example, diagnostic data element 29D may include an indication regarding effectiveness of a breaking mechanism of mobile platform 11 (e.g., influencing deceleration of mobile platform 11), indication regarding operation of a motor of mobile platform 11 (e.g., influencing acceleration of mobile platform 11), indication regarding operation of a steering mechanism of mobile platform 11, and the like.

[0065] As shown in Figs. 2 and 3, system 10 may include a data acquisition module 100, adapted to acquire, from one or more sensors 20 (e.g., 21, 23, 25, 27 and / or 29) at least one data element 20D (e.g., 21D, 23D, 25D, 27D and / or 29D respectively), and transfer the at least one data element 20D (now denoted 120D) to be further analyzed by a data analysis module 200. As elaborated herein, data acquisition module 100 may control a volume, or rate of transferred data 120D in one or more ways:

[0066] For example, data acquisition module 100 may include a sensor configuration module 110, adapted to control, or configure one or more sensors 20 so as to acquire data 20D at a predetermined rate, e.g., by changing a sample rate, or a resolution of the relevant sensor 20. For example, sensor 20 (e.g., scene sensor 21) may include a stereo camera device, and respective scene data stream 20D (e.g., 21D) may include one or more pairs of images (e.g., a pair of video streams), depicting a scene surrounding platform 11. Sensor configuration module 110 control, or configure scene sensor 21 to acquire the video streams 2 ID at a predetermine frame rate, a predetermined resolution, a predetermined FOV, a predetermined dynamic range (e.g., number of bits representing each brightness level), usingFOR-P-008-PCT a predetermined data compression algorithm (e.g., Joint Photographic Experts Group (JPEG), Moving Picture Experts Group (MPEG)), and the like.

[0067] Additionally, or alternatively, data acquisition module 100 may include a data transfer module 120 adapted to sample, filter and / or adapt sensor(s) data 20D so as to generate and / or transfer data 120D at a filtered form (e.g., at a predetermined volume or rate).

[0068] Pertaining to the example of scene sensor 21 that is a stereo camera device, data acquisition module 100 may obtain scene data stream 21D (e.g., stereo video data stream) from scene sensor 21 (e.g., stereo camera) at a first volume or data rate, and sample scene data stream 21D so as to generate a filtered form of data 20D (e.g., 21D), denoted herein as data 120D, and transfer data 120D to data analysis 200 at a second (e.g., lower) volume or rate.

[0069] Reference is also made to Fig. 4, which is a block diagram, depicting another portion of system 10 (e.g., same as system 10 of Figs. 2 and 3) for scene understanding, according to some embodiments.

[0070] As shown in Figs. 2 and 4, system 10 may further include a data analysis module 200. Data analysis module 200 may obtain one or more sensor data elements 20D such as scene data streams 21D via data acquisition module 100, either in their original form 20D (e.g., 21D, obtained from scene sensors 21), or in filtered form 120D.

[0071] According to some embodiments, data analysis module 200 may include a movement analysis module 290, configured to analyze the at least one scene data stream 20D (21D) or 120D. Based on this analysis, movement analysis module 290 may determine a movement scenario data element 290MS, representing a dynamic relation between mobile platform 11 and its surroundings.

[0072] The term “dynamic” is used here in a sense that movement scenario data element 290MS may describe changes in a scene surrounding platform 11, due to movement of objects in the surroundings, and / or movement of platform 11 itself. Additionally, the term “dynamic” may be used in this context to indicate that movement scenario data element 290MS may be adjusted by system 10 in an iterative manner, e.g., repeatedly over time, to reflect changes in relationship between mobile platform 11 and its surroundings.

[0073] For example, data analysis module 200 may include one or more object detection, or object recognition modules 210. Object recognition module(s) 210 may be configured toFOR-P-008-PCT detect at least one object 210OB in data 21D / 120D (e.g., video data streams), as known in the art. Non-limiting examples for such objects 210OB include pedestrians, animals, vehicles, walls, road signs, road markings, buildings, trees and the like.

[0074] Movement scenario data element 290MS may be a data structure (e.g., one or more tables, a linked list, and the like) that may include a reference to one or more objects 210OB identified in the scene by object recognition module(s) 210.

[0075] Additionally, or alternatively, movement scenario data element 290MS may include spatial information characterizing these objects 210OB. For example, data 21D may originate from a scene sensor 21 such as a stereo-camera device that may produce depthindicative information. Movement scenario data element 290MS may thus include a type of a relevant object 210OB (e.g., a pedestrian, a vehicle, a wall, road signs, road markings, buildings, trees, etc.), and spatial information characterizing that object 210OBJ in relation to platform 11. Such spatial information may include, for example (i) a relative position vector 290PV representing a relative position (e.g., bearing and distance) between at least one relevant object 210OB and platform 11, (ii) a relative motion vector 290MV defining relative motion (e.g., velocity, acceleration, bearing) between at least one relevant object 210OB and platform 11, (iii) a distance between platform 11 and at least one respective object 210OB, and the like.

[0076] In other words, during a current iteration, data analysis module 200 may employ object recognition algorithm 210 on scene data stream 21D / 120D, to identify one or more objects in the scene. Movement analysis module 290 may then predict at least one relative motion vector 290MV between the one or more identified objects 210OBJ and mobile platform 11 in a subsequent iteration, e.g., representing a future location of objects 210OBJ.

[0077] According to some embodiments, movement analysis module 290 may predict a relative motion vector 290MV by analyzing scene data stream 120D to assess a trajectory of a relevant identified obj ect 21 OOB J, in relation to mobile platform 11. Movement analysis module 290 may further obtain a platform movement data element (also referred to as a conduction data element 27D), representing at least one of a position, an orientation, a velocity, an acceleration, a deceleration, a steering angle, and / or a trajectory of mobile platform 11. Movement analysis module 290 may thereby predict the relative motion vector based on (i) the trajectory of the recognized object and (ii) the platform movement data element 27D.FOR-P-008-PCT

[0078] Movement scenario data element may therefore include (i) identification of one or more identified objects 210OBJ (e.g., “car”, “pedestrian”, “tree”) and (ii) the predicted relative motion vector 290MV to that object 210OBJ.

[0079] As shown in Figs. 2, 3 and 4, system 10 may include a dynamic adjustment module 300. As elaborated herein, dynamic adjustment module 300 may obtain one or more movement scenario data element 290MS from data analysis module 200. Based on movement scenario data element 290MS, may control data acquisition module 100 so as to modify a data transfer rate of at least one scene data streams 21D / 120D.

[0080] In a complementary manner, and as elaborated herein, dynamic adjustment module 300 may collaborate with data analysis module 200 to control allocation of one or more computational resources (e.g., of computing device 1 of Fig. 1) based on movement scenario data element 290MS, so as to analyze the at least one scene data streams 21D / 120D.

[0081] For example, object 210OB may be a static object such as a tree, and relative motion vector 290MV may indicate that object 210OB is located behind mobile platform 11, and is relatively moving away from mobile platform 11. Dynamic adjustment module 300 may thereby control data transfer module 120 to reduce sample rate of scene data stream 20D (e.g., 2 ID). Dynamic adjustment module 300 may thus reduce a rate or volume of transferred data 120D that originates from stereo camera 21, pointed at tree object 210OB.

[0082] Dynamic adjustment module 300 may further control allocation of (e.g., deallocate) one or more computational resources (e.g., memory, CPU cores, and the like) of data analysis module 200, to efficiently support analysis of the reduced rate of scene data stream 20D (e.g., 21D) / 120D.

[0083] In another example, object 210OB may be a dynamic object such as a pedestrian, and relative motion vector 290MV may indicate that object 210OB is located in from of mobile platform 11, and is relatively moving toward mobile platform 11. Dynamic adjustment module 300 may thereby control sensor configuration module 110 to increase a sample rate of scene sensor 2 ID, to increase a rate or volume of transferred data 120D that originates from stereo camera 21, pointed at the pedestrian object 210OB.

[0084] Dynamic adjustment module 300 may further control allocation (e.g., allocate) one or more computational resources (e.g., memory, CPU cores, and the like) of data analysis module 200, to efficiently support analysis of the increased rate of scene data stream 20D (e.g., 21D) / 120D.FOR-P-008-PCT

[0085] System 10 may thereby optimize scene data 21D acquisition and analysis according to (a) available computing and communication resources, and (b) according to scene complexity and object relevance. System 10 may proceed to analyze the scene surrounding mobile platform 11 based on the optimized scene data 21D acquisition.

[0086] For example, data analysis module 200 may include a scene model generation module 220, configured to construct a three dimensional (3D) model 220MDL, such as a depth map of the scene surrounding platform 11, based on the one or more scene data streams 2 ID of the modified data transfer rate 120D.

[0087] 3D model 220MDL may be a data structure (e.g., a 3D matrix) that may incorporate movement scenario data element 290MS (e.g., objects 210OB, position vectors 290PV, motion vectors 290MV) as a dynamic, comprehensive 3D representation of platform 11. As elaborated herein, 3D model 220MDL may allow embodiments of the invention to provide various autonomous and / or assistive capabilities for conducting platform 11. Such capabilities may include, for example path planning, collision avoidance, autonomous driving, and the like.

[0088] Additionally, or alternatively, scene model generator 220 may utilize the optimally adjusted data transfer rate of scene data stream 21D / 120D, to understand portions of a scene surrounding platform 11, based on relevance or importance of each portion.

[0089] Pertaining to the examples of a tree obj ect 21 OOB and a pedestrian obj ect 21 OOB : a first portion of 3D model 220MDL may reflect the pedestrian object 21 OOB, and may be characterized by a first, superior spatial resolution. A second portion of 3D model 220MDL may reflect the tree obj ect 21 OOB, and may be characterized by a second, inferior resolution.

[0090] It may be appreciated that the process of constructing a 3D model 220MDL of the scene surrounding mobile platform 11 may be an iterative, or repetitive one, to relate to the dynamic, changing nature of the platform’s 11 surroundings.

[0091] In other words, in each repetition or iteration, system 10 may (a) acquire data streams 20D (21D) from scene sensors 21 mounted on mobile platform 11; (b) analyze at least one of the one or more scene data streams 2 ID, to determine a current movement scenario data element 290MS; (c) based on the movement scenario data element 290MS, configure sensor 21 or modify a data transfer rate 120D of at least one of the one or more scene data streams 2 ID;FOR-P-008-PCT(d) optionally control allocation of one or more computational resources (e.g., memory, CPU cores, and the like), to efficiently support analysis of the scene data stream 20D (e.g., 21D) / 120D; and (e) construct, or update 3D model 220MDL of the scene, based on the one or more scene data streams 120D of the modified data transfer rate.

[0092] As shown in Fig. 2, system 10 may include, or may be communicatively connected to an ADAS system 30. For example, mobile platform 11 may include a terrestrial vehicle, conducted by a human user, and ADAS system 30 may provide assistive driving functionality to the human driver based on 3D model 220MDL. For example, ADAS system 30 may be configured to utilize 3D model 220MDL so as to produce collision warning, when platform 11 appears to approach an object according to 3D model 220MDL.

[0093] Additionally, or alternatively, system 10 may include, or may be associated with (e.g., communicatively connected to) a controller 40 of mobile platform 11. Controller 40 may be configured to control at least one motor or actuator of mobile platform 11 based on the 3D scene model 220MDL. System 10 may thereby provide autonomous driving capabilities for mobile platform 11, while optimizing sensor data acquisition and analysis.

[0094] In other words, system 10 may utilize a minimally required rate, or modified data rate 120D of sensor stream data 20D (e.g., from two or more cameras having an overlapping FOV), to generate a 3D scene model 220MDL, that is a 3D representation of the scene surrounding mobile platform 11. System 10 may subsequently collaborate with controller 40 (e.g., a controller of a steering mechanism, a controller of a breaking mechanism, a controller of an accelerator, and the like) to conduct mobile platform 11 based on the scene model 220MDL.

[0095] According to some embodiments, movement analysis module 290 may calculate, for at least one (e.g., each) of the one or more identified objects 210OBJ, a relevance score 290RLV based on: (i) a type of the identified object and (ii) the predicted relative motion vector 290MV. Movement analysis module 290 may subsequently adjust the movement scenario data element 290MS to include the relevance score 290RLV of the one or more objects 210OBJ. Movement analysis module 290 may adjust the movement scenario data element 290MS to include the one or more object-specific relevance scores 290RLV.

[0096] Data acquisition module 100 may subsequently modify the data transfer rate (e.g., change rate of data stream 120D) of at least one of the one or more scene data streams 20D (e.g., 2 ID) based on the adjusted movement scenario data element 290MS. For example,FOR-P-008-PCT data acquisition module 100 may increase a data transfer rate of data stream 120D for objects having a high relevance score 290RLV, and / or decrease a data transfer rate of data stream 120D for objects having a low relevance score 290RLV. As elaborate herein, dynamic adjustment module 300 may optionally control allocation of one or more computational resources (e.g., memory, CPU cores, and the like), to efficiently analyze the scene data stream 20D (e.g., 21D) / 120D of the modified data transfer rate.

[0097] As elaborated herein, sensors data 20D may include environmental data 25D, indicating environment-related conditions at the location of platform 11. Movement analysis module 290 may adjust, or calculate object-specific 210OBJ relevance score 290RLV further based on the at least one environmental data element. For example, environmental data 25D may indicate a condition of adverse visibility (e.g., poor lighting, precipitation, fog, snow, etc.). Movement analysis module 290 may subsequently increase relevance score 290RLV accordingly, to indicate a necessity for enhanced tracking of the relevant object 210OBJ. Data acquisition module 100 may subsequently modify (e.g., increase) the data transfer rate of data stream 120D of at least one of the one or more scene data streams 2 ID, based on the adjusted (e.g., elevated) relevance score 290RLV. As elaborate herein, dynamic adjustment module 300 may subsequently control allocation of one or more computational resources, to efficiently analyze the scene data stream 20D (e.g., 21D) / 120D of the modified data transfer rate.

[0098] As elaborated herein, sensors data 20D may include at least one diagnostic data 25D, indicating potential issues that may affect performance of mobile platform 11. Movement analysis module 290 may adjust, or calculate object-specific 210OBJ relevance score 290RLV further based on the at least one diagnostic data element. For example, diagnostic data 25D may indicate a condition of poor performance (e.g., poor breaking performance, poor steering performance, etc.) of platform 11. Movement analysis module 290 may subsequently increase relevance score 290RLV accordingly, to indicate a necessity for enhanced tracking of the relevant object 210OBJ. Data acquisition module 100 may subsequently modify (e.g., increase) the data transfer rate of data stream 120D of at least one of the one or more scene data streams 2 ID, based on the adjusted (e.g., elevated) relevance score 290RLV. As elaborate herein, dynamic adjustment module 300 may optionally control allocation of one or more computational resources, to efficiently analyze the scene data stream 20D (e.g., 21D) / 120D of the modified data transfer rate.FOR-P-008-PCT

[0099] Additionally, or alternatively, movement analysis module 290 may adjust, or calculate relevance score 290RLV further based on structural and / or mechanical properties 60 of platform 11.

[0100] For example, data analysis module 200 may obtain (e.g., via input device 7 of Fig. 1) at least one platform attribute data element 60, characterizing platform 11.

[0101] Platform attribute data element 60 may represent at least one dimension, such as a weight, a length, a width, and / or a height of mobile platform 11.

[0102] In another example, platform attribute data element 60 may represent at least one kinematic or navigational requirement, or constraint of mobile platform 11. Kinematic or navigational constraints are limitations imposed by the platform’s 11 design and configuration on its motion. Examples for such requirements or constraints include a minimal turning radius, a maximal steering angle, a track width, limitations on acceleration or deceleration, and the like.

[0103] Movement analysis module 290 may adjust, or calculate relevance score 290RLV further based on the at least one platform attribute data element 60, to allow data acquisition and analysis which is appropriate to the specific platform’s characteristics. For example, when platform 11 is a truck, having a large minimal turning radius, relevance scores 290RLV of peripheral objects 210OBJ may be high, reflected a need to continuously and accurately monitor their position in relation to platform 11. Alternatively, when platform 11 is a robot having a relatively small aspect ratio (and small minimal turning radius), relevance scores 290RLV of peripheral objects 210OBJ may be low, reflected a need to concentrate on monitoring objects that are within a smaller FOV in platform’s 11 direction of movement.

[0104] Additionally, or alternatively, movement analysis module 290 may adjust, or calculate relevance score 290RLV further based on a condition of a road, or terrain through which platform 11 is traversing.

[0105] For example, data analysis module 200 may include a road assessment module 260, adapted to obtain (e.g., via input device 7 of Fig. 1) and / or calculate a numerical road condition value 260RD representing a condition of the road, e.g., where a low road condition value 260RD represent a bumpy, low quality road, and a high road condition value 260RD represent a smooth, high quality road. For example, conduction sensors 27 may include an accelerometer 27, and road assessment module 260 may determine road bumpiness based on the data output 20D of accelerometer sensor 27, to obtain road condition value 260RD.FOR-P-008-PCT

[0106] Movement analysis module 290 may subsequently adjust, or calculate relevance score 290RLV further based on road condition value 260RD, so as to assign a higher relevance score to objects 210OBJ when road conditions are poor (e.g., bumpy road, 260RD low), and assign a lower relevance score to objects 210OBJ when road conditions are good (e.g., smooth road, 260RD high).

[0107] Additionally, or alternatively, movement analysis module 290 may analyze the scene data stream 120D to determine a scene complexity data element 290CPX, representing quantity, distribution, and or relevance of the one or more identified objects 210OBJ.

[0108] For example, a relevance 290RLV of a static object 210OBJ (e.g., a wall, a tree, etc.) whose relative motion vector 290MV does not indicate a collision course with platform 11 (e.g., when platform 11 is conducted in parallel to object 210OBJ) may be assigned a low value. In another example, a relevance 290RLV of a dynamic object 210OBJ (e.g., an animal) whose relative motion vector 290MV indicates high likelihood of collision (e.g., when platform 11 is conducted in a general direction toward object 210OBJ) may be assigned a high value.

[0109] Movement analysis module 290 may subsequently calculate scene complexity data element 290CPX as a function (e.g., a weighted sum) of the number of identified objects 210OBJ and their respective relevance scores 290RLV. E.g., scenes that include a large number of highly relevant 290RLV objects 210OBJ would be assigned a high complexity score 290CPX, whereas scenes that include a small number of low-relevance 290RLV objects 210OBJ would be assigned a low complexity score 290CPX.

[0110] Additionally, or alternatively, movement analysis module 290 may calculate one or more scene complexity data elements 290CPX further based on spatial distribution 290DB of objects 210OBJ.

[0111] For example, scene complexity data element 290CPX may be, or may include a matrix, where each entry may pertain to a specific spatial region or sector surrounding platform 11. Movement analysis module 290 may assign a first (low) value of complexity score to a first entry in complexity data element 290CPX, pertaining to a first region or sector in a scene (e.g., in movement scenario 290MS) where there are only a few objects 210OBJ having low relevance scores 290RLV. In another example, movement analysis module 290 may assign a second (high) value of complexity score in 290CPX to a second entry in complexity data element 290CPX, pertaining to a second region or sector in a scene (e.g., inFOR-P-008-PCT movement scenario 290MS) where there are many objects 210OBJ having high relevance scores 290RLV. Any other combination is also possible.

[0112] According to some embodiments, movement analysis module 290 may adjust the movement scenario data element 290MS to include the one or more scene complexity data elements 290CPX. Data acquisition module 100 may subsequently modify the data transfer rate (e.g., change rate of data stream 120D) of at least one of the one or more scene data streams 20D (e.g., 21D) based on the adjusted movement scenario data element 290MS. As elaborate herein, dynamic adjustment module 300 may optionally control allocation of one or more computational resources, to efficiently analyze the scene data stream 20D (e.g., 2 ID) / 120D of the modified data transfer rate.

[0113] Pertaining to the same example, data acquisition module 100 may modify (e.g., decrease) a rate of data stream 120D originating from a stereo camera sensor 21 directed to the first region or sector in a scene, having low complexity 290CPX (e.g., having not identified any objects 210OB) and / or relevance 290RLV scores. Data acquisition module 100 may do so, for example, by configuring stereo camera sensor 21 to decrease image resolution in the relevant sector or region, or by down-sampling data stream 2 ID originating from stereo camera sensor 21. As elaborate herein, dynamic adjustment module 300 may deallocate one or more computational resources (e.g., CPU 2 cores, memory 5 space), to efficiently analyze the scene data stream 21D of the decreased data transfer rate.

[0114] In another example, data acquisition module 100 may modify (e.g., increase) a rate of data stream 120D originating from a stereo camera sensor 21 directed to the second region or sector in a scene, having high complexity 290CPX (e.g., having identified multiple obj ects 210OB, such as pedestrians) and / or relevance 290RLV scores. Data acquisition module 100 may do so, for example, by configuring stereo camera sensor 21 to increase image resolution in the relevant sector or region, or by avoiding down-sampling of data stream 21D. As elaborate herein, dynamic adjustment module 300 may subsequently allocate additional computational resources (e.g., CPU 2 cores, memory 5 space), to efficiently analyze the scene data stream 2 ID of the increased data transfer rate.

[0115] In another example, data acquisition module 100 may modify acquisition of data that originates from a specific ROI (e.g., section or rectangle) in camera sensor 21. Such modification may be comparable to a fovea in a human retina, having a relatively high density of light receptors. For example, data acquisition module 100 may increase aFOR-P-008-PCT resolution of such a “fovea ROI” of in camera sensor 21, to resemble increased attention towards a specific direction or section (e.g., “tunnel vision”) when driving in a tunnel, or decrease the resolution of the “fovea ROF when driving in an open, residential environment, where a pedestrian might “pop” behind parking vehicles. As elaborate herein, dynamic adjustment module 300 may subsequently allocate (or deallocate) computational resources (e.g., CPU cores, memory space), to efficiently analyze the scene data stream 2 ID of the modified data transfer rate.

[0116] Additionally, or alternatively, data acquisition module 100 may configure at least one sensor 20 (e.g., scene sensor 21), for example in order to modify a rate of data stream 120D, based on a statistical analysis, or a machine-learning (ML) based analysis of movement scenario data element 290MS.

[0117] For example, dynamic adjustment module 300 may include an ML-based model, denoted herein as “sensor ML model 330”.

[0118] During a training stage, system 10 may receive a training dataset 330DS that may include a plurality of annotated movement scenario data element 290MS. Movement scenario data element 290MS of training dataset 330DS may be annotated in a sense that they may be associated with respective requirements for functionality of at least one of the one or more scene sensors 21. Such requirements for functionality may include, for example definition of a direction or sector of interest, a range of interest, a required FOV, a required resolution, a required frame rate, a required image dynamic range, a required image contrast, and the like.

[0119] As known in the art, system 10 may subsequently utilize a training scheme (e.g., a backward propagation scheme), to train ML model 330, while using training dataset 330DS as supervisory information.

[0120] In a subsequent (e.g., inference) stage, ML model 330 may be configured to receive data representing a target movement scenario data element 290MS of interest. Based on the training, ML model 330 may predict, or produce a prediction 330PR that may include a requirement for function (e.g., direction of interest, a range of interest, a required FOV, required resolution, required frame rate, etc.) of at least one of the one or more scene sensors 21.

[0121] In other words, based on its training, ML model 330 may map between (a) characteristics of a scenario data element 290MS (e.g., complexity 290CPX, distributionFOR-P-008-PCT290DB, and relevance 290RLV of a scene, object motion 290MV and / or object position vectors 290PV), and (b) predicted requirements 330PR of functionality of one or more scene sensors 21 of platform 11.

[0122] It may be appreciated that the training stage of ML model 330 may precede a subsequent inference of pretrained ML model 330 on incoming scenario data elements 290MS. Additionally, or alternatively, the training and inference stages of ML model 330 may be intermittent, allowing system 10 to refine the training of ML model 330 over time.

[0123] According to some embodiments, ML model 330 may provide predicted requirements 330PR as a requirement 300RQ for data acquisition module 100. Data acquisition module 100 may, in turn, determine an acquisition parameter 110AC, or a configuration of at least one scene sensor 21 based on the predicted functional requirement 330PR.

[0124] Acquisition parameter 110AC of the one or more scene sensors 21 may include, for example, an operation (e.g., ‘on’ / ‘off) of a specific scene sensors 21.

[0125] Additionally, or alternatively, acquisition parameter 110AC may include a frame rate of a scene sensor 21, a resolution (e.g., a number of pixels per section) of the scene sensor 21, and / or a grey-level quantization or a color channel (e.g., R, G and B) of the scene sensor, dictating a number of colors that represent a scene.

[0126] Additionally, or alternatively, acquisition parameter 110AC may include an FOV of the scene sensor, and / or a definition of a Region Of Interest (ROI), such as a rectangle or a subsection within the FOV of the scene sensor 21.

[0127] Additionally, or alternatively, acquisition parameter 110AC of reflective scene sensors 21 such as radars and LIDARs may include a determined range or depth of the scene sensor 21.

[0128] It may be appreciated that by dynamically adjusting acquisition parameter 110AC (e.g., resolution, frame rate, etc.) of scene sensors 21, embodiments of the invention may modify (e.g., increase or decrease) the data transfer rate of at least one respective scene data stream 2 ID.

[0129] For example, functional requirement 33 OPR (now requirement 300RQ) may include a required FOV that is limited to a specific sector and / or direction, so as to produce a rate-limited data stream 21D. In another example, functional requirement 330PR (now requirement 300RQ) may include a required frame rate or resolution of sensor 21.FOR-P-008-PCT

[0130] Data acquisition module 100 may subsequently configure, or control scene sensor 21 based on the determined acquisition parameter 110AC (e.g., based on configuration 300RQ), thereby modifying the data transfer rate of at least one respective scene data stream 2 ID. As elaborate herein, dynamic adjustment module 300 may subsequently control allocation (e.g., allocate, or deallocate) one or more computational resources (e.g., CPU 2 cores, memory 5 space), to efficiently analyze the scene data stream 2 ID of the modified data transfer rate.

[0131] Additionally, or alternatively, dynamic adjustment module 300 may include a scene complexity prediction module 320, configured to produce a prediction 320PC of complexity of a scene at a location surrounding platform 11, as elaborated herein. Dynamic adjustment module 300 may subsequently provide predicted complexity 320PC as a requirement 300RQ for data acquisition module 100. Data acquisition module 100 may, in turn, determine an acquisition parameter 110AC, or a configuration of at least one scene sensor 21 based on the predicted scene complexity 320PC. Additionally, or alternatively, dynamic adjustment module 300 may collaborate with data analysis module 200 to control allocation of computational resources (e.g., memory 5, CPU cores 2 of Fig. 1, and / or communication channels 20C of Fig. 2) to efficiently analyze data stream 20D (21D) / 120D of sensor 21.

[0132] According to some embodiments, analysis module 200 may include a mapping module 270, adapted to provide (e.g., by use of at least one location sensor 23) at least one location data element 23D (now 270D) indicating a location, and / or orientation of platform 11 in relation to its surroundings. Additionally, or alternatively, location data element 270D may include an indication, or a type of a terrain or surroundings of platform 11.

[0133] Prediction 320PC of scene complexity may thereby be based on mapping, and / or location information 270D. In other words, ML model 320 may employ external maps and / or internally stored historical data to predict a value 320PC of scene complexity based on the location data 270D such as location and orientation of platform 11.

[0134] According to some embodiments, during a training stage, system 10 may receive a training dataset 320DS that may include a plurality of annotated location data element 270D. Data elements 270D of training dataset 320DS may be annotated in a sense that they may be associated with respective values of scene complexity 290CPX (e.g., obtained from movement analysis module 290).FOR-P-008-PCT

[0135] As known in the art, system 10 may subsequently utilize a training scheme (e.g., a backward propagation scheme), to train ML model 320, while using training dataset 320DS as supervisory information.

[0136] In a subsequent (e.g., inference) stage, ML model 320 may be configured to receive data representing a target location data element 270D of interest. Based on the training, ML model 320 may predict, or produce a prediction 320PC of scene complexity corresponding to the incident location data element 270D. In other words, based on its training, ML model 320 may map between (a) location, orientation and time of platform 11, and (b) predicted complexity 320PC.

[0137] According to some embodiments, ML model 320 may subsequently provide predicted complexity 320PC as a requirement 300RQ for data acquisition module 100. Data acquisition module 100 may subsequently determine an acquisition parameter 110AC, or a configuration of at least one scene sensor 21, to adjust a rate of data stream 20D (21D) / 120D of sensor 21 based on the predicted scene complexity 320PC (e.g., higher rate of data stream 120D for a higher predicted scene complexity 320PC value).

[0138] Additionally, or alternatively, dynamic adjustment module 300 may collaborate with data analysis module 200 to control allocation of computational resources (e.g., memory 5, CPU cores 2 of Fig. 1, and / or communication channels 20C of Fig. 2) to efficiently analyze data stream 20D (21D) / 120D of sensor 21.

[0139] In other words, system 10 may apply ML-based model 320 on an incident platform location data element 270D, to produce a predicted scene complexity value 320PC. System 10 may subsequently modify a data transfer rate of at least one of the one or more scene data streams 21D / 120D further based on the predicted scene complexity value 320PC.

[0140] For example, in a first geographical region and time (e.g., an urban center, at rush hour) a scene complexity may be expected to be high, in a sense that object recognition module 210 may be expected to identify an abundance of objects 210OBJ (e.g., cars) of interest. Predicted complexity 320PC in that first geographical region and time may be respectively high. Data acquisition module 100 may receive predicted scene complexity 320PC as a requirement 300RQ, and may thereby configuration at least one scene sensor 21 to provide data stream 21D with a high data rate (e.g., high frame rate, high resolution), corresponding to the predicted high complexity 320PC.FOR-P-008-PCT

[0141] In another example, in a second geographical region and time (e.g., a desert road at night) a scene complexity may be expected to be low, in a sense that object recognition module 210 may not identify many objects 210OBJ (e.g., pedestrians) of interest. Predicted complexity 320PC in the second geographical region and time may be respectively low. Data acquisition module 100 may receive predicted scene complexity 320PC as a requirement 300RQ, and may thereby configuration at least one scene sensor 21 to provide data stream 21D with a low data rate (e.g., low frame rate), corresponding to the predicted low complexity 320PC.

[0142] As shown in Fig. 4, data analysis module 200 may include a resource allocation module 295, configured to control allocation of least one computational resource for analysis of scene data stream 2 ID.

[0143] As explained herein, in a first iteration, movement analysis module 290 may calculate a first movement scenario data element 290MS. The term “iteration” may be used in this context to relate to a specific position or location of platform 11, and / or a specific snapshot (e.g., a frame) of data stream 21D / 120D. Movement scenario data element 290MS of the first iteration may represent a scene, or a portion of a scene that requires a high level of attention or accuracy by system 10 (e.g., as expressed by requirement 300RQ). For example, movement scenario data element 290MS of the first iteration may include, or may be associated with high complexity 290CPX (e.g., a large number of identified objects 210OBJ), a high relevance score 290RLV (highly relevant objects, in the direction of platform 11), proximate objects 210OBJ (e.g., as expressed by position vectors 290PV), objects 210OBJ that are in a collision course with platform 11 (e.g., as expressed by motion vectors 290MV), and the like.

[0144] As elaborated herein, data analysis module 200 may employ data acquisition module 100 to configure relevant sensors 21 to acquire data stream 21D / 120D according to the high required level of accuracy and attention 300RQ (e.g., increase data rate of stream 21D / 120D) in a subsequent iteration (e.g., for a subsequent frame).

[0145] Alternatively, movement scenario data element 290MS of the first iteration may represent a scene, or a portion of a scene that requires a low level of attention or accuracy by system 10 (e.g., as expressed by requirement 300RQ). Data analysis module 200 may subsequently employ data acquisition module 100 to configure relevant sensors 21 to acquire data stream 21D / 120D so as to correspond to the low required level of accuracy and attentionFOR-P-008-PCT300RQ (e.g., decrease data rate of stream 21D / 120D) in a subsequent iteration (e.g., for a subsequent frame).

[0146] In a complementary manner, resource allocation module 295 may allocate computational resources of system 10 (e.g., computing device 1 of Fig. 1) to support analysis of the increased data rate of stream 21D / 120D of the subsequent iteration (e.g., subsequent frame).

[0147] In other words, based on movement scenario data element 290MS, resource allocation module 295 may control allocation of at least one computational resource for analysis of the at least one scene data stream 21D / 120D in the subsequent iteration.

[0148] For example, resource allocation module 295 may allocate, or reallocate memory space (e.g., memory 5 of Fig. 1) for analyzing scene data stream 21D / 120D (e.g., by movement analysis module 290 and / or scene model generator 220), based on movement scenario 290MS (e.g., based on requirement 300RQ).

[0149] In another example, resource allocation module 295 may deallocate, or release memory space (e.g., memory 5 of Fig. 1) for analyzing scene data stream 21D / 120D by movement analysis module 290 and / or scene model generator 220, based on movement scenario 290MS (e.g., based on requirement 300RQ).

[0150] In another example, resource allocation module 295 may allocate, or release, one or more processors (e.g., CPU cores 2 of Fig. 1) to a software process or thread of analyzing scene data stream 21D / 120D (e.g., software processes of movement analysis module 290 and / or scene model generator 220) , based on movement scenario 290MS (e.g., based on requirement 300RQ).

[0151] In yet another example, resource allocation module 295 may allocate, or release, one or more wired, or wireless communication channels 20C of sensors 21, based on movement scenario 290MS (e.g., based on requirement 300RQ).

[0152] Additionally, or alternatively, dynamic adjustment module 300 may include a second ML based model, denoted herein as “resource ML model 310”.

[0153] During a training stage, system 10 may receive a training dataset 310DS that may include a plurality of annotated movement scenario data element 290MS. Movement scenario data element 290MS of training dataset 310DS may be annotated in a sense that they may be associated with respective requirements for computation resource allocation of for analyzing corresponding data streams 21D / 120D. Such requirements for functionalityFOR-P-008-PCT may include, for example definition of a required memory space 5 of Fig. 1, a required number, and / or assignment of CPU cores for software processed of analyzing data streams 21D / 120D by analysis module 200 (e.g., by movement analysis module 290, by scene model generator module 220, etc.), allocation of communication channels 20C, and the like.

[0154] As known in the art, system 10 may subsequently utilize a training scheme (e.g., a backward propagation scheme), to train ML model 310, while using training dataset 310DS as supervisory information.

[0155] In a subsequent (e.g., inference) stage, ML model 310 may be configured to receive data representing a target movement scenario data element 290MS of interest. Based on the training, ML model 310 may predict, or produce a prediction 31 OPR that may include a requirement for allocation of at least one computational resource.

[0156] In other words, based on its training, ML model 310 may map between (a) characteristics of a scenario data element 290MS (e.g., complexity 290CPX, distribution 290DB, and relevance 290RLV of a scene, object motion 290MV and / or object position vectors 290PV), and (b) predicted requirements 31 OPR of allocation of computational resources of platform computing device 1 of system 10.

[0157] It may be appreciated that the training stage of ML model 310 may precede a subsequent inference of pretrained ML model 310 on incoming scenario data elements 290MS. Additionally, or alternatively, the training and inference stages of ML model 310 may be intermittent, allowing system 10 to refine the training of ML model 310 over time.

[0158] According to some embodiments, ML model 330 may provide predicted requirements 31 OPR as a requirement 300RQ for data acquisition module 100. Resource allocation module 295 may (or may not) modify allocation of at least one computational resource (e.g., CPU core, 2, memory 5, communication channel 20C) of system 10, for analyzing the at least one scene data stream 20D (21D) / 120D.

[0159] As known in the art, a tradeoff exists in the field of Artificial Intelligence (Al) and Machine Learning (ML), between generalization of underlying ML functions, and corresponding cost and complexity of ML architectures. For example, in the field of language processing, Large Language Models (LLMs) are designed to understand and generate human language with high proficiency, but typically involve enormous architectures, incorporating billions of parameters. In comparison, Natural LanguageFOR-P-008-PCTProcessing (NLP) models are typically designed for specific tasks (e.g., machine translation), and are typically smaller and focused on performing well on that particular task.

[0160] According to some embodiments, two or more of the ML-based models of dynamic adjustment module 300 (e.g., 310, 320 and / or 330) may be implemented by a single, unified model 300ML. For Example model 300ML may be configured to generate two or more of: (i) a predicted complexity 320PC for data acquisition module 100 (ii) a functional requirement 330PR for sensor configuration module 110, and (iii) a predicted requirement for computational resource 31 OPR (for data acquisition module 100).

[0161] Additionally, or alternatively, one or more (e.g., all) the ML-based models of dynamic adjustment module 300, and object recognition module 210 may be implemented by a single, unified ML-based model 200ML. In other words, ML-based model 200ML may be configured to identify significant objects (e.g., pedestrians on the road), to determine important regions in the scene, and also (i) modify scene sensor acquisition parameters and / or (ii) modify usage of computing resources.

[0162] It may be appreciated that such functionality may be achieved by implementing an attention mechanism in the architecture of unified ML-based model 200ML. Such an architecture may provide superior mapping between data 2 ID representing a scene, and corresponding requirements for (i) computational resources 31 OPR and / or (ii) sensor configuration 33 OPR.

[0163] Reference is now made to Fig. 5, which is a flow diagram, depicting a method of understanding a scene in a multi-sensor platform (e.g., platform 11 of Fig. 2), in an iterative process, by at least one processor (e.g., processor, or controller 2 of Fig. 1), according to some embodiments of the invention.

[0164] As shown in step S 1005, in at least one iteration of the iterative process, the at least one processor 2 may acquire, from one or more scene sensors (e.g., sensors 20 of Fig. 3), mounted on the mobile platform 11, one or more respective scene data streams (e.g., 20D, or 21D of Fig. 3). Scene data streams 20D (21D) may represent a scene surrounding the mobile platform 11.

[0165] As shown in step S 1010, in at least one iteration of the iterative process, the at least one processor 2 may analyze at least one of the one or more scene data streams 20D (2 ID), to determine a movement scenario data element (e.g., 290MS of Fig. 4), representing a dynamic relation between mobile platform 11 and its surroundings.FOR-P-008-PCT

[0166] As shown in step S 1015, in at least one iteration of the iterative process, the at least one processor 2 may modify a data transfer rate of at least one of the one or more scene data streams 20D (21D), based on the movement scenario data element 290MS.

[0167] As shown in step S 1020, in at least one iteration of the iterative process, the at least one processor 2 may constructing a model (e.g., 220MDL of Fig. 4) of the scene, based on the one or more scene data streams of the modified data transfer rate.

[0168] As shown in step S1025, the at least one processor 2 may be associated with a controller (such as controller 2 of Fig. 1) of the mobile platform 11. In at least one iteration of the iterative process, the at least one processor 2 may communicate with the controller of mobile platform 11, to control at least one motor or actuator of the mobile platform 11, so as to conduct the mobile platform based on the scene model 220MDL.

[0169] As elaborated herein, embodiments of the invention may provide a practical application for improving the technology of autonomous vehicles and / or ADAS systems. Embodiments of the invention may further improve computing technology on mobile platforms, by adaptively, and dynamically allocating computing resources, and configuring sensors, based on rapidly (e.g., multiple times per second) changing environments and needs.

[0170] For example, embodiments of the invention may enhance computational efficiency, reducing the processing load of on-board processors, and facilitating real-time data analysis and scene understanding in autonomous vehicles in real time.

[0171] In another example, embodiments of the invention may optimize on-board computer functionality by carefully, and dynamically selecting and allocating computing resources, thereby improving computing power consumption, which is especially beneficial for battery-operated autonomous vehicles.

[0172] Unless explicitly stated, the method embodiments described herein are not constrained to a particular order or sequence. Furthermore, all formulas described herein are intended as examples only and other or different formulas may be used. Additionally, some of the described method embodiments or elements thereof may occur or be performed at the same point in time.

[0173] While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes, and equivalents may occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.FOR-P-008-PCT

[0174] Various embodiments have been presented. Each of these embodiments may of course include features from other embodiments presented, and embodiments not specifically described may include various features described herein.

Claims

FOR-P-008-PCTCLAIMS1. A method of understanding a scene by at least one processor in an iterative process, wherein at least one iteration of the iterative process comprises: acquiring, from one or more scene sensors mounted on a mobile platform, one or more respective scene data streams, representing a scene surrounding the mobile platform; analyzing at least one of the one or more scene data streams, to determine a movement scenario data element, representing a dynamic relation between the mobile platform and its surroundings; based on the movement scenario data element, modifying a data transfer rate of at least one of the one or more scene data streams; and constructing a model of the scene, based on the one or more scene data streams of the modified data transfer rate.

2. The method of claim 1, wherein the at least one processor is associated with a controller of the mobile platform, and wherein said controller is configured to control at least one motor or actuator of the mobile platform, so as to conduct the mobile platform based on the scene model.

3. The method according to any one of claims 1 -2, wherein the one or more scene sensors comprises one or more cameras having timewise-overlapping, or physically-overlapping Field Of View (FOV), and wherein the method further comprises constructing scene model that is a three dimensional (3D) model of the scene, based on the one or more scene data streams of the modified data rate.

4. The method according to any one of claims 1-3, further comprising: applying a first machine-learning (ML) based model on the movement scenario data element to predict a first requirement for function of at least one of the one or more scene sensors; based on the first requirement, determining an acquisition parameter of at least one scene sensor; and controlling the scene sensor based on the determined acquisition parameter, to modify the data transfer rate of at least one respective scene data stream.FOR-P-008-PCT5. The method according to any one of claims 1-4, wherein at least one iteration further comprises: based on the movement scenario data element, controlling allocation of at least one computational resource for analysis of the at least one scene data stream, in a subsequent iteration.

6. The method according to any one of claims 1-5, wherein controlling allocation of at least one computational resource comprises: applying a second ML-based model on the movement scenario data element to predict a second requirement for analysis of at least one scene data stream; and based on the second requirement, modifying allocation of at least one computational resource of the at least one processor, for analyzing the at least one scene data stream.

7. The method according to any one of claims 1-6, wherein analyzing a scene data stream comprises: applying an object recognition algorithm on the scene data stream, to identify one or more objects in the scene; and predicting a relative motion vector between the one or more identified objects and the mobile platform in a subsequent iteration, wherein the movement scenario data element comprises: (i) the identification of the one or more object and (ii) the predicted relative motion vector.

8. The method of claim 7, wherein predicting the relative motion vector comprises: analyzing the scene data streams to assess a trajectory of the identified object in relation to the mobile platform; obtaining a platform location data element, representing at least one of: a position, an orientation, a velocity, an acceleration and a trajectory of the mobile platform; and calculating the relative motion vector based on (i) the trajectory of the recognized object and (ii) the platform location data element.

9. The method according to any one of claims 7-8, wherein analyzing a scene data stream further comprises:FOR-P-008-PCT for each of the one or more identified objects, calculating a relevance score based on: (i) a type of the identified object and (ii) the predicted relative motion vector; and adjusting the movement scenario data element to include the relevance score.

10. The method according to any one of claims 7-9, further comprising: obtaining at least one environmental data element, representing an environment- related condition at a location of the mobile platform; and adjusting the relevance score based on the at least one environmental data element.

11. The method according to any one of claims 7-10, further comprising: obtaining at least one diagnostic data element, indicating a potential issues affecting a performance of the mobile platform; and adjusting the relevance score based on the at least one diagnostic data element.

12. The method according to any one of claims 7-11, further comprising: obtaining at least one platform attribute data element representing at least one of (i) a dimension of the mobile platform and (ii) a navigational requirement of the mobile platform; and adjusting the relevance score based on the at least one platform attribute data element.

13. The method according to any one of claims 7-12, further comprising: analyzing the scene data stream to determine a scene complexity data element, representing a quantity and a distribution of the one or more identified objects; adjusting the movement scenario data element to include the scene complexity data element; and modifying the data transfer rate of at least one of the one or more scene data streams based on the adjusted movement scenario data element.

14. The method of claim 13 further comprising controlling allocation of at least one computing resource, to efficiently analyze the one or more scene data streams, corresponding to the modified data rate.FOR-P-008-PCT15. The method according to any one of claims 13-14, further comprising: applying a third ML-based model on the platform location data element to produce a predicted scene complexity value; and modifying the data transfer rate of at least one of the one or more scene data streams further based on the predicted scene complexity value.

16. The method of claim 15, wherein at least two of the first, second, and third ML-based models are implemented by a single, unified ML model entity.

17. The method according to any one of claims 15-16, wherein the object recognition algorithm, and at least one of the first, second, and third ML-based models are implemented by a single, unified ML model entity.

18. The method according to any one of claims 1-17, wherein the one or more scene sensors are selected from a list consisting of: a single imaging device, one or more single imaging devices, having a timewise overlapping FOV, two or more imaging devices, configured to have a physically -overlapping FOV, one or more LIDAR sensors, and one or more radar sensors.

19. The method according to any one of claims 4-18, wherein the acquisition parameter of the one or more scene sensors is selected from a list consisting of: a frame rate of a scene sensor of the one or more scene sensors, a resolution of the scene sensor, a grey-level quantization of the scene sensor, a quantization of a color channel of the scene sensor, an FOV of the scene sensor, a definition of a Region Of Interest (ROI) within the FOV of the scene sensor, and a range of the scene sensor.

20. A system for understanding a scene, the system comprising: a non-transitory memory device, wherein modules of instruction code are stored, and at least one processor associated with the memory device, and configured to execute the modules of instruction code, whereupon execution of said modules of instruction code, the at least one processor is configured to execute an iterative process, wherein each iteration of the iterative process comprises:FOR-P-008-PCT acquiring, from one or more scene sensors mounted on a mobile platform, one or more respective scene data streams, representing a scene surrounding the mobile platform; analyzing at least one of the one or more scene data streams, to determine a movement scenario data element, representing a dynamic relation between the mobile platform and its surroundings; based on the movement scenario data element, modifying a data transfer rate of at least one of the one or more scene data streams; and constructing a model of the scene, based on the one or more scene data streams of the modified data transfer rate.

21. The system of claim 20, wherein the at least one processor is further associated with a controller of the mobile platform, and wherein said controller is configured to control at least one motor or actuator of the mobile platform, so as to conduct the mobile platform based on the scene model.

22. The system according to any one of claims 20-21, wherein the one or more scene sensors comprises one or more cameras having timewise-overlapping, or physically- overlapping Field Of View (FOV), and wherein the at least one processor is further configured to construct scene model that is a three dimensional (3D) model of the scene, based on the one or more scene data streams of the modified data rate.

23. The system according to any one of claims 20-22, wherein the at least one processor is further configured to: apply a first machine-learning (ML) based model on the movement scenario data element to predict a first requirement for function of at least one of the one or more scene sensors; based on the first requirement, determine an acquisition parameter of at least one scene sensor; and control the scene sensor based on the determined acquisition parameter, to modify the data transfer rate of at least one respective scene data stream.FOR-P-008-PCT24. The system according to any one of claims 20-23 , wherein at least one iteration further comprises: based on the movement scenario data element, controlling allocation of at least one computational resource for analysis of the at least one scene data stream, in a subsequent iteration.

25. The system according to any one of claims 20-24, wherein controlling allocation of at least one computational resource comprises: applying a second ML-based model on the movement scenario data element to predict a second requirement for analysis of at least one scene data stream; and based on the second requirement, modifying allocation of at least one computational resource of the at least one processor, for analyzing the at least one scene data stream.

26. The system according to any one of claims 20-25, wherein said at least one processor if configured to analyze a scene data stream by: applying an object recognition algorithm on the scene data stream, to identify one or more objects in the scene; and predicting a relative motion vector between the one or more identified objects and the mobile platform in a subsequent iteration, wherein the movement scenario data element comprises: (i) the identification of the one or more object and (ii) the predicted relative motion vector.

27. The system of claim 26, wherein predicting the relative motion vector comprises: analyzing the scene data streams to assess a trajectory of the identified object in relation to the mobile platform; obtaining a platform location data element, representing at least one of: a position, an orientation, a velocity, an acceleration and a trajectory of the mobile platform; and calculating the relative motion vector based on (i) the trajectory of the recognized object and (ii) the platform location data element.

28. The system according to any one of claims 26-27, wherein said at least one processor if configured to analyze a scene data stream further by:FOR-P-008-PCT for each of the one or more identified objects, calculating a relevance score based on: (i) a type of the identified object and (ii) the predicted relative motion vector; and adjusting the movement scenario data element to include the relevance score.

29. The system according to any one of claims 26-28, wherein said at least one processor is further configured to: obtain at least one environmental data element, representing an environment-related condition at a location of the mobile platform; and adjust the relevance score based on the at least one environmental data element.

30. The system according to any one of claims 26-29, wherein said at least one processor is further configured to: obtain at least one diagnostic data element, indicating a potential issues affecting a performance of the mobile platform; and adjust the relevance score based on the at least one diagnostic data element.

31. The system according to any one of claims 26-30, wherein said at least one processor is further configured to: obtain at least one platform attribute data element representing at least one of (i) a dimension of the mobile platform and (ii) a navigational requirement of the mobile platform; and adjust the relevance score based on the at least one platform attribute data element.

32. The system according to any one of claims 26-31, wherein said at least one processor is further configured to: analyze the scene data stream to determine a scene complexity data element, representing a quantity and a distribution of the one or more identified objects; adjust the movement scenario data element to include the scene complexity data element; and modify the data transfer rate of at least one of the one or more scene data streams based on the adjusted movement scenario data element.FOR-P-008-PCT33. The system of claim 32, wherein said at least one processor is further configured to control allocation of at least one computing resource, to efficiently analyze the one or more scene data streams, corresponding to the modified data rate.

34. The system according to any one of claims 32-33, wherein said at least one processor is further configured to: apply a third ML-based model on the platform location data element to produce a predicted scene complexity value; and modify the data transfer rate of at least one of the one or more scene data streams further based on the predicted scene complexity value.

35. The system of claim 34, wherein at least two of the first, second, and third ML-based models are implemented by a single, unified ML model entity.

36. The system according to any one of claims 34-35, wherein the object recognition algorithm, and at least one of the first, second, and third ML-based models are implemented by a single, unified ML model entity.

37. The system according to any one of claims 20-36, wherein the one or more scene sensors are selected from a list consisting of: a single imaging device, one or more single imaging devices, having a timewise overlapping FOV, two or more imaging devices, configured to have a physically -overlapping FOV, one or more LIDAR sensors, and one or more radar sensors.

38. The system according to any one of claims 23-37, wherein the acquisition parameter of the one or more scene sensors is selected from a list consisting of: a frame rate of a scene sensor of the one or more scene sensors, a resolution of the scene sensor, a grey-level quantization of the scene sensor, a quantization of a color channel of the scene sensor, an FOV of the scene sensor, a definition of a Region Of Interest (ROI) within the FOV of the scene sensor, and a range of the scene sensor.FOR-P-008-PCT39. A mobile platform comprising: at least one motor or actuator, a controller configured to control the at least one motor or actuator, one or more scene sensors, and at least one processor, wherein the at least one processor is configured to perform an iterative process, and wherein at least one iteration of the iterative process comprises: acquiring, from the one or more scene sensors, one or more respective scene data streams, representing a scene surrounding the mobile platform; analyzing at least one of the one or more scene data streams, to determine a movement scenario data element, representing a dynamic relation between the mobile platform and its surroundings; based on the movement scenario data element, modifying a data transfer rate of at least one of the one or more scene data streams; constructing a model of the scene, based on the one or more scene data streams of the modified data transfer rate; and communicating said controller, to control the at least one motor or actuator, so as to conduct the mobile platform based on the scene model.

Images