Method and system for determining whether a device is ready to move

Abstract

To provide a computer-implemented method that determines whether a device is ready to move.SOLUTION: The present invention relates to a computer-implemented method that determines whether a device is ready to move, and the computer-implemented method includes: acquiring a series of images related to the device (10); extracting motion information from the series of images (12), the motion image having information related to a motion of at least one object in the series of images; and determining whether the device is ready to move based upon the motion information (28).SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] 1. Field of the Invention The present disclosure relates to the field of automation, and in particular to a method and system for determining whether a device is ready to move. The method and system may find application in all kinds of industries, including the automotive industry. [Background technology]

[0002] 2. Description of Related Technology In recent years, there has been an increasing demand for automation across all areas of technology. In particular, the field of autonomous vehicles has experienced significant developments to partially or fully automate driving tasks typically performed by a human driver. However, in vehicles, and more specifically in shared vehicles such as buses, shuttles, trains, etc., the driver (or in some cases, the conductor or pilot) typically performs tasks other than the pure driving task, such as managing the opening and closing of doors, determining whether the vehicle can depart, assisting disabled persons, handling emergency situations, and the like.

[0003] Therefore, there is a need for more automation, particularly with regard to determining whether a vehicle, or more generally a device, is ready to move.

[0004] The following references disclose methods related to the field of computer vision. [Reference 1] Zheng Ge et al., YOLOX: Exceeding YOLO Series in 2021, arXiv:2107.08430v2 [Reference 2] Nicolai Wojke et al., Simple Online and Realtime Tracking with a Deep Association Metric, arXiv:1703.07402v1 [Reference 3] Liang-Chieh Chen et al., Rethinking Atrous Convolution for Semantic Image Segmentation, arXiv:1706.05587v3 [Reference 4] D. Vega-Hernandez, AM Herrera-Navarro, and H. Jimenez-Hernandez, Background Subtraction Model Based on Adaptable MOG, 2012 IEEE Ninth Electronics, Robotics and Automotive Mechanics Conference, Cuernavaca, Mexico, 2012, pp.54-59, doi:10.1109 / CERMA.2012.17 [Reference 5] D. Fierens, G. Van den Broeck, J. Renkens, D. Shterionov, B. Gutmann, I. Thon, G. Janssens, and L. De Raedt, Inference and learning in probabilistic logic programs using weighted Boolean formulas, Theory and Practice of Logic Programming, 15:3, pp.358-401, Cambridge University Press, 2015. Summary of the Invention

[0005] In this regard, the present disclosure relates to a computer-implemented method for determining whether a device is ready to move, the computer-implemented method comprising: acquiring a series of images of the device; extracting motion information from the sequence of images, the motion information comprising information about the motion of at least one object in the sequence of images; determining whether the device is ready to move based on the motion information; Includes.

[0006] The steps of the method may be performed by one or more processors. The method is hereinafter referred to as a determination method.

[0007] An apparatus may have one or more moving parts. "An apparatus is ready to move" means that at least one of the one or more moving parts is ready to move.

[0008] Acquiring the series of images may include, for example, acquiring the series through an image acquisition module such as a camera, or obtaining an already acquired series from a database, for example, a local or remote server or the like.

[0009] Additionally, the acquiring step may include any desired pre-processing of the series of images, such as rectifying, cropping, resizing, rotating, etc., or combinations thereof.

[0010] A sequence of images comprises a plurality of images. A sequence of images may comprise a plurality of consecutive frames of a video clip. Alternatively, a sequence of images may comprise a plurality of non-consecutive frames of a video clip, for example, selected at a given sampling frequency (e.g., every third frame of the video clip). A sequence of images need not be sampled at a regular frequency.

[0011] A series of images related to a device means that the series of images shows at least a portion of the device and / or at least a portion of the device's environment with which the device may interact. For example, if the device is a vehicle, the series of images may show the vehicle cabin or vehicle surroundings. If the device is a robotic system, the series of images may at least partially show the range of motion of the robotic system. By monitoring the series of images related to the device, the determination method can determine whether allowing movement of the device will have any undesirable impact on its environment.

[0012] Specifically, the determination method extracts motion information from the sequence of images. The motion information comprises information about the motion of at least one object shown in some or all of the sequence of images. In this disclosure, object should be understood as a general term meaning any obstacle to the device, including a living being (e.g., human, animal, plant) or object. The information about motion may comprise position in one or more dimensions, velocity, acceleration, or any combination thereof. Alternatively or additionally, the information about motion may comprise an indicator that qualifies the amount and / or type of motion of the object.

[0013] Hereinafter, unless otherwise stated, an "object" refers to at least one object. More generally, as used herein, and for brevity unless the context indicates otherwise, "a," "an," and "the" refer to "at least one" or "each," and are intended to include the plural form as well. Conversely, the common use of the plural also includes the singular form.

[0014] The motion information is taken into account to determine whether the device is ready to move. When the motion information is extracted from a series of images, the reliability of the determination method is improved. By checking the motion around and / or within the device, it is ensured that the initial movement of the device does not cause major surprises or trigger any accidents. Therefore, the proposed determination method provides suitable automation to determine whether the device is ready to move, reducing the need for human supervision. Based on this determination, further actions, such as the actual movement of the device, can be taken.

[0015] Optionally, the extracting includes detecting at least one object in the sequence of images and estimating the motion of the at least one object based on the sequence of images. Object detection per se is known in the art. Estimating the motion of the detected object may rely on the evolution of geometric properties of the object in the sequence of images, for example, one or more feature points of the object in the object's bounding box, the object's center of gravity, etc.

[0016] Optionally, the estimation includes deriving a global motion from the sequence of images and associating one or more parts of the global motion with one or more of the detected at least one object. In this way, instead of first determining the objects and then estimating the motion of each of the objects individually, motion information can be determined more accurately even when objects overlap.

[0017] Optionally, the global motion is derived based on a semantic mask obtained from the sequence of images. The semantic mask is a mask, i.e., an element that reveals a part of the image while masking other pixels of the image, the revealed part being determined based on a semantic segmentation of the image. Semantic segmentation and mask calculation itself are known in the field of image processing and are easy to use. As a result, the proposed method is efficient and robust.

[0018] Optionally, the overall motion comprises a map of the rate or amount of motion throughout the sequence of images. The map may assign a rate / amount of motion to each pixel or area of ​​each image. By assigning a rate / amount of motion to pixels or areas of an image instead of assigning motion at the object level, cases can be properly identified where sub-portions of an object have excessive motion while the overall movement of the object is limited. As a result, the use of such a map improves the accuracy of the method.

[0019] Optionally, the at least one object comprises at least one of a human being, an animal, a plant, or an object.

[0020] Optionally, the device is a vehicle or manufacturing equipment. Vehicles include, but are not limited to, cars, buses, shuttles, trains, elevators, boats, airplanes, cable transport, etc. Manufacturing equipment includes, but is not limited to, production lines, robotic systems, etc. Accidental movement of vehicles or manufacturing equipment can have dramatic consequences. Therefore, the decision method improves the safety of these devices.

[0021] Optionally, the method comprises tracking at least one object throughout the sequence, which may be done by tracking any relevant parts and / or properties of the object, for example the bounding box of the object.

[0022] Optionally, the method further comprises running a classification model to output, based on the motion information, an indicator representing whether the at least one object is moving in a manner that makes movement of the device unsafe.

[0023] The classification model may be a machine learning model, such as a decision tree, or another type of model. The classification model may take the motion information as input and output an indicator.

[0024] The indicator represents the degree to which the object's motion reduces the object's readiness for (possibly abrupt) movement of the device. For example, if it is determined that the object moves a lot, the object may be deemed unsafe for movement of the device because it is not stable and may tip over or inside out due to movement of the device. Conversely, object motion that does not affect the object's stability may not be deemed to make movement of the device unsafe. If the indicator indicates that movement of the device is unsafe, it may be determined that the device is not ready to move. The classification model allows the determination method to make a clear decision on whether the device is ready to move.

[0025] Optionally, the method includes filtering out positional motions from the motion information. As a result, positional motions are ignored from the motion information. In this way, the motion information may more accurately convey information about the stability of the object. Positional motions include object motions that do not change at least one of the object's overall position, the object's center of gravity, or the object's attitude (e.g., angular position) by more than a threshold. For example, actions such as reading or waving to someone may qualify as positional motions.

[0026] Optionally, the determining includes outputting at least one probability representing whether the device is ready to move. By outputting a probability, the determining method can be easily incorporated into other systems, making operation of the device easier. Instead of a probability, the determining may include outputting a binary flag.

[0027] Optionally, the determination includes applying a logic-based algorithm to the plurality of pieces of motion information. The logic-based algorithm is, for example, a rule-based and / or probability-based algorithm that applies the rules and / or probabilities to multiple inputs, i.e., the plurality of pieces of motion information, to determine an output. By combining a motion detection method and a logic-based algorithm based on the plurality of pieces of motion information, the method is not only more reliable because it is based on multiple detections, but also simpler because the pieces of motion information are combined by the logic-based algorithm. This avoids the need for complex models to handle a wide variety of situations. In contrast, combining the pieces of motion information through a logic-based algorithm can leverage the benefits of simple detection and the robustness of logic-based reasoning, bringing human insight to the method. This architecture also makes the determination method self-adaptive, for example, to unknown types of movement.

[0028] Optionally, the decision making method is performed in real time to determine whether the device is ready to move based on current conditions regarding the device's surrounding environment.

[0029] The present disclosure further provides a system for determining whether a device is ready to move, comprising: an acquisition module for acquiring a series of images of the device; an extractor for extracting motion information from the sequence of images, the motion information comprising information about the motion of at least one object in the sequence of images; a decision module that determines whether the device is ready to move based on the motion information; The present invention relates to a system comprising:

[0030] A system, hereinafter referred to as a determination system, may be configured to perform the above-described determination method and may have some or all of the above-described features. The determination system may have a computer hardware structure.

[0031] The present disclosure further relates to a vehicle comprising the above-mentioned determination system and a camera, wherein the acquisition module is configured to acquire a series of images from the camera, which may be an interior camera, i.e. a camera that acquires images of the interior of the vehicle, for example the interior of the vehicle cabin.

[0032] Optionally, the vehicle is an autonomous vehicle. The decision system may implement non-driving functions of the autonomous vehicle.

[0033] The present disclosure further relates to a computer program set comprising instructions for performing the steps of the above-described determination method when said program is executed by at least one computer, said set of programs may be in any programming language, in source code, object code, or code intermediate between source code and object code, for example in a partially compiled form, or in any other desired form.

[0034] The present disclosure further relates to at least one computer-readable recording medium having recorded thereon at least one computer program comprising instructions for performing the steps of the above-mentioned determining method. The recording medium may be any entity or device capable of storing a program. For example, the medium may comprise a storage means such as a ROM, e.g., a CD ROM or a microelectronic circuit ROM, or a magnetic storage means, e.g., a diskette (floppy disk) or a hard disk.

[0035] Alternatively, the recording medium may be an integrated circuit having the program embedded therein, the circuit being configured to perform, or be used in the performance of, the method. [Brief explanation of the drawings]

[0036] The features, advantages, and technical and industrial importance of preferred embodiments of the present invention will be described below with reference to the accompanying drawings, in which like reference numerals refer to like elements.

[0037] [Figure 1] 1 is a schematic flow chart of a method for determining whether a device is ready to move, according to an embodiment. [Figure 2] 1 is a detailed flowchart of a method for determining whether a device is ready to move, according to an embodiment. [Figure 3] FIG. 1 is a side view of a vehicle according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0038] A method for determining whether a device is ready to move will be described with reference to Figures 1 and 2. In this embodiment, the device is a vehicle. However, the following description may be translated to other types of devices, such as manufacturing equipment. Figure 1 shows an overview of the method, while Figure 2 shows the embodiment in more detail.

[0039] As described above with reference to Figure 1, the determination method includes step 10 of acquiring a series of images of the device. Step 10 may include any suitable pre-processing of the images to make them ready for the next steps in the determination method.

[0040] The determination method further includes a step 12 of extracting motion information from the sequence of images. The motion information comprises information about the motion of at least one object in the sequence of images. In the following, for simplicity, it is assumed that the object is a person, but the object may also be an animal, a plant, or an object. When an image may show multiple objects, the extracted motion information may relate to multiple objects. Furthermore, the extraction may target specific objects, such as humans, pets, luggage, etc., while treating other objects (e.g., seats, seat belts, retention bars, etc.) as background.

[0041] In a possible application, the determination method is configured to determine, based on a series of images of the interior of a vehicle, whether the vehicle occupants are sufficiently secured or stable for the vehicle to depart without risking one of the vehicle occupants falling. Although departure is mentioned, other types of movement are encompassed, such as accelerating, braking, turning, etc. More generally, the movement involved will depend on the type of device.

[0042] For example, step 12 may include an object detection step 14 and a motion calculation step 20. Object detection step 14 aims to detect at least one object (e.g., a person) in the sequence of images. For example, object detection step 14 may include a step 16 of identifying one or more objects in the sequence of images and, optionally, a step 18 of tracking the identified objects throughout the sequence of images.

[0043] Furthermore, the motion calculation step 20 may include deriving a global motion from the sequence of images. In particular, the motion calculation step 20 may include a step 22 of semantic segmentation of the sequence of images to separate objects in the images from one another, and a step 24 of estimating the global motion of the segmented objects.

[0044] The determination method may take the output of the object detection step 14, i.e. the location of the object in the sequence of images, and the output of the motion calculation step 20, i.e. the aforementioned overall motion, and associate one or more parts of the overall motion with one or more of the at least one detected object in a motion association step 26. The motion association step 26 outputs motion information.

[0045] Then, in step 28, based on the motion information, the decision method determines whether the device is ready to move. For example, the decision method, e.g., step 28, may include step 30 of filtering out stationary motion from the motion information output from step 26, e.g., to retain only motion information related to stability.

[0046] Further, the decision method, e.g., step 28, may include step 32 of running a classification model on the motion information output from step 28 to classify movements according to their impact on the decision to allow or disallow movement of the device. For example, the classification model may classify movements into stability-related and non-stability-related movements based on whether the motion information affects the stability of the person. The classification model may output an indicator representing whether at least one object is moving in a manner that makes movement of the device (e.g., vehicle departure) unsafe.

[0047] The outputs of each of steps 30, 32 may be combined to determine whether the device is ready to move in step 34. Step 34 may produce a flag or probability 36 indicating whether the device is ready to move. Thus, vehicle departure may be determined automatically and reliably.

[0048] An embodiment of the determination method will be described in more detail with reference to Figure 2. However, as can be seen from Figure 1, the architecture of the proposed determination method is modular, and variations of each module can be suitably used by those skilled in the art without departing from the principles expressed herein. As detailed below, the determination method may include using at least one machine learning model.

[0049] 2 shows an example of performing step 10 of acquiring a series of images of a device. Here, a plurality of video frames 40 can be acquired using, for example, a camera. The camera can be a fisheye camera. Fisheye cameras are known in the art and allow for capturing a large field of view. However, other types of cameras are envisioned. In the example of a vehicle, the camera can be mounted inside the vehicle. The camera can view the interior cabin of the vehicle, for example, to monitor the movements of occupants in order to perform a decision method.

[0050] The series of images can be RGB (red / green / blue) images, or more generally, single-channel or multi-channel images. An RGB image of a scene comprises pixels having red, green, and blue values ​​to form colors that represent the colors of the scene (including black and white or grayscale). A single-channel image is an image in which each pixel has a color or intensity represented by a single value, while a multi-channel image is an image in which each pixel has a color or intensity represented by multiple values. Grayscale is an example of a single-channel scale, while RGB is an example of a multi-channel scale.

[0051] In step 42, a number of video frames 40 are acquired. Step 42 may be performed in real time. Step 42 may acquire all available video frames 40, or may sample the video frames 40 at any desired rate.

[0052] In some cases, the determination method may include a step 44 of pre-processing the sequence of images. For example, step 44 may include reprojecting or rectifying the sequence of images, e.g., to convert images from fisheye images into rectified images. Other operations such as cropping, resizing, rotating, compressing, etc. are possible, alone or in combination. Step 44 outputs pre-processed images 46 (e.g., rectified images) taking into account the extraction of motion information therefrom.

[0053] Specifically, the preprocessed image 46 is output to an object detection step 14 and a motion calculation step 20. Although the object detection step 14 and the motion calculation step 20 are depicted as being parallel and independent of each other, the object detection step 14 and the motion calculation step 20 may be performed sequentially.

[0054] As indicated above, in object detection step 14, the decision method may first proceed to step 16, which involves identifying one or more objects in a sequence of images, here preprocessed images 46. Step 16 may be applied individually to each preprocessed image 46. Step 16 may rely on an object detector model, e.g., a human detector such as YoloX [Reference 1]. Many object detector models, such as YoloX, are trained based on rectified images, so in such cases, the fisheye images need to be rectified. However, if the object detector model can function from the fisheye images, e.g., by fine-tuning the images, rectification is not necessary. Therefore, preprocessing step 44 is optional and essentially depends on the requirements of step 16, which involves identifying one or more objects in a sequence of images.

[0055] The object detector model may output characteristics, such as contours, centroids, etc., that allow identification of objects in the preprocessed images 46. In this example, the object detector model outputs a bounding box 17 for each object in each image. The bounding box 17 may be a two-dimensional or three-dimensional bounding box.

[0056] The bounding boxes 17 serve as input to step 18, which tracks the identified objects throughout the sequence of images. Step 18 may rely on an object tracker model such as DeepSort [Reference 2]. Tracking the bounding boxes (or any other characteristic that describes the detected objects) makes it possible to establish relationships between the bounding boxes from one frame to another. In other words, it makes it possible to determine which objects are moving.

[0057] Thus, in this embodiment, step 18 outputs a bounding box of the tracked person 19. More generally, step 14 outputs object detections in the sequence of images.

[0058] As indicated above, in the motion calculation step 20, the determination method may first proceed to a step 22 of semantic segmentation of a sequence of images, here preprocessed images 46. Step 22 may rely on a semantic segmentation model such as DeepLabv3 [Reference 3]. In particular, the semantic segmentation model may be configured to output a semantic mask 23 obtained from the sequence of images, in particular a respective semantic mask 23 for each image of the sequence of images. The semantic mask 23 may be a binary mask. In particular, the semantic mask may reveal objects (e.g., people) detected in the image while masking everything else, such as the walls, floor, seats, and windows of the vehicle cabin.

[0059] The semantic mask 23 serves as input to a step 24 of estimating the global motion of the segmented object. The global motion may comprise a map of the rate of motion or amount of motion throughout the sequence of images.

[0060] For example, to obtain such a map, step 24 may include a step 50 of subtracting the semantic mask of one image from the semantic mask of an adjacent image (e.g., the one immediately preceding or following it in a sequence of images). The displacement between two adjacent semantic masks indicates which pixels of the detected object have moved between the two images. This difference therefore provides information about the overall motion of the detected object. Obtaining the map based on the semantic mask makes it possible to constrain the area over which the movement needs to be calculated. This improves the robustness of the motion estimation, since (irrelevant) background motion is filtered out.

[0061] Step 50 may rely on a background subtraction model such as MOG [Reference 4]. In particular, step 50 may output a global motion map 52.

[0062] Step 24 may further include a step 54 of filtering the global motion map 52. A semantic mask 23 is applied to the global motion map 52 to filter out light changes and sudden environmental motion. In doing so, step 54 outputs a more robust motion map 56, assigning an amount or percentage of motion to a particular person for each image. Thus, the motion map 56 provides the global motion of the segmented objects.

[0063] In this embodiment, the motion association step 26 described above may take as input the motion map 56 and the tracked person's bounding box 19, and associate one or more portions of the overall motion described by the motion map 56 with one or more of the detected objects described by the tracked person's bounding box 19. Thus, the association step may combine pixel-by-pixel information from the motion map 56 with region-based detection from the tracked person's bounding box 19 to derive the amount of motion at (an individual).

[0064] Thus, the determination method may estimate the motion of at least one object based on a sequence of images.

[0065] The determination method then proceeds to step 30, where fixed-position motion is filtered out. As described above, fixed-position motion refers to object movement that does not change the overall position of the object. For example, object movement may be determined to be fixed-position motion if the amount or ratio of bounding box displacement is less than a threshold. For example, the ratio of displacement for a bounding box may be calculated as 1-IoU, where IoU refers to the intersection-over-union ratio of bounding box areas in two consecutive frames. Conversely, if the amount or ratio of displacement is higher than a threshold, it may be determined that an object (e.g., a person) is moving overall within the vehicle.

[0066] In parallel, the decision method may proceed to step 32 of executing a classification model. In this example, the classification model is a decision tree. For example, step 32 may use a custom-built decision tree, which is itself known by those skilled in the art. The decision tree may be a machine learning model that is configured to take into account the estimated motion and output, based on the estimated motion (corresponding to the motion information), an indicator representing whether at least one object is moving in a way that makes movement of the device unsafe.

[0067] In an example, the estimated motion comprises a mask, and the classification model may be configured to determine a contour of pixels indicated by the mask and derive an area defined within this contour, whereupon the classification model may be trained to determine the indicators described above.

[0068] In step 34, the decision method determines whether the device is ready to move. Essentially, step 34 combines the indicators provided by the classification model in step 32 with the motion-in-place filtered motion information of step 30 to limit the influence of motion-in-place on the method's decisions.

[0069] In an example, step 34 may include applying a logic-based algorithm to the plurality of motion information, i.e., the indicators of step 32 and the in-place motion-filtered motion information of step 30. The logic-based algorithm may be a probabilistic inference with appropriate inference rules. For example, step 34 may rely on a logic-based engine such as ProbLog [Reference 5]. However, other logic-based algorithms may be used, for example, based on binary (Boolean) logic, fuzzy logic, etc.

[0070] For example, the reasoning rules may describe a probabilistic model that makes decisions based on multiple pieces of information. In the example, the decision tree decision (i.e., the indicator of step 32) is represented by two predicates: dTREE_GO, with a probability of, for example, 0.5, and dTREE_STOP, with a probability of, for example, 0.5. These predicates represent the likelihood of a method that determines whether the device is ready to move based on the decision tree model. The bounding box reasoner decision (i.e., the motion information filtered by the in-place motion of step 30) is represented by three predicates: bReasoner_INPLACE, bReasoner_MOVINGPERSON, and bReasoner_NEWPERSON, with probabilities of, for example, 0.4, 0.4, and 0.2, respectively. These predicates represent the method's evaluation of the current situation based on the bounding box reasoner, which can evaluate the position or movement of a person in the scene.

[0071] In this example, the reasoning rules may combine the decision tree and bounding box reasoner judgments to make a final decision on how to make the decision. This final decision may be represented by two predicates, ai_conductor_GO and ai_conductor_STOP, which represent how to determine whether the device is ready to move or not, respectively. These rules combine various predicates from the decision tree and bounding box reasoner with various probabilities, resulting in a set of possible outputs. For example, if the decision tree suggests that the device is ready to move and the bounding box reasoner detects a person in position (i.e., undergoing a positional motion), there is a 0.9 probability, for example, that the AI ​​system will proceed. This combination allows the decision method to make more nuanced decisions based on both the decision tree's analysis of the situation and the bounding box reasoner's evaluation.

[0072] Step 34 may output any type of decision regarding whether the device is ready to move, for example, at least one probability representing whether the device is ready to move.

[0073] Experiments conducted by the inventors show that the proposed decision method achieved superior judgment with an accuracy of over 80% in 200 real-world videos, where superior judgment means that the decision method accurately determined whether the vehicle was ready to depart compared to what a human supervisor concluded.

[0074] 3 shows a vehicle 80 comprising a camera 82 and a decision system 84 for determining whether the vehicle 80 is ready to move. The decision system comprises an acquisition module 86 for acquiring a series of images relating to the vehicle 80, an extractor 88 for extracting motion information from the series of images, the motion information comprising information relating to the motion of at least one object in the series of images, and a decision module 90 for determining whether the vehicle 80 is ready to move based on the motion information.

[0075] The determination system 84 may comprise electronic circuitry, a processor (shared, dedicated, or group), combinatorial logic, memory for executing one or more software programs, and / or other suitable components for providing the described functionality. In other words, the determination system 84 may be a computing device. The determination system 84 may be connected to a memory (storage medium) that may store data, such as a computer program, that, when executed, performs the assistance method according to the present disclosure.

[0076] The determination system 84, in particular the acquisition module 86, has an input for receiving a digital image or a stream of digital images. In particular, the determination system 84, in particular the acquisition module 86, may be connected to an optical sensor, in particular one or more digital cameras, for example the above-mentioned camera 82, which may be a fisheye camera. The digital cameras are configured to be able to record a scene, in particular to be able to output digital data to the determination system 84, in particular the acquisition module 86.

[0077] If the determination method is performed in real time, the at least one image may be acquired in real time by an image acquisition module, such as a video acquisition module, for example a camera 82 or the like (video camera, photo camera, etc.). Alternatively or additionally, for example in the case of post-processing in a film process, the at least one image may be acquired in advance and later acquired by the determination system 84.

[0078] While this disclosure refers to specific preferred embodiments, modifications to those examples may be provided without departing from the overall scope of the invention as defined by the claims. In particular, individual features of the various embodiments shown / described may be combined in additional embodiments. Accordingly, the description and drawings should be regarded in an illustrative, rather than a restrictive, sense. The invention disclosed in this specification includes the following aspects. [Aspect 1] 1. A computer-implemented method for determining whether a device is ready to move, the computer-implemented method comprising: acquiring a series of images of the device; extracting motion information from the sequence of images, the motion information comprising information about the motion of at least one object in the sequence of images; determining whether the device is ready to move based on the motion information; 11. A computer-implemented method comprising: [Aspect 2] 2. The method of claim 1, wherein the extracting comprises detecting the at least one object in the series of images and estimating motion of the at least one object based on the series of images. Aspect 3 The method of aspect 2, wherein the estimation includes deriving an overall motion from the series of images and associating one or more portions of the overall motion with one or more of the detected at least one object. Aspect 4 4. The method of claim 3, wherein the global motion is derived based on a semantic mask obtained from the sequence of images. Aspect 5 5. The method of any one of aspects 3 to 4, wherein the overall motion comprises a map of a rate of motion or an amount of motion throughout the sequence of images. Aspect 6 5. The method of any one of aspects 1 to 4, wherein the at least one object comprises at least one of a human being, an animal, a plant, or an object, or the device is a vehicle or manufacturing equipment. Aspect 7 5. The method of any one of aspects 1 to 4, comprising tracking the at least one object throughout the sequence. Aspect 8 5. The method of any one of aspects 1 to 4, further comprising: executing a classification model, such as a decision tree, to output, based on the motion information, an indicator representing whether the at least one object is moving in a manner that makes movement of the device unsafe. Aspect 9 5. The method of any one of aspects 1 to 4, further comprising filtering out fixed-position motion from the motion information. Aspect 10 Aspect 5. The method of any one of aspects 1 to 4, wherein the determining includes outputting at least one probability representing whether the device is ready to move. Aspect 11 5. The method of any one of aspects 1 to 4, wherein the determining comprises applying a logic-based algorithm to a plurality of the motion information. Aspect 12 1. A system for determining whether a device is ready to move, comprising: an acquisition module for acquiring a series of images of the device; an extractor for extracting motion information from the sequence of images, the motion information comprising information about the motion of at least one object in the sequence of images; a decision module that determines whether the device is ready to move based on the motion information; A system comprising: Aspect 13 A vehicle comprising the system described in aspect 12 and a camera, such as an internal camera, wherein the acquisition module is configured to acquire the series of images from the camera, and optionally, the vehicle is an autonomous vehicle. Aspect 14 A computer program set comprising instructions for performing the steps of the method according to any one of aspects 1 to 4 when the program set is executed by at least one computer. Aspect 15 10. At least one computer-readable recording medium having recorded thereon at least one computer program comprising instructions for performing the steps of the method according to any one of aspects 1 to 4.

Claims

1. 1. A computer-implemented method for determining whether a device is ready to move, the computer-implemented method comprising: acquiring a series of images of the device; extracting motion information from the sequence of images, the motion information comprising information about the motion of at least one object in the sequence of images; determining whether the device is ready to move based on the motion information; Including, the extracting includes detecting the at least one object in the sequence of images and estimating motion of the at least one object based on the sequence of images; the estimation includes deriving a global motion from the sequence of images and associating one or more portions of the global motion with one or more of the detected at least one object. Computer-implemented methods.

2. The method of claim 1 , wherein the global motion is derived based on a semantic mask obtained from the sequence of images.

3. The method of claim 1 or 2, wherein the overall motion comprises a map of the rate or amount of motion throughout the sequence of images.

4. The method of claim 1 or 2, wherein the at least one object comprises at least one of a human being, an animal, a plant, or an object, or the device is a vehicle or manufacturing equipment.

5. The method of claim 1 or 2, comprising tracking the at least one object through the sequence.

6. 3. The method of claim 1 or 2, further comprising running a classification model, such as a decision tree, to output an indicator based on the motion information indicating whether the at least one object is moving in a manner that makes movement of the device unsafe.

7. The method of claim 1 or 2, further comprising filtering out in-place motion from the motion information.

8. The method of claim 1 or 2, wherein the determining comprises outputting at least one probability representing whether the device is ready to move.

9. The method of claim 1 or 2, wherein the determining comprises applying a logic-based algorithm to a plurality of the motion information.

Images