Method and system for determining whether device is ready to move

A computer-implemented method and system analyze images to determine if a device is ready to move by extracting motion information and applying classification models, addressing the need for further automation in vehicles and enhancing safety.

JP2024160967AActive Publication Date: 2024-11-15TOYOTA JIDOSHA KK
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2024073106
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-04-27
Filing Date
2024-04-26
Publication Date
2024-11-15
Estimated Expiration
2044-04-26

AI Technical Summary

Technical Problem

There is a need for further automation in determining whether a vehicle or device is ready to move, particularly in shared vehicles, to ensure safety and reduce human supervision.

Method used

A computer-implemented method and system that analyzes a series of images to extract motion information of objects within or around the device, using object detection and semantic segmentation, and applies classification models to determine if the device is ready to move, considering the stability and safety of these objects.

Benefits of technology

The method provides accurate and reliable automation in determining if a device is ready to move, reducing the need for human supervision and enhancing safety by ensuring that initial movement does not cause accidents.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024160967000001_ABST
    Figure 2024160967000001_ABST
Patent Text Reader

Abstract

To provide a computer-implemented method that determines whether a device is ready to move.SOLUTION: The present invention relates to a computer-implemented method that determines whether a device is ready to move, and the computer-implemented method includes: acquiring a series of images related to the device (10); extracting motion information from the series of images (12), the motion image having information related to a motion of at least one object in the series of images; and determining whether the device is ready to move based upon the motion information (28).SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] 1. Field of the Invention The present disclosure relates to the field of automation, and in particular to a method and system for determining whether a device is ready to move. The method and system may find application in all kinds of industries, including the automotive industry. [Background technology]

[0002] 2. Description of Related Technology In recent years, the demand for automation has increased across all fields of technology. In particular, the field of autonomous vehicles has experienced great developments to partially or fully automate the driving tasks that are usually performed by a driver. However, in vehicles, and more specifically in shared vehicles, such as buses, shuttles, trains, etc., the driver (or sometimes the conductor or pilot) usually performs tasks other than the pure driving task, such as managing the opening and closing of doors, deciding whether the vehicle can depart, assisting disabled people, dealing with emergency situations, and the like.

[0003] Thus, there is a need for more automation, particularly with regard to determining whether a vehicle, or more generally an apparatus, is ready to move.

[0004] The following references disclose methods related to the field of computer vision. [Reference 1] Zheng Ge et al., YOLOX:Exceeding YOLO Series in 2021, arXiv:2107.08430v2 [Reference 2] Nicolai Wojke et al., Simple Online and Realtime Tracking with a Deep Association Metric, arXiv:1703.07402v1 [Reference 3] Liang-Chieh Chen et al., Rethinking Atrous Convolution for Semantic Image Segmentation, arXiv:1706.05587v3 [Reference 4] D. Vega-Hernandez, AM Herrera-Navarro, and H. Jimenez-Hernandez, Background Subtraction Model Based on Adaptable MOG, 2012 IEEE Ninth Electronics, Robotics and Automotive Mechanics Conference, Cuernavaca, Mexico, 2012, pp.54-59, doi:10.1109 / CERMA.2012.17 [Reference 5] D. Fierens, G. Van den Broeck, J. Renkens, D. Shterionov, B. Gutmann, I. Thon, G. Janssens, and L. De Raedt, Inference and learning in probabilistic logic programs using weighted Boolean formulas, Theory and Practice of Logic Programming, 15:3, pp.358-401, Cambridge University Press, 2015. Summary of the Invention

[0005] In this regard, the present disclosure relates to a computer-implemented method for determining whether a device is ready to move, the computer-implemented method comprising: acquiring a series of images relating to the device; extracting motion information from the sequence of images, the motion information comprising information regarding the motion of at least one object in the sequence of images; determining whether the device is ready to move based on the motion information; Includes.

[0006] The steps of the method may be performed by one or more processors. The method is hereinafter referred to as a determination method.

[0007] An apparatus may have one or more moving parts. "An apparatus is ready to move" means that at least one of the one or more moving parts is ready to move.

[0008] Acquiring the sequence of images may include, for example, acquiring the sequence via an image acquisition module such as a camera, or obtaining an already acquired sequence from a database, for example a local or remote server, or the like.

[0009] Additionally, the acquiring step may include any desired pre-processing of the sequence of images, such as rectifying, cropping, resizing, rotating, etc., or combinations thereof.

[0010] The sequence of images comprises a plurality of images. The sequence of images may comprise a plurality of consecutive frames of a video clip. Alternatively, the sequence of images may comprise a plurality of non-consecutive frames of a video clip, for example selected at a given sampling frequency (e.g., every third frame of the video clip). The sequence of images need not be sampled at a regular frequency.

[0011] A series of images relating to a device means that the series of images shows at least a portion of the device and / or at least a portion of the device's environment with which the device may interact. For example, if the device is a vehicle, the series of images may show the vehicle cabin or vehicle surroundings. If the device is a robotic system, the series of images may at least partially show the range of motion of the robotic system. By monitoring the series of images relating to the device, the determination method may determine whether allowing movement of the device will have any undesirable effects on its environment.

[0012] Specifically, the determination method extracts motion information from the sequence of images. The motion information comprises information regarding the motion of at least one object shown in some or all of the sequence of images. In this disclosure, object should be understood as a general term meaning any obstacle to the device, including a living being (e.g., human, animal, plant) or an object. The information regarding the motion may comprise position in one or more dimensions, velocity, acceleration, or any combination thereof. Alternatively or additionally, the information regarding the motion may comprise an indicator that qualifies the amount and / or type of motion of the object.

[0013] Hereinafter, unless otherwise stated, an "object" refers to at least one object. More generally, as used herein, and for brevity, unless the context indicates otherwise, "a," "an," and "the" refer to "at least one" or "each," and are intended to include the plural forms as well. Conversely, the common use of the plural includes the singular form as well.

[0014] The motion information is taken into account to determine whether the device is ready to move. When the motion information is extracted from a series of images, the reliability of the determination method is improved. By checking the motion around and / or within the device, it is ensured that the initial movement of the device does not cause any big surprises or trigger any accidents. Thus, the proposed determination method provides a suitable automation to determine whether the device is ready to move, reducing the need for human supervision. Based on this determination, further actions such as the actual movement of the device can be performed.

[0015] Optionally, the extraction includes detecting at least one object in the sequence of images and estimating a motion of the at least one object based on the sequence of images. Object detection per se is known in the art. The estimation of the motion of the detected object may rely on the evolution of geometric properties of the object in the sequence of images, for example, one or more feature points of the object in the bounding box of the object, the center of gravity of the object, etc.

[0016] Optionally, the estimation includes deriving a global motion from the sequence of images and associating one or more portions of the global motion with one or more of the detected at least one object. Instead of first determining the objects and then estimating the motion of each of the objects individually, in this way motion information can be determined more accurately even when objects overlap.

[0017] Optionally, the global motion is derived based on a semantic mask obtained from the sequence of images. The semantic mask is a mask, i.e. an element that reveals a part of the image while masking other pixels of the image, the revealed part being determined based on a semantic segmentation of the image. Semantic segmentation and mask computation itself are known in the field of image processing and are easy to use. As a result, the proposed method is efficient and robust.

[0018] Optionally, the overall motion comprises a map of the rate or amount of motion throughout the sequence of images. The map may assign a rate / amount of motion to each pixel or area of ​​each image. By assigning a rate / amount of motion to pixels or areas of images instead of assigning motion at the object level, cases can be properly identified where sub-parts of an object have excessive motion while the overall movement of the object is limited. As a result, the accuracy of the method is improved by using such a map.

[0019] Optionally, the at least one object comprises at least one of a human being, an animal, a plant, or an object.

[0020] Optionally, the equipment is a vehicle or manufacturing equipment. Vehicles include, but are not limited to, cars, buses, shuttles, trains, elevators, boats, aircraft, cable transport, etc. Manufacturing equipment includes, but are not limited to, manufacturing lines, robotic systems, etc. Accidental movement of vehicles or manufacturing equipment can have dramatic consequences. Thus, the decision method improves the safety of these equipment.

[0021] Optionally, the method comprises tracking at least one object throughout the sequence, which may be done by tracking any relevant parts and / or properties of the object, for example a bounding box of the object.

[0022] Optionally, the method further comprises executing a classification model to output, based on the motion information, an indicator representative of whether the at least one object is moving in a manner that makes movement of the device unsafe.

[0023] The classification model may be a machine learning model, such as a decision tree, or another type of model. The classification model may take the motion information as input and output an indicator.

[0024] The indicator represents the degree to which the motion of the object reduces the object's readiness for (possibly abrupt) movement of the device. For example, if it is determined that the object moves a lot, the object may be deemed unsafe for the device to move because it is not stable and may tip over or be turned inside out by the device movement. Conversely, object motion that does not affect the object's stability may not be deemed to make the device movement unsafe. If the indicator indicates that the device movement is unsafe, it may be determined that the device is not ready to move. The classification model allows the decision method to make an unambiguous decision of whether the device is ready to move.

[0025] Optionally, the method includes filtering out positional motions from the motion information. As a result, positional motions are ignored from the motion information. In this way, the motion information may more accurately convey information about the stability of the object. Positional motions include motions of an object that do not change at least one of the object's overall position, the object's center of gravity, or the object's attitude (e.g., angular position) by more than a threshold. For example, actions such as reading or waving to someone may qualify as positional motions.

[0026] Optionally, the determination includes outputting at least one probability representing whether the device is ready to move. By outputting a probability, the determination method can be easily incorporated into other systems, making operation of the device easier. Instead of a probability, the determination may include outputting a binary flag.

[0027] Optionally, the determination includes applying a logic-based algorithm to the plurality of motion information. The logic-based algorithm is, for example, a rule-based and / or probability-based algorithm that applies the rules and / or probabilities to the plurality of inputs, i.e., the plurality of motion information, to determine an output. By combining a motion detection method and a logic-based algorithm based on the plurality of motion information, the method is not only more reliable because it is based on multiple detections, but also simpler because the pieces of motion information are combined by the logic-based algorithm. This avoids the need for complex models to handle a wide variety of situations. In contrast, by putting together the pieces of motion information through a logic-based algorithm, it is possible to take advantage of simple detection and the certainty of logic-based reasoning that brings human insight to the method. This architecture also makes the decision method self-adaptive, for example to unknown types of movement.

[0028] Optionally, the decision making method is performed in real time to determine whether the device is ready to move based on a current situation regarding the environment surrounding the device.

[0029] The present disclosure further provides a system for determining if a device is ready to move, comprising: an acquisition module for acquiring a series of images relating to the device; an extractor for extracting motion information from a sequence of images, the motion information comprising information about the motion of at least one object in the sequence of images; a decision module that decides whether the device is ready to move based on the motion information; The present invention relates to a system comprising:

[0030] A system, hereinafter referred to as a decision system, may be configured to perform the decision method described above and may have some or all of the features described above. The decision system may have a computer hardware structure.

[0031] The present disclosure is further directed to a vehicle comprising the above-mentioned determination system and a camera, the acquisition module being configured to acquire a sequence of images from the camera, which may be an interior camera, i.e. a camera acquiring images of the interior of the vehicle, for example the interior of the vehicle cabin.

[0032] Optionally, the vehicle is an autonomous vehicle. The decision system may implement non-driving functions of the autonomous vehicle.

[0033] The present disclosure further relates to a computer program set comprising instructions for carrying out the steps of the above-mentioned determining method when said program is executed by at least one computer, said set of programs may be in source code, object code, or intermediate code between source code and object code, for example in a partially compiled form, using any programming language, or in any other desired form.

[0034] The present disclosure further relates to at least one computer readable recording medium, having at least one computer program recorded thereon, the computer program including instructions for performing the steps of the above-mentioned determining method. The recording medium may be any entity or device capable of storing a program. For example, the medium may include a storage means such as a ROM, for example a CD ROM or a microelectronic circuit ROM, or a magnetic storage means, for example a diskette (floppy disk) or a hard disk.

[0035] Alternatively, the recording medium may be an integrated circuit in which the program is embedded, the circuit being configured to perform, or for use in the performance of, the method. [Brief description of the drawings]

[0036] The features, advantages, and technical and industrial importance of preferred embodiments of the present invention are described below with reference to the accompanying drawings, in which like reference numerals refer to like elements, and in which:

[0037] [Figure 1] 1 is a schematic flow chart of a method for determining whether a device is ready to move, according to an embodiment; [Diagram 2] 4 is a detailed flowchart of a method for determining whether a device is ready to move, according to an embodiment. [Diagram 3] FIG. 1 is a side view of a vehicle according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] A method for determining whether an apparatus is ready to move is described with reference to Figures 1 and 2. In this embodiment, the apparatus is a vehicle. However, the following description can be translated to other types of apparatus, such as manufacturing equipment. Figure 1 shows an overview of the method, while Figure 2 shows the embodiment in more detail.

[0039] As described above with reference to Figure 1, the determination method includes a step 10 of acquiring a series of images of the device. Step 10 may include any suitable pre-processing of the images to make them ready for the next step in the determination method.

[0040] The method further comprises a step 12 of extracting motion information from the sequence of images. The motion information comprises information about the motion of at least one object in the sequence of images. In the following, for simplicity, it is assumed that the object is a person, but the object may also be an animal, a plant or an object. When an image may show multiple objects, the extracted motion information may relate to multiple objects. Furthermore, the extraction may target specific objects, e.g., humans, pets, baggage, etc., while treating other objects (e.g., seats, seat belts, retaining bars, etc.) as background.

[0041] In a possible application, the determination method is configured to determine, based on a series of images of the interior of a vehicle, whether the vehicle occupants are sufficiently secured or stable for the vehicle to depart without risking causing one of the vehicle occupants to fall. Although departure is mentioned, other types of movements are encompassed, e.g. accelerating, braking, turning, etc. More generally, the movements involved depend on the type of device.

[0042] For example, step 12 may include an object detection step 14, which aims to detect at least one object (e.g. a person) in the sequence of images. For example, object detection step 14 may include a step 16 of identifying one or more objects in the sequence of images, and, optionally, a step 18 of tracking the identified objects throughout the sequence of images.

[0043] Furthermore, the motion calculation step 20 may include deriving a global motion from the sequence of images. In particular, the motion calculation step 20 may include a step 22 of semantic segmentation of the sequence of images to separate objects in the images from one another, and a step 24 of estimating the global motion of the segmented objects.

[0044] The determination method may take the output of the object detection step 14, i.e. the location of objects in the sequence of images, and the output of the motion calculation step 20, i.e. the aforementioned overall motion, and associate one or more parts of the overall motion to one or more of the at least one detected object in a motion association step 26. The motion association step 26 outputs motion information.

[0045] Then, in step 28, based on the motion information, the decision method determines whether the device is ready to move. For example, the decision method, e.g., step 28, may include a step 30 of filtering out positional motion from the motion information output from step 26, e.g., to retain only motion information related to stability.

[0046] Further, the decision method, e.g., step 28, may include step 32 of running a classification model on the motion information output from step 28 to classify movements according to the movement's impact on the decision to allow or not allow the movement of the device. For example, the classification model may classify movements into stability-related and non-stability-related movements based on whether the motion information affects the stability of the person. The classification model may output an indicator representing whether at least one object is moving in a manner that makes the movement of the device (e.g., vehicle departure) unsafe.

[0047] The outputs of each of steps 30, 32 may be combined to determine whether the device is ready to move in step 34. Step 34 may produce a flag or probability 36 indicating whether the device is ready to move. Thus, the departure of the vehicle may be determined automatically and reliably.

[0048] An embodiment of the determination method is described in more detail with reference to Fig. 2. However, as can be understood from Fig. 1, the architecture of the proposed determination method is modular and variations of each module can be suitably used by those skilled in the art without departing from the principles expressed herein. As detailed below, the determination method may include using at least one machine learning model.

[0049] FIG. 2 shows an example of performing a step 10 of acquiring a series of images of a device, where a number of video frames 40 can be acquired, for example, using a camera. The camera can be a fish-eye camera. Fish-eye cameras as such are known in the art and allow acquiring a large field of view. However, other types of cameras are envisaged. In the example of a vehicle, the camera can be mounted inside the vehicle. The camera can, for example, view the interior cabin of the vehicle to monitor the movements of the occupants in order to perform the decision method.

[0050] The series of images can be RGB (red / green / blue) images, or more generally, single-channel or multi-channel images. An RGB image of a scene comprises pixels having red, green and blue values ​​to form colors (including black and white or grayscale) that represent the colors that the scene has. A single-channel image is an image in which each pixel has a color or intensity represented by a single value, and a multi-channel image is an image in which each pixel has a color or intensity represented by multiple values. Grayscale is an example of a single-channel scale, while RGB is an example of a multi-channel scale.

[0051] A number of video frames 40 are acquired at step 42. Step 42 may occur in real time. Step 42 may acquire all available video frames 40, or may sample the video frames 40 at any desired rate.

[0052] In some cases, the method may include a step 44 of pre-processing the sequence of images. For example, step 44 may include reprojecting or rectifying the sequence of images, for example to convert images from a fisheye image into rectified images. Other operations such as cropping, resizing, rotating, compressing, etc. are possible, alone or in combination. Step 44 outputs pre-processed images 46 (e.g. rectified images) taking into account the extraction of motion information therefrom.

[0053] Specifically, the pre-processed image 46 is output to an object detection step 14 and a motion calculation step 20. Although the object detection step 14 and the motion calculation step 20 are depicted as being parallel and independent of each other, the object detection step 14 and the motion calculation step 20 may be performed sequentially.

[0054] As indicated above, in the object detection step 14, the decision method may first proceed to step 16 of identifying one or more objects in a sequence of images, here pre-processed images 46. Step 16 may be applied individually to each pre-processed image 46. Step 16 may rely on an object detector model, for example a human detector such as YoloX [Reference 1]. Many object detector models, such as YoloX, are trained on rectified images, so in such cases the fisheye images need to be rectified. However, if the object detector model can work from the fisheye images, for example by fine-tuning on the images, then no rectification is necessary. Thus, the pre-processing step 44 is optional and essentially depends on the requirements of step 16 of identifying one or more objects in a sequence of images.

[0055] The object detector model may output characteristics, e.g., contours, centroids, etc., that allow identification of objects in the preprocessed images 46. In this example, the object detector model outputs a bounding box 17 for each object in each image. The bounding boxes 17 may be two-dimensional or three-dimensional bounding boxes.

[0056] The bounding boxes 17 serve as input to step 18, which tracks the identified objects throughout the sequence of images. Step 18 may rely on object tracker models such as DeepSort [Reference 2]. Tracking the bounding boxes (or any other characteristic that describes the detected objects) makes it possible to establish relationships between the bounding boxes from one frame to another. In other words, it makes it possible to determine which objects are moving.

[0057] Thus, in this embodiment, step 18 outputs a bounding box of the tracked person 19. More generally, step 14 outputs object detections in the sequence of images.

[0058] As indicated above, in the motion calculation step 20, the decision method may first proceed to a step 22 of semantic segmentation of the sequence of images, here the preprocessed images 46. Step 22 may rely on a semantic segmentation model such as DeepLabv3 [Reference 3]. In particular, the semantic segmentation model may be configured to output a semantic mask 23 obtained from the sequence of images, in particular a respective semantic mask 23 for each image of the sequence of images. The semantic mask 23 may be a binary mask. In particular, the semantic mask may reveal objects (e.g. people) detected in the image, while masking everything else, e.g. walls, floors, seats and windows of the vehicle cabin.

[0059] The semantic mask 23 serves as input to a step 24 of estimating the global motion of the segmented objects. The global motion may comprise a map of the rate of motion or the amount of motion throughout the sequence of images.

[0060] For example, to obtain such a map, step 24 may include a step 50 of subtracting the semantic mask of an image from the semantic mask of an adjacent image (e.g. the one immediately preceding or following in the sequence of images). The displacement between two adjacent semantic masks indicates which pixels of the detected object have moved between the two images. This difference therefore provides information about the overall motion of the detected object. Obtaining the map based on the semantic mask makes it possible to constrain the area where the movement needs to be calculated. This improves the robustness of the motion estimation since the (irrelevant) background motion is filtered out.

[0061] Step 50 may rely on a background subtraction model such as MOG [Reference 4]. In particular, step 50 may output a global motion map 52.

[0062] Step 24 may further include a step 54 of filtering the global motion map 52. A semantic mask 23 is applied to the global motion map 52 to filter out light changes and sudden environmental motion. In doing so, step 54 outputs a more robust motion map 56, assigning an amount or percentage of motion for each image to a particular person. Thus, the motion map 56 provides the global motion of the segmented objects.

[0063] In this embodiment, the motion association step 26 described above may take as input the motion map 56 and the tracked person's bounding box 19, and associate one or more portions of the overall motion described by the motion map 56 with one or more of the detected objects described by the tracked person's bounding box 19. Thus, the association step may combine pixel-by-pixel information from the motion map 56 with region-based detections from the tracked person's bounding box 19 to derive an amount of (individual) motion.

[0064] Thus, the method may estimate the motion of at least one object based on a sequence of images.

[0065] The method then proceeds to step 30 of filtering out in-place motion. As mentioned above, in-place motion refers to object movement that does not change the overall position of the object. For example, an object movement may be determined to be in-place motion if the amount or ratio of the bounding box displacement is less than a threshold. For example, the ratio of the displacement for the bounding box may be calculated as 1-IoU, where IoU refers to the intersection-over-union ratio of the bounding box area in two consecutive frames. Conversely, if the amount or ratio of the displacement is higher than a threshold, it may be determined that the object (e.g., a person) is moving overall in the vehicle.

[0066] In parallel, the decision method may proceed to step 32 of executing a classification model. In this example, the classification model is a decision tree. For example, step 32 may use a custom-built decision tree, which is itself known by a person skilled in the art. The decision tree may be a machine learning model, which is configured to take into account the estimated motion and to output, based on the estimated motion (corresponding to the motion information), an indicator representing whether at least one object is moving in a manner that makes the movement of the device unsafe.

[0067] In an example, the estimated motion comprises a mask, and the classification model may be configured to determine a contour of pixels indicated by the mask and derive an area defined within this contour. The classification model may then be trained to determine the indicators described above.

[0068] In step 34, the decision method determines whether the device is ready to move. Essentially, step 34 combines the indicators provided by the classification model in step 32 with the in-place motion filtered motion information of step 30 to limit the effect of in-place motion on the method's decisions.

[0069] In an example, step 34 may include applying a logic-based algorithm to the plurality of motion information, i.e., the indicators of step 32 and the in-place motion filtered motion information of step 30. The logic-based algorithm may be a probabilistic inference with appropriate reasoning rules. For example, step 34 may rely on a logic-based engine such as ProbLog [Reference 5]. However, other logic-based algorithms may be used, for example based on binary (Boolean) logic, fuzzy logic, etc.

[0070] For example, the reasoning rules may describe a probabilistic model that makes decisions based on multiple pieces of information. In the example, the decision tree decision (i.e., the indicator of step 32) is represented by two predicates, namely, dTREE_GO with a probability of, for example, 0.5 and dTREE_STOP with a probability of, for example, 0.5. These predicates represent the likelihood of a method based on the decision tree model to determine whether the device is ready to move. The bounding box reasoner decision (i.e., the motion information filtered by the motion in place of step 30) is represented by three predicates, namely, bReasoner_INPLACE, bReasoner_MOVINGPERSON, and bReasoner_NEWPERSON with probabilities of, for example, 0.4, 0.4, and 0.2, respectively. These predicates represent the evaluation of a method based on the bounding box reasoner with respect to the current situation, which allows the position or movement of a person in the scene to be evaluated.

[0071] In this example, the reasoning rules may combine the decision tree and bounding box reasoner judgments to make a final decision on the decision method. This final decision may be represented by two predicates, ai_conductor_GO and ai_conductor_STOP, which respectively represent a method to determine whether the device is ready to move or not. These rules combine various predicates from the decision tree and the bounding box reasoner with various probabilities, resulting in a set of possible outputs. For example, if the decision tree suggests that the device is ready to move and the bounding box reasoner detects a person in place (i.e. undergoing a fixed motion), there is, for example, a probability of 0.9 that the AI ​​system will proceed. This combination allows the decision method to make more nuanced decisions based on both the decision tree's analysis of the situation and the bounding box reasoner's evaluation.

[0072] Step 34 may output any type of decision regarding whether the device is ready to move, for example at least one probability representing whether the device is ready to move.

[0073] Experiments conducted by the inventors show that the proposed decision method achieved superior judgment with an accuracy of over 80% in 200 real-world videos, where superior judgment means that the decision method accurately determined whether the vehicle is ready to depart compared to what a human manager concluded.

[0074] 3 shows a vehicle 80 comprising a camera 82 and a decision system 84 for determining whether the vehicle 80 is ready to move. The decision system comprises an acquisition module 86 for acquiring a sequence of images relating to the vehicle 80, an extractor 88 for extracting motion information from the sequence of images, the motion information comprising information regarding the motion of at least one object in the sequence of images, and a decision module 90 for determining whether the vehicle 80 is ready to move based on the motion information.

[0075] The determination system 84 may include electronic circuits, processors (shared, dedicated, or group), combinatorial logic circuits, memory for executing one or more software programs, and / or other suitable components for providing the described functionality. In other words, the determination system 84 may be a computing device. The determination system 84 may be connected to a memory (recording medium), which may store data, such as a computer program, that, when executed, performs the assistance method according to the present disclosure.

[0076] The determination system 84, in particular the acquisition module 86, has an input for receiving a digital image or a stream of digital images. In particular, the determination system 84, in particular the acquisition module 86, may be connected to an optical sensor, in particular one or more digital cameras, for example the above-mentioned camera 82, which may be a fish-eye camera. The digital camera is configured to be able to record a scene, in particular to be able to output digital data to the determination system 84, in particular the acquisition module 86.

[0077] If the decision method is performed in real time, the at least one image may be acquired in real time by an image acquisition module, such as a camera 82 or the like (video camera, photo camera, etc.). Alternatively or additionally, for example in the case of post-processing of a film process, the at least one image may be acquired in advance and later acquired by the decision system 84.

[0078] Although the present disclosure refers to specific preferred embodiments, modifications to the examples may be provided without departing from the overall scope of the invention as defined by the claims. In particular, individual features of the various embodiments shown / described may be combined in additional embodiments. Accordingly, the description and drawings should be regarded in an illustrative rather than a restrictive sense.

Claims

1. 1. A computer-implemented method for determining whether a device is ready to move, the computer-implemented method comprising: acquiring a series of images of the device; extracting motion information from the sequence of images, the motion information comprising information regarding the motion of at least one object in the sequence of images; determining whether the device is ready to move based on the motion information; 4. A computer-implemented method comprising:

2. The method of claim 1 , wherein the extracting comprises: detecting the at least one object in the sequence of images; and estimating a motion of the at least one object based on the sequence of images.

3. The method of claim 2 , wherein the estimation comprises deriving a global motion from the sequence of images and associating one or more portions of the global motion with one or more of the detected at least one object.

4. The method of claim 3 , wherein the global motion is derived based on a semantic mask obtained from the sequence of images.

5. The method of claim 3 or 4, wherein the overall motion comprises a map of the rate or amount of motion throughout the sequence of images.

6. The method of claim 1 , wherein the at least one object comprises at least one of a human being, an animal, a plant, or an object, or the device is a vehicle or manufacturing equipment.

7. A method according to claim 1 , further comprising tracking the at least one object through the sequence.

8. 5. The method of claim 1 , further comprising: executing a classification model, such as a decision tree, to output an indicator based on the motion information representing whether the at least one object is moving in a manner that makes movement of the device unsafe.

9. The method of claim 1 , further comprising filtering out positional motion from the motion information.

10. The method of claim 1 , wherein the determining comprises outputting at least one probability representative of whether the device is ready to move.

11. The method of claim 1 , wherein the determining comprises applying a logic-based algorithm to a plurality of the motion information.

12. 1. A system for determining whether a device is ready to move, comprising: an acquisition module for acquiring a series of images relating to the device; an extractor for extracting motion information from the sequence of images, the motion information comprising information about the motion of at least one object in the sequence of images; a decision module that decides whether the device is ready to move based on the motion information; A system comprising:

13. A vehicle comprising the system of claim 12 and a camera, such as an internal camera, wherein the acquisition module is configured to acquire the sequence of images from the camera, and optionally, the vehicle is an autonomous vehicle.

14. A computer program set comprising instructions for performing the steps of the method according to any one of claims 1 to 4 when said program set is executed by at least one computer.

15. 5. At least one computer readable recording medium having recorded thereon at least one computer program comprising instructions for performing the steps of the method according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • Vehicle control method, vehicle control system, and vehicle control device

    JP2020003936A