Device for detecting behavior of mobile body by using image captured from the mobile body

Abstract

To provide a device which detects a plurality of still objects or the like in images captured from a mobile body and detects a behavior of the mobile body from tendencies of optical flows of the detected still objects or the like by arithmetic processing being as simple as possible.SOLUTION: A mobile body behavior detection device 1 for detecting a behavior of a mobile body by using images captured from the mobile body detects images of a plurality of still objects or still areas in a plurality of images consecutively captured by a camera 12 mounted on a vehicle 10 being the mobile body, calculates displacement vectors of the images of the still objects or still areas in the plurality of images, calculates statistical values of the displacement vectors of the images of the still objects or still areas, and detects a behavior of the mobile body on the basis of the statistical values.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a device for detecting the behavior of a moving object, and more particularly to a device for detecting the behavior of a moving object using images captured from the moving object. [Background technology]

[0002] In an image captured from a moving body such as a vehicle, the image captured therein will be displaced relative to the image due to the movement of the moving body, and therefore a technique for detecting the movement of the moving body from the image in the image captured from the moving body has been proposed. For example, Patent Document 1 discloses a configuration for an in-vehicle image processing device that recognizes a moving body from an image captured by a camera, performs a masking process on the moving body recognized in the image, and estimates the spatial movement or behavior of the vehicle, including the speed, based on the masked image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-148611 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology for detecting the behavior of a moving body from the movement of images captured in images taken from the moving body as described above is advantageous in that it makes it possible to know the behavior of the moving body from only the recorded images taken from the moving body after the fact, for example, when analyzing or diagnosing the operation of the moving body, or it makes it possible to detect the behavior of the moving body in real time if the sensor for detecting the behavior of the moving body is not installed on the moving body or if it breaks down.

[0005] Incidentally, it is possible to calculate the speed of the moving body at that time by detecting the image of a stationary object in an image taken from a moving body, detecting the actual distance to the stationary object, and then measuring the moving speed of the image of the stationary object within the image. However, the calculation process required to accurately perform such a process from detecting the image of the stationary object to calculating the speed of the moving body is somewhat complicated. In this regard, an image typically contains images of multiple stationary objects (buildings, signs, etc.) and stationary areas (roads) (hereinafter referred to as "stationary objects, etc."). As will be described in detail later, the movement (optical flow) of the images of these stationary objects, etc. tends to be longer when the moving object is moving at a high speed than when the moving object is moving at a low speed, and tends to vary more when moving straight than when turning right or left (the direction is not fixed). Therefore, by simply looking at the tendency of the optical flow of the stationary objects, etc., the general behavior of the moving object (high-speed straight movement, low-speed straight movement, stopping, turning right, turning left) can be understood without performing complex calculations such as precisely detecting the actual distance of the stationary object and calculating the speed of the moving object. In some cases, understanding the general behavior is sufficient for analyzing and understanding the driving and behavior of the moving object. Furthermore, if the images of stationary objects, etc. can be accurately recognized in an image, the general behavior of the moving object can be understood by simple calculations that detect the tendency of their optical flow. This finding is utilized in the present invention.

[0006] Thus, the main objective of the present invention is to detect multiple stationary objects, etc. in an image taken from a moving body, and to detect the behavior of the moving body from the tendency of the optical flow of the detected stationary objects, etc. using as simple a calculation process as possible. [Means for solving the problem]

[0007] According to the present invention, the above problem is solved by a device that detects the behavior of a moving object using an image captured from the moving object, means for detecting images of a plurality of stationary objects or stationary areas in a plurality of images successively captured by a camera mounted on the moving body; means for calculating a displacement vector of an image of the stationary object or stationary region in the plurality of images; means for calculating statistics of displacement vectors of the image of the stationary object or stationary area; means for detecting the behavior of the moving object based on the statistical value; This is achieved by an apparatus comprising:

[0008] In the above configuration of the present invention, the "mobile body" may be any mobile body, such as a vehicle such as an automobile, an AGV (Automatic Guided Vehicle), a robot, or a drone. The camera may be a camera mounted on the mobile body to capture the surroundings of the mobile body, such as a drive recorder, or a camera set to capture the surroundings of the mobile body, such as a smartphone, tablet, or handheld camera. The "stationary object or stationary area" may be any stationary object or stationary area that may be present around the mobile body, such as a building, sign, traffic light, utility pole, guardrail, road (white line, crosswalk), street tree, tree, sky, or celestial body. The "plurality of consecutively captured images" may be multiple images captured sequentially by the same camera at a predetermined interval that may be set as appropriate, and may be either a moving image or a still image. The detection of images of stationary objects or stationary areas in images may be achieved by extracting images using semantic segmentation technology or the like, and then using a machine learning model trained to recognize images of stationary objects or stationary areas. A "displacement vector of an image of a stationary object or stationary area" is a vector expressed by the difference in coordinates of the image of a stationary object or stationary area in a two-dimensional coordinate space set in an image. A "statistical value of a displacement vector" may be the average value, median, variance, standard deviation, etc. of the component or overall length of each direction of the displacement vector. In the present invention, "behavior of a moving object" may be stopping, moving forward (high speed, low speed), moving backward (high speed, low speed), turning right, turning left, etc. of the moving object. As described below, the behavior of a moving object is determined according to the length of the displacement vector, the average value and variation of each component, and the sign of the statistical value. Note that each of the above means of the present invention is realized by the operation of a computer device in accordance with a program.

[0009] In the above-described device of the present invention, the displacement vectors, i.e., optical flows, of the images of multiple stationary objects or stationary areas recognized in multiple images continuously captured by a camera mounted on a mobile body are detected, and then statistics of these displacement vectors are calculated, and the behavior of the mobile body as listed above is detected based on these statistics. In such a configuration, detection of the distance to the actual stationary object or stationary area, calculation of the relative velocity of the stationary object or stationary area, and conversion of the distance in the image to the actual distance are not performed, and the displacement of the image in the image is used, thereby significantly reducing the amount of calculation processing. Furthermore, since the device of the present invention only requires detection of the relative movement of stationary objects, etc. in the image, the camera can be placed at any location on the mobile body and its position can be moved, which is advantageous in that it has a high degree of freedom in the placement of the camera on the mobile body.

[0010] As already mentioned, in the device of the present invention, recognition of images of stationary objects and stationary regions in an image is performed using a machine learning model. Different machine learning models may be used depending on the environment surrounding the mobile object. For example, when the mobile object is a vehicle traveling on a well-maintained road, a model trained to better recognize stationary objects and stationary regions commonly found around well-maintained roads may be used. When the mobile object is traveling on a sidewalk or in an area where vehicles do not enter (inside a building or a shopping district), a model trained to better recognize stationary objects and stationary regions commonly found on sidewalks and in areas where vehicles do not enter may be used. When the mobile object is traveling through fields and hills, a model trained to better recognize stationary objects and stationary regions commonly found in fields and hills may be used. This improves the recognition accuracy of stationary objects and stationary regions in an image depending on the scene in which the mobile object is moving, enables accurate selection of targets for calculating displacement vectors, reduces noise during statistical value calculation, and improves the accuracy of the statistical values, i.e., the accuracy of behavior detection. [Effects of the Invention]

[0011] Thus, the device for detecting the behavior of a moving object using images captured from the moving object according to the present invention does not require complex and cumbersome calculations, such as converting the position of a stationary object in an image to the actual distance of the stationary object, but instead attempts to detect and understand the behavior of the moving object based on the tendency of the optical flow of the stationary object in the image. The device's calculation process is relatively simple, which is advantageous in that it allows for rapid detection of the behavior of a moving object. Furthermore, the present invention recognizes images of multiple stationary objects or stationary areas and uses them to calculate statistical values ​​of their displacement vectors. Therefore, even if a moving object happens to be included in the recognized image and that object moves, the effect of that movement is lost in the statistical processing, significantly reducing the possibility of erroneous behavior determination. (When the actual distance of an object in a certain image is individually detected, if that object is a moving object, the movement of the object may result in an erroneous result.)

[0012] Other objects and advantages of the present invention will become apparent from the following description of preferred embodiments of the invention. [Brief explanation of the drawings]

[0013] [Figure 1] Fig. 1(A) is a schematic diagram of a device for detecting the behavior of a moving object according to this embodiment, and Fig. 1(B) is a block diagram showing the configuration of the device according to this embodiment. [Figure 2] 2(A) to 2(C) are diagrams illustrating the process of recognizing images of stationary objects and masking images of moving objects in an image in a device according to this embodiment. (A) is an image taken from a moving object (vehicle) traveling on a well-maintained roadway, (B) is an image taken from a moving object traveling on a sidewalk, and (C) is an image taken from a moving object traveling through fields and mountains. The left image is the original image, and the right image is an image with the moving object images masked. [Figure 3]3A to 3D are diagrams showing examples of optical flow (image movement) of stationary objects in an image detected by the device according to this embodiment. In the diagrams, white circles indicate the positions of the images of stationary objects recognized at a certain point in time, and white arrows indicate the amount and direction of their subsequent displacement. [Figure 4] 4A to 4D are diagrams showing only examples of optical flow (image movement) of stationary objects in images detected by the device according to this embodiment. In the diagrams, black circles represent the positions of images of stationary objects recognized at a certain point in time, and lines represent the amount and direction of their subsequent displacement. [Figure 5] FIG. 5 is a diagram illustrating that the behavior of a moving object can be determined based on the statistical values ​​of the displacement vectors corresponding to the optical flow of a stationary object or the like in an image in the device according to this embodiment. [Explanation of symbols]

[0014] 1...mobile object behavior detection device, 2...display monitor, 10...vehicle, 12...vehicle-mounted camera, 13...image memory BEST MODE FOR CARRYING OUT THE INVENTION

[0015] Device configuration As shown in FIG. 1A , the mobile object behavior detection device 1 according to this embodiment detects the behavior of a mobile object, such as stopping, moving straight, or turning right or left, from the movement of images of stationary objects captured in surrounding images captured by a camera 12 of a mobile object 10, such as a vehicle or robot, i.e., optical flow. The camera 12 may typically be a camera mounted on the mobile object, such as a drive recorder. However, since it is sufficient to detect the relative movement of stationary objects in the images, the camera may be located anywhere on the mobile object and its position may be movable. Furthermore, the camera may be a smartphone, tablet, or handheld camera, as long as it is fixed to the mobile object. The behavior of the mobile object detected by the device according to this embodiment is displayed on a display monitor 2 or the like and may be used to analyze the behavior of the mobile object from only images captured from the mobile object, for example, in post-event analysis or diagnosis of the operation of the mobile object. Although not shown, the device may also be used to detect the behavior of the mobile object in real time when the mobile object does not have a sensor for detecting the behavior or when the sensor is broken. The moving object behavior detection device 1 is configured by a computer device that operates according to a program.

[0016] More specifically, as shown in FIG. 1B, the mobile object behavior detection device 1 includes a database that stores images captured by the camera 12; a stationary object image recognition unit that recognizes images of multiple stationary objects in the images; a displacement vector detection unit that detects the displacement vectors of each stationary object from its position in multiple consecutive images; a displacement vector statistical value calculation unit that calculates statistical values ​​of the detected displacement vectors to grasp the trend of the multiple detected displacement vectors; and a behavior determination unit that determines the behavior of the mobile object based on the calculated statistical values. Each of the above units is realized by the operation of a computer device in accordance with a program. The configuration of the device of this embodiment may be realized in a computer device installed in the mobile object, or in an external or cloud-based computer device. In this case, images are transmitted to the external computer device via a network. The operation of each unit is described in detail below.

[0017] Operation of each part of the device (1) Image database In the image database of the device of this embodiment, a plurality of images taken by the same camera on a moving object at predetermined intervals that may be set as appropriate are sequentially stored. The images may be either moving images or still images.

[0018] (2)Stationary object image recognition unit The still object image recognition unit recognizes images of multiple still objects, etc. in multiple sequentially captured images. Specific examples of still objects, etc. that may be present around a moving object include buildings, signs, traffic lights, utility poles, guardrails, roads (white lines, crosswalks), street trees, trees, the sky, celestial bodies, etc. Recognition of images of still objects, etc. may be achieved by extracting images using semantic segmentation technology, etc., and then using a machine learning model trained to recognize images of still objects, etc. In this case, when images of moving objects such as cars, bicycles, people, and animals are recognized, those images may be masked so as not to affect the recognition of images of still objects, etc.

[0019] The types of images of stationary objects and the like captured in an image vary depending on the surrounding environment in which the mobile object is moving. Therefore, in this embodiment, different specialized models may be used depending on the surrounding environment of the mobile object so as to accurately recognize images that appear in each surrounding environment. Specifically, for example, as shown in FIG. 2(A), when the mobile object is a car traveling on a well-maintained roadway, images of moving objects such as cars and people that appear around the roadway are masked (right figure), while a model trained to accurately recognize images of stationary objects such as buildings, signs, traffic lights, utility poles, guardrails, roads (white lines, crosswalks), and street trees is used. Furthermore, as shown in FIG. 2(B), when the mobile object is traveling in an area where cars cannot enter, such as on a sidewalk or inside a building, images of people are masked (right figure), while a model trained to accurately recognize images of stationary objects frequently seen inside a sidewalk or building is used. Furthermore, as shown in Figure 2(C), in the case of a moving object moving through fields and mountains, images of moving objects such as animals are masked (right image), while a model trained to accurately recognize images of stationary objects such as trees, the ground, and the sky is used.

[0020] (3) Displacement vector detection unit When the stationary object image recognition unit detects images of stationary objects, etc., in multiple images, the difference between the position coordinates of each of the multiple images of the stationary object, etc., is calculated as the optical flow of the stationary object, etc., to detect a displacement vector of each image of the stationary object, etc. The interval between images for calculating such a difference may be set appropriately. As a result, as depicted in Figures 3 and 4, displacement vectors with different tendencies are detected depending on the behavior of the moving object. For example, when the moving object is stopped, as shown in Figures 3 and 4(A), the images of each stationary object, etc., hardly displace, resulting in a zero vector. When the moving object is moving straight forward, as shown in Figures 3 and 4(B), the images of each stationary object, etc., displace radially outward from the center of the image, resulting in a displacement vector that points radially outward from the center of the image. In this case, the faster the moving speed of the moving object, the longer the length of the vector per unit time. When a moving object is turning right or left, the images of each stationary object, etc., are displaced to the left or right, respectively, as shown in Figures 3, 4(C) or 4(D), and therefore the displacement vector becomes a vector pointing to the left or right.

[0021] (4) Displacement vector statistical value calculation unit The displacement vector statistical value calculation unit calculates statistical values ​​of displacement vectors to grasp the tendency of displacement vectors of images of multiple stationary objects, etc., recognized in the image. In this regard, as generally depicted in FIG. 5 , when a moving object is moving straight, the displacement vectors of the images of stationary objects, etc., become longer as the speed increases and point in a radial direction relative to the center of the image. Therefore, the average length of the displacement vector becomes longer as the speed increases, and the variance of each of the vertical and horizontal components of the displacement vector increases. Note that forward and backward movement is distinguished by whether the direction of the displacement vector is outward or inward in the radial direction. Furthermore, when a moving object is turning right or left, the directions of the displacement vectors are aligned, so the variance of each of the vertical and horizontal components of the displacement vector decreases. Note that right and left turns are distinguished by the sign of the average value of the horizontal component of the displacement vector. Therefore, in an embodiment, the statistical values ​​of the displacement vector may specifically be the average length of the displacement vector, the average values ​​and variances of the vertical and horizontal components of the displacement vector, etc.

[0022] (5) Behavior determination unit The behavior determination unit determines the behavior of the moving object based on the statistical values ​​of the displacement vectors of the images of the plurality of stationary objects, etc. Specifically, referring again to FIG. 5, when the average length of the displacement vector is shorter than a predetermined value and the variance of the displacement vector components is smaller than a predetermined value, the moving object is determined to be stationary. When the average length of the displacement vector is shorter than a predetermined value and the variance of the displacement vector components is larger than a predetermined value, the moving object is determined to be moving straight at a low speed. When the average length of the displacement vector is longer than a predetermined value and the variance of the displacement vector components is larger than a predetermined value, the moving object is determined to be moving straight at a high speed. Note that when the moving object is moving straight, if the displacement vector points radially outward from the center of the image, it is determined to be moving forward, and if the displacement vector points toward the center of the image, it is determined to be moving backward (when the average length from the center of the image to the starting point of the displacement vector is longer than the average length to the end point of the displacement vector, it may be determined to be moving forward, and vice versa). When the average length of the displacement vector is longer than a predetermined value and the variance of the displacement vector components is smaller than a predetermined value, it is determined that the moving object is turning right or left at a low speed. Whether it is turning right or left is determined by the sign of the average value of the lateral components of the displacement vector.

[0023] Thus, in the device of this embodiment, the behavior of a moving object is detected from the tendency of the optical flow of images of multiple stationary objects, etc. in images captured from the moving object. The device of this embodiment does not perform complex and cumbersome calculations such as measuring the actual distance of individual stationary objects, etc., and also statistically refers to the optical flow of images of multiple stationary objects, etc., which is advantageous in that it is less susceptible to errors and mistakes in recognizing individual images.

[0024] The above description has been made in relation to the embodiments of the present invention, but it will be apparent that many modifications and changes will be readily apparent to those skilled in the art, and the present invention is not limited to the above-described exemplary embodiments, but can be applied to various devices without departing from the concept of the present invention.

Claims

[Claim 1] A device for detecting behavior of a moving object using an image captured from the moving object, means for detecting images of a plurality of stationary objects or stationary areas in a plurality of images successively captured by a camera mounted on the moving body; means for calculating a displacement vector of an image of the stationary object or stationary region in the plurality of images; means for calculating statistics of displacement vectors of the image of the stationary object or stationary area; means for detecting the behavior of the moving object based on the statistical value; An apparatus comprising:

Images