Optical flow calculation method, processor, and computer program using relative speed

Abstract

The present invention is a method for calculating optical flow considering the relative speed of a processor, and may include the steps of acquiring driving images of the surroundings of a moving object as the moving object moves, acquiring camera movement speed data as the moving object moves, and calculating optical flow within the driving images based on the camera movement speed data.

Description

Technology Field

[0001] The present invention relates to a method for calculating optical flow considering relative speed, a processor, and a computer program. Background Technology

[0002] As electronic equipment in vehicles has recently become more advanced, autonomous driving technology is also advancing significantly.

[0003] Autonomous driving technology utilizes a vehicle detection or tracking method based on optical flow that enables real-time vehicle detection by calculating optical flow using a camera mounted on the vehicle.

[0004] Conventional optical flow calculation methods used in autonomous driving utilize optical flow models trained based on stationary cameras. However, in environments where the camera moves in accordance with the vehicle's movement, there is a problem in that errors related to camera motion are reflected in the optical flow results.

[0005] To solve this problem, wrapping technology is used to map the current pixel to the corresponding pixel in the next frame, but it cannot properly eliminate errors in the optical flow caused by camera movement.

[0006] Therefore, a method for calculating optical flow that considers camera movement is required. The problem to be solved

[0007] The present invention aims to provide a method for calculating optical flow that takes into account the movement of a camera.

[0008] More specifically, the present invention aims to provide an optical flow calculation method, a processor, and a computer program that consider relative velocity so that movement is not reflected in a stationary object due to the movement of a camera. means of solving the problem

[0009] A method for calculating optical flow considering the relative speed of a processor according to the present invention for solving the above technical problem may include the steps of acquiring driving images of the surroundings of a moving body as the moving body moves, acquiring moving speed data of a camera as the moving body moves, and calculating optical flow within the driving images based on the moving speed data of the camera.

[0010] In addition, the step of calculating the optical flow can calculate the optical flow within the second driving image by comparing pixels between the first driving image and the second driving image that are sequentially captured in time among the driving images to determine the position where the pixels have moved.

[0011] In addition, the step of calculating the optical flow calculates an optical flow that reflects the relative velocity of an object in the driving image using an optical flow model with applied relative velocity, and the optical flow model can be learned in a direction that reduces relative velocity loss based on the relative velocity of a stationary object and the moving speed of a camera.

[0012] In addition, the step of calculating the optical flow may apply the relative speed of the object within the first driving image and the second driving image by considering the pixels that moved according to the driving of the moving object based on the moving speed data of the camera.

[0013] In addition, the first driving image may be a driving image taken immediately before the second driving image.

[0014] In addition, the above optical flow may be data representing the moved positions of pixels within a driving image as vectors.

[0015] In addition, the method may further include the step of acquiring position data of the moving body and the step of acquiring road surface condition data around the moving body based on the position data.

[0016] In addition, the road surface condition data may include information regarding the slope, curvature, and roughness of the road surface.

[0017] Additionally, the method further includes a step of calculating noise data of the camera based on the road surface condition data, and the noise data may be data regarding the direction and magnitude of shaking of the camera due to the road surface condition while the moving object is driving.

[0018] In addition, it may further include a step of correcting the optical flow calculated based on the above noise data. Effects of the invention

[0019] According to the present invention, by considering relative speed, an accurate optical flow of the movement of an object within a driving image can be calculated regardless of whether the camera is moving.

[0020] In addition, the present invention calculates optical flow by considering relative speed, thereby enabling greater focus on the actual moving object within the driving image. Brief explanation of the drawing

[0021] FIG. 1 is a schematic diagram showing a method for calculating optical flow considering relative velocity according to one embodiment of the present invention. FIG. 2 is a flowchart illustrating a method for calculating optical flow considering relative velocity according to an embodiment of the present invention. FIG. 3 is an exemplary diagram showing optical flow within a driving image according to one embodiment of the present invention. FIG. 4 is an exemplary diagram showing optical flow within a driving image according to one embodiment of the present invention. FIG. 5 is a flowchart illustrating a method for calculating optical flow considering relative velocity according to an embodiment of the present invention. FIG. 6 is a block diagram showing the configuration of a processor according to one embodiment of the present invention. Specific details for implementing the invention

[0022] The following description merely illustrates the principles of the invention. Therefore, those skilled in the art may invent various devices that embody the principles of the invention and are included within the concept and scope of the invention, even if they are not explicitly described or illustrated in this specification. Furthermore, all conditional terms and embodiments listed in this specification are, in principle, explicitly intended only for the purpose of enabling an understanding of the concept of the invention and should be understood as not being limited to the embodiments and conditions specifically listed elsewhere.

[0023] The aforementioned objectives, features, and advantages will become clearer through the following detailed description in conjunction with the attached drawings, and accordingly, a person skilled in the art to which the invention pertains will be able to easily implement the technical concept of the invention.

[0024] In addition, in describing the invention, if it is determined that a detailed description of known technology related to the invention may unnecessarily obscure the essence of the invention, such detailed description will be omitted. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

[0025] FIG. 1 is a schematic diagram showing a method for calculating optical flow considering relative velocity according to one embodiment of the present invention.

[0026] Referring to FIG. 1, a camera (200) mounted in a moving body (300) can capture driving images by photographing the surroundings of the moving body (300) as the moving body (300) moves.

[0027] Here, the camera (200) is a device that captures driving images of a moving object's movement with at least one other device, and may be a black box or a recording device mounted on the moving object (300) for autonomous driving. The moving object (300) refers to moving objects such as vehicles, personal mobility devices, robot vacuum cleaners, motorcycles, etc., and may include all objects capable of movement, such as a pedestrian equipped with a body cam or a pedestrian equipped with a head cam. Hereinafter, the present invention will be described based on an automobile.

[0028] And, the camera (200) can transmit the captured driving images and the moving speed data of the camera (200) according to the driving of the moving body to a processor (100) that calculates an optical flow considering the relative speed.

[0029] And, the processor (100) can calculate an optical flow within a driving image by considering the movement of the camera (200) according to the driving of the moving body (300) using the driving image and various data.

[0030] That is, the processor (100) can calculate an optical flow that takes into account the relative speed so that the movement of the camera (200) is not reflected in the stationary object.

[0032] In this regard, a method for calculating an optical flow considering the relative speed of the processor (100) is described with reference to FIG. 2.

[0033] FIG. 2 is a flowchart illustrating a method for calculating optical flow considering relative velocity according to an embodiment of the present invention.

[0034] Referring to FIG. 2, a processor (100) that calculates an optical flow considering relative speed can acquire driving images taken around the moving body (300) as the moving body (300) moves (S100).

[0035] Specifically, the processor (100) can obtain driving images that are captured sequentially over time from the camera (200). Here, the driving images may refer to images taken of the front, rear, side, etc. of the moving body (300) while the moving body (300) is driving.

[0036] Next, the processor (100) can obtain movement speed data of the camera (200) according to the movement of the moving body (300) from the camera (200) (S200).

[0037] At this time, since the movement speed of the moving body (300) and the movement speed of the camera (200) are substantially the same, the processor (100) can use the movement speed data of the moving body (300) as the movement speed data of the camera (200).

[0038] And, the processor (100) can calculate optical flow within the driving image based on the camera's movement speed data (S300). Here, optical flow may refer to data representing the moved positions of pixels within the driving image as vectors, and may also be an optical flow feature map.

[0039] At this time, the processor (100) may use an optical flow model that has been learned to calculate optical flow in order to calculate optical flow, and during the learning process, a first method using an optical flow model with relative velocity applied and a second method using an existing optical flow model to calculate optical flow and then applying relative velocity may be selectively used.

[0040] First, looking at the first method, the processor (100) can calculate an optical flow that reflects the relative velocity of a stationary object (or a non-moving object) within a driving image using an optical flow model with relative velocity applied. Here, the optical flow model with relative velocity applied determines information (such as relative velocity per pixel) about a non-moving object within the driving image through a preprocessing process, uses the measured values ​​of the relative velocity of the determined object and the camera's movement speed as ground truth, adds a relative velocity loss term, and can be trained in a direction that reduces the loss through various loss functions.

[0041] In this case, since the periphery of an object changes depending on speed or camera position as the camera moves, the region of interest containing the object within the driving image can be cropped during the preprocessing stage, and information about the object can be determined from the cropped area. Here, the method for determining the relative speed of the object during the preprocessing stage may vary depending on the number of cameras.

[0042] For example, in the case of a single camera, the relative velocity of objects per pixel can be determined using the Euclidean distance between the pixels constituting the objects in the first driving image and the pixels in the second driving image that match those pixels, and the time interval between the first driving image and the second driving image.

[0043] As another example, in the case of multiple cameras, the position value of an object can be estimated by utilizing the parallax between the driving image captured by the first camera and the driving image captured by the second camera, and the relative velocity of the object can be determined by differentiating the estimated position value. Here, parallax may refer to the difference in the direction of an object when viewed from two different points.

[0044] That is, the processor (100) can calculate an optical flow that reflects the relative speed of an object in a driving image by using an optical flow model learned in a direction that reduces relative speed loss based on the relative speed of a stationary object and the moving speed of a camera.

[0046] Next, looking at the second method, the processor (100) can calculate the optical flow by comparing each pixel within the first driving image and the second driving image, which are sequentially captured in time among the driving images, using an existing optical flow model, and representing the position where the pixel moved as a vector, and then apply the relative speed of the object based on the movement speed data of the camera (300). At this time, the processor (100) can use the movement speed data of the camera (200) at the time the first driving image and the second driving image were captured.

[0047] Here, the first driving image is a driving image taken immediately before the second driving image, and depending on the settings of the camera (200), it may be a driving image taken immediately before the frame or a frame taken at a predetermined interval earlier.

[0048] Additionally, in the optical flow calculation step (S300), the processor (100) may use an existing optical flow model to divide each pixel in the first driving image and the second driving image, which are taken in succession, into a plurality of unit blocks having a predetermined number of pixels, and calculate the optical flow by comparing each divided unit block and representing the moved position as a vector, and then apply the relative speed of the object based on the moving speed data of the camera (300).

[0049] That is, the processor (100) can calculate an optical flow by comparing pixels between a first driving image and a second driving image, taking into account pixels that have moved according to the driving of the moving body (300) based on the moving speed data of the camera (200).

[0050] Meanwhile, the processor (100) can calculate an optical flow by variably changing the frame interval between the first driving image and the second driving image according to the current resource state or the size, capacity, quality, etc. of the driving image.

[0051] Through this, the processor (100) can perform the optical flow calculation method more efficiently.

[0053] Hereinafter, a method for calculating optical flow will be explained with further reference to FIG. 3.

[0054] FIG. 3 is an exemplary diagram showing optical flow within a driving image according to one embodiment of the present invention.

[0055] Referring to FIG. 3, FIG. 3(a) shows a first driving image (31) and a second driving image (32) taken sequentially over time, and FIG. 3(b) shows a second driving image (33) in which an optical flow is calculated and displayed. Here, only a portion of the optical flow is shown for convenience of explanation.

[0056] The processor (100) can compare pixels within the first driving image (31) and the second driving image (32), which are sequentially captured in time among the driving images, and produce an optical flow that indicates the position where the pixel moved as a vector, such as the second driving image (33).

[0057] That is, the processor (100) can calculate the optical flow more accurately for stationary objects (e.g., background, trees, signs, etc.) and moving objects (e.g., moving vehicles, pedestrians, etc.) in a driving image by calculating the optical flow considering the relative speed.

[0059] Next, with reference to FIG. 4, a conventional optical flow calculation method and an optical flow calculation method considering relative velocity according to the present invention will be compared and explained.

[0060] FIG. 4 is an exemplary diagram showing optical flow within a driving image according to one embodiment of the present invention.

[0061] Referring to FIG. 4, FIG. 4 (a) is a driving image showing optical flow calculated using a conventional optical flow calculation method, and FIG. 4 (b) is a driving image showing optical flow calculated using an optical flow calculation method considering relative speed according to the present invention.

[0062] When using a conventional optical flow calculation method, as shown in FIG. 4 (a), a stationary object (41) also has a direction of movement, and an optical flow with an error reflected in the movement of a moving object (42) can be calculated.

[0063] In contrast, when using the optical flow calculation method considering relative speed of the present invention, as shown in FIG. 4 (b), the stationary object (43) does not have a direction of movement, and the movement of the moving object (42) can be calculated as an optical flow that reflects the actual movement without error.

[0064] In this way, by removing the movement speed data of the camera (200), the optical flow can reflect the movement of the actual object.

[0065] In addition, when using the optical flow calculated by considering the relative speed according to the present invention, one can focus more on the actual moving object.

[0067] Meanwhile, the processor (100) of the present invention may additionally perform a correction process to more accurately reflect the movement of an object in a driving image in the optical flow.

[0068] This will be explained with reference to Fig. 5.

[0069] FIG. 5 is a flowchart illustrating a method for calculating optical flow considering relative velocity according to an embodiment of the present invention.

[0070] Referring to FIG. 5, after the optical flow calculation step (S300), the processor (100) can obtain position data of the moving body (300) according to the movement of the moving body (300) from the moving body (300) or the camera (200) (S400). At this time, the position data may be the position at the time when the driving image used to calculate the optical flow was captured.

[0071] In addition, location data of the moving body (300) can also be obtained based on GPS and camera, and can be obtained in different ways depending on the number of cameras.

[0072] For example, when there is one camera, position data can be obtained through relative depth, and when there are two or more cameras, position data can be obtained based on the parallax of an object in the same image captured by multiple cameras.

[0073] Next, the processor (100) can obtain road surface condition data around the moving body (300) based on location data (S500). Specifically, the processor (100) can obtain road surface condition data corresponding to the location data by requesting it from a related server (not shown). Here, the related server is a public institution server related to the road on which the moving body (300) is traveling, and may be a road construction server.

[0074] In addition, road surface condition data may include information on the slope, curvature, and roughness of the road surface.

[0075] And, the processor (100) can calculate noise data of the camera (200) based on road surface condition data (S600).

[0076] Here, the noise data may include data regarding the direction and magnitude of shaking of the camera (200) due to the road surface conditions while the moving body (300) is driving.

[0077] Specifically, the processor (100) can calculate the movement of the moving body (300) according to the road surface condition using road surface condition data, previously stored detailed data of the moving body (300), and movement speed data of the moving body (300) to produce noise data of the camera (200). Here, the detailed data of the moving body (300) may include factors that affect the movement of the moving body (300), such as the tire size, air pressure, and suspension condition of the moving body (300).

[0078] Next, the processor (100) can correct the optical flow calculated based on the calculated noise data (S700).

[0079] Specifically, the processor (100) can correct the optical flow so that noise reflected in the optical flow calculated based on noise data is canceled out.

[0080] Through this, the processor (100) can correct the movement of the camera (200) according to the road surface condition so that it is not reflected in the optical flow.

[0081] Meanwhile, the optical flow calculated and corrected by the present invention described above can be used in various fields such as autonomous driving and surrounding object recognition.

[0083] Next, the configuration of the processor (100) will be described with reference to FIG. 6.

[0084] FIG. 6 is a block diagram showing the configuration of a processor (100) according to one embodiment of the present invention.

[0085] Referring to FIG. 6, the processor (100) may be composed of a communication unit (110), an optical flow calculation unit (120), an optical flow correction unit (130), a storage unit (140), and a control unit (150) in whole or in part.

[0086] The communication unit (110) can transmit and receive various data required by the processor (100). Specifically, the communication unit (110) can receive driving images and movement speed data from the camera (200).

[0087] In addition, the communication unit (110) may receive movement speed data from the moving body (300) or road surface condition data from a related organization.

[0088] The optical flow calculation unit (120) can calculate the optical flow from the driving image.

[0089] Specifically, the optical flow calculation unit (120) may use an optical flow model that has been learned to calculate optical flow in order to calculate optical flow, and may selectively use a first method of using an optical flow model that applies relative velocity during the learning process and a second method of calculating optical flow through an existing optical flow model and then applying relative velocity.

[0090] For example, the optical flow calculation unit (120) can calculate an optical flow that reflects the relative speed of an object in a driving image by using an optical flow model learned in a direction that reduces relative speed loss based on the relative speed of a stationary object and the moving speed of a camera.

[0091] As another example, the optical flow calculation unit (120) calculates the optical flow within the second driving image by comparing pixels between the first driving image and the second driving image that are sequentially captured in time among the driving images to determine the position where the pixels have moved, and can apply the relative speed of the object within the first driving image and the second driving image by considering the pixels that have moved according to the driving of the moving body (300) based on the moving speed data of the camera (200).

[0092] The optical flow correction unit (130) can correct the calculated optical flow.

[0093] Specifically, the optical flow correction unit (130) calculates the movement of the moving body (300) according to the road surface condition using road surface condition data, previously stored detailed data of the moving body (300), and moving speed data of the moving body, and calculates noise data of the camera (200), and can correct the optical flow using the calculated noise data.

[0094] The storage unit (140) can store various data required by the processor (100).

[0095] Specifically, the storage unit (140) may store detailed data of the moving body (300), driving images, optical flow models, movement speed data of the camera (200), position data of the moving body (300), etc.

[0096] Additionally, the storage unit (140) may store a program recorded on a computer-readable recording medium on which program code for executing an optical flow calculation method considering relative speed is recorded.

[0097] The control unit (150) can control the overall operation of the processor (100).

[0098] Specifically, the control unit (150) can control the communication unit (110) to receive road surface condition data from a related server.

[0099] Additionally, the control unit (150) may control the optical flow calculation unit (120) to calculate the optical flow from the driving image.

[0100] Meanwhile, the present invention described above can calculate an accurate optical flow of an object's movement regardless of whether the camera (200) moves.

[0101] Furthermore, the various embodiments described herein may be implemented, for example, in a recording medium readable by a computer or similar device using software, hardware, or a combination thereof.

[0102] According to hardware implementation, the embodiments described herein may be implemented using at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers, microcontrollers, microprocessors, and other electrical units for performing functions. In some cases, the embodiments described herein may be implemented as the control module itself.

[0103] According to software implementation, embodiments such as the procedures and functions described herein may be implemented in separate software modules. Each of the software modules may perform one or more functions and operations described herein. Software code may be implemented as a software application written in a suitable programming language. The software code may be stored in a memory module and executed by a control module.

[0104] The above description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications, changes, and substitutions within the scope of the essential characteristics of the present invention without departing from its nature.

[0105] Accordingly, the embodiments disclosed in this invention and the accompanying drawings are intended to illustrate, not limit, the technical concept of the invention, and the scope of the technical concept of the invention is not limited by such embodiments and accompanying drawings. The scope of protection of this invention shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of this invention.

Claims

Claim 1 A method for calculating optical flow considering the relative speed of a processor comprises: a step of acquiring driving images of the surroundings of a moving object as the moving object moves; a step of acquiring moving speed data of a camera as the moving object moves; and a step of calculating an optical flow that reflects the relative speed of an object within the driving images using an optical flow model with a learned relative speed applied based on a loss function determined based on the moving speed data of the camera. The learning of the optical flow model is characterized by identifying a stationary object within the driving images for learning, and including a loss term within the loss function that defines the error between the relative speed between the identified stationary object and the camera and the moving speed of the camera, thereby training the optical flow model in a direction in which the loss of the loss function becomes smaller. Claim 2 A method for calculating optical flow according to claim 1, wherein the step of calculating the optical flow comprises comparing pixels between a first driving image and a second driving image that are sequentially captured in time among the driving images to determine the position where the pixels have moved and calculating the optical flow within the second driving image. Claim 3 In claim 2, the step of calculating the optical flow is characterized in that the optical flow model is learned in a direction in which the relative velocity loss is reduced based on the relative velocity of a stationary object and the moving speed of a camera. Claim 4 In claim 2, the step of calculating the optical flow is characterized by applying the relative speed of an object within the first driving image and the second driving image by considering pixels that have moved according to the driving of the moving body based on the moving speed data of the camera. Claim 5 An optical flow calculation method according to claim 2, characterized in that the first driving image is a driving image taken immediately before the second driving image. Claim 6 A method for calculating optical flow according to claim 1, characterized in that the optical flow is data representing the moved positions of pixels within a driving image as vectors. Claim 7 An optical flow calculation method according to claim 1, further comprising: a step of acquiring position data of the moving body; and a step of acquiring road surface condition data around the moving body based on the position data. Claim 8 An optical flow calculation method according to claim 7, characterized in that the road surface condition data includes information regarding the slope of the road surface, the curvature of the road surface, and the roughness of the road surface. Claim 9 An optical flow calculation method according to claim 7, further comprising the step of calculating noise data of the camera based on the road surface condition data, wherein the noise data is data regarding the direction and magnitude of shaking of the camera due to the road surface condition while the moving body is driving. Claim 10 A method for calculating optical flow according to claim 9, further comprising the step of correcting the optical flow calculated based on the noise data. Claim 11 A program recorded on a computer-readable recording medium having program code for executing an optical flow calculation method according to any one of claims 1 to 10. Claim 12 A computer-readable recording medium storing a program that performs an optical flow calculation method according to any one of claims 1 to 10.

Images