Method and computer system for increasing camera video frame

Abstract

The present invention relates to a method and computer system for increasing camera image frames using an existing camera without adding a camera. A method for increasing camera image frames performed in a computing system according to one embodiment of the present invention, in a method for processing images captured by two or more cameras, may include: receiving a first image, which is an image captured by a first camera, and a second image, which is an image captured by the second camera; and sampling the first image at a specific period T1 and sampling the second image, wherein the sampling time of the second image is different from the sampling time of the first image.

Description

Technology Field

[0001] The present invention relates to a method and computer system for increasing camera image frames using an existing camera without adding a camera. Background Technology

[0002] Recently, vehicles equipped with autonomous driving functions or Advanced Driver Assistance Systems (ADAS) determine various risks by utilizing images from multiple cameras mounted on the vehicle. Since the distance traveled by the vehicle between video frames captured by cameras increases when the vehicle is traveling at high speeds compared to when it is traveling at low speeds, advanced driver assistance systems require video inputs at faster intervals to reliably handle dangerous situations. However, there is a problem in that the number of video frames that an image processing device can process per second is limited. The problem to be solved

[0003] The present invention relates to a method and computer system for increasing camera image frames using an existing camera without adding a camera.

[0004] The present invention relates to a vehicle capable of stably handling dangerous situations during high-speed driving by providing video frames at faster intervals.

[0005] Other unspecified objects of the present invention may be further considered to the extent that they can be easily inferred from the following detailed description and effects. means of solving the problem

[0006] A method for increasing camera image frames performed in a computing system according to an embodiment of the present invention for achieving the above-described purpose may include, in a method for processing images captured by two or more cameras, a step of receiving a first image, which is an image captured by a first camera, and a second image, which is an image captured by the second camera; and a step of sampling the first image at a specific period T1 and sampling the second image, wherein the sampling time of the second image is different from the sampling time of the first image.

[0007] The sampling period in the second image is the T1 period, and the sampling time of the second image can be sampled such that a time interval of T / 2 occurs between the sampling time of the first image and the sampling time of the second image.

[0008] The above-described camera image frame increase method may further include: a simultaneous sampling step of extracting a first frame, which is a frame at a specific time in the first image, by simultaneous sampling from the first image and the second image, and extracting a second frame, which is a frame at the specific time in the second image; a step of obtaining information regarding a common area between the first frame and the second frame; and a step of cropping the frames sampled in the step of sampling to create the gap using the information regarding the common area.

[0009] The above-described method for increasing camera image frames further includes the step of extracting feature points from the first frame and the second frame; and the step of obtaining information regarding the common area may obtain information regarding the common area of ​​the first frame and the second frame using the feature point information.

[0010] The above-described method for increasing camera image frames may further include: a step of calculating information for correcting distortion due to the difference in field of view between the first camera and the second camera; and a step of correcting the cropped frames using the information for correcting distortion.

[0011] At least one of the first camera and the second camera may be pivoted such that a portion of the field of view of the first camera and a portion of the field of view of the second camera overlap.

[0012] The first camera above is a fixed camera or a variable camera, and the second camera above may be a variable camera.

[0013] The step of extracting the above feature points may extract the feature points using at least one of the SIFT (Scale-Invariant Feature Transform) algorithm, SURF (Speeded Up Robust Feature) algorithm, Harris corner detection algorithm, FAST (Features from Accelerated Segment Test) algorithm, AKAZE algorithm, and ORB (Oriented FAST and Rotated BRIEF).

[0014] It may further include a data storage step of combining and storing frames sampled from the first image and frames sampled from the second image.

[0015] A method for increasing camera image frames performed in a computing system according to another embodiment of the present invention for achieving the above-described purpose comprises, in a method for processing images captured by two or more cameras, the step of receiving from the first camera frames sampled at a specific period T1 from an image captured by the first camera; the step of receiving from the second camera frames sampled at a period T1 from a second image captured by the second camera; and the step of transmitting a control signal to the second camera such that a time interval between the sampling time of the second image and the sampling time of the first image occurs, wherein the frames received from the second camera are frames sampled using the control signal.

[0016] The above camera image frame increase method may further include: a step of obtaining information regarding a common area using a first frame, which is a frame at a specific time in the first image, and a second frame, which is a frame at the specific time in the second image, extracted by simultaneous sampling from the first image and the second image; and a step of cropping the frames sampled at the T1 period received from the first camera and the second camera using the information regarding the common area.

[0017] A computing system according to one embodiment of the present invention for achieving the above-described purpose comprises: a processor; and a memory communicating with the processor, wherein the memory stores instructions that cause the processor to perform operations, and the operations may include: an operation of receiving a first image, which is an image captured by a first camera, and a second image, which is an image captured by a second camera; and an operation of sampling the first image at a specific period T1 and sampling the second image, wherein the sampling time of the second image is different from the sampling time of the first image.

[0018] The above operations may further include: a simultaneous sampling operation of extracting a first frame, which is a frame at a specific time in the first image, by simultaneous sampling from the first image and the second image, and extracting a second frame, which is a frame at the specific time in the second image; an operation of obtaining information regarding a common area between the first frame and the second frame; an operation of cropping the frames sampled in the step of sampling to create the gap using the information regarding the common area; an operation of calculating information for correcting distortion due to the difference in field of view between the first camera and the second camera; and an operation of correcting the cropped frames using the information for correcting distortion.

[0019] A vehicle according to another embodiment of the present invention for achieving the above-described purpose comprises, in the vehicle, a first camera mounted on the vehicle; a second camera mounted on the vehicle; and a computing device, wherein the computing device comprises a processor; and a memory communicating with the processor, wherein the memory stores instructions that cause the processor to perform operations, and the operations may include: an operation of receiving a first image, which is an image captured by the first camera, and a second image, which is an image captured by the second camera; and an operation of sampling the first image at a specific period T1 and sampling the second image, wherein the sampling time of the second image is different from the sampling time of the first image.

[0020] When the vehicle enters a preset condition, it may further include an operation of pivoting at least one of the first camera and the second camera.

[0021] Meanwhile, a program stored in a recording medium according to another embodiment of the present invention for achieving the above-described purpose may include program code for performing the above-described method for increasing camera image frames. Effects of the invention

[0022] According to the present invention, video frames at faster intervals can be provided using an existing camera without adding or changing the camera.

[0023] In addition, according to the present invention, autonomous driving performance or the performance of an advanced driver assistance system can be improved during high-speed driving of a vehicle.

[0024] Even if an effect is not explicitly mentioned herein, the effects and potential effects described in the following specification expected by the technical features of the present invention are treated as described in the specification of the present invention. Brief explanation of the drawing

[0025] FIG. 1 is a block diagram illustrating an example of an internal component of a computing system according to an embodiment of the present invention. FIG. 2 is a schematic diagram showing a case where a computing system according to one embodiment of the present invention is applied to a vehicle. FIG. 3 is a flowchart relating to a method for increasing camera image frames according to an embodiment of the present invention. FIG. 4 is a schematic diagram illustrating an example of sampling a first image and a second image such that the sampling times may intersect each other. FIG. 5 is a flowchart relating to a method for increasing camera image frames according to another embodiment of the present invention. Figure 6 is a schematic diagram showing an example of performing camera pivoting, using the case of a vehicle as an example. FIG. 7 is a first flowchart relating to a method for increasing camera image frames according to another embodiment of the present invention. FIG. 8 is a second flowchart relating to a method for increasing camera image frames according to another embodiment of the present invention. Specific details for implementing the invention

[0026] The following description merely illustrates the principles of the present invention. Therefore, those skilled in the art may invent various devices that embody the principles of the present invention and are included within the concept and scope of the present invention, even though they are not explicitly described or illustrated in this specification. Furthermore, all conditional terms and embodiments listed in this specification are, in principle, explicitly intended only for the purpose of understanding the concept of the present invention and should be understood not as being limited to the embodiments and conditions specifically listed as such.

[0027] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions may include plural expressions unless the context clearly indicates otherwise. The term "and / or" includes a combination of multiple related described items or any of the multiple related described items. Additionally, the term "part" as used herein may refer to components and functions provided within the parts, which may be combined into a smaller number of components and parts or further separated into additional components and parts.

[0028] The aforementioned objectives, features, and advantages will become clearer through the following detailed description in conjunction with the attached drawings, and accordingly, a person skilled in the art to which the present invention pertains will be able to easily implement the technical concept of the present invention.

[0029] In the attached drawings, identical or corresponding components are given the same reference numerals. Additionally, in the description of the following embodiments, the description of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0030] Additionally, the terms 'module' or 'part' as used in the specification refer to software or hardware components, and the 'module' or 'part' performs certain roles. However, the meaning of 'module' or 'part' is not limited to software or hardware. The 'module' or 'part' may be configured to reside in an addressable storage medium or configured to run one or more processors. Thus, as an example, the 'module' or 'part' may include at least one of components such as software components, task components, class components, and object-oriented software components. Furthermore, the 'module' or 'part' may include at least one of processes, drivers, microcode, segments of program code, attributes, procedures, subroutines, firmware, circuits, functions, data, databases, data structures, or variables. The functions provided within the components and 'modules' or 'parts' may be combined into a smaller number of components and 'modules' or 'parts', or further separated into additional components and 'modules' or 'parts'. 'Software' may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0031] FIG. 1 is a block diagram illustrating an example of an internal component of a computing system according to an embodiment of the present invention.

[0032] For example, a camera image frame increase method according to an embodiment of the present invention can be implemented through a computing system (100) of FIG. 1. As illustrated in FIG. 1, the computing system (100) may include an input / output interface (102), a memory (104), a communication interface (106), a processor (108), and a bus (110) as components for executing the camera image frame increase method. The computing system (100) may further include storage (114), etc.

[0033] The input / output interface (102) may be a means for interfacing with an input / output device. The input device may include a device such as a keyboard, mouse, or microphone. The output device may include a device such as a display or speaker. Alternatively, the input / output interface (102) may be a means for interfacing with a device in which the functions for input and output are integrated into one, such as a touchscreen. The input / output device may be a device separate from the computing system (100), or it may be configured as a single system with the computing system (100).

[0034] The memory (104) stores programming and data configurations that provide the functions of some or all of the modules described herein. For example, it may include logic that enables the execution of selected embodiments of the camera image frame increase method described below. The memory (104) may include volatile memory, permanent, virtual, or other memory for storing information used by or output by the computing system (100). For example, the memory (104) may include a non-perishable mass storage device such as Random Access Memory (RAM), Dynamic RAM, Read Only Memory (ROM), and / or a disk drive as a computer-readable recording medium. Here, the non-perishable mass storage device such as the ROM and the disk drive may be included in the computer device (100) as a separate storage (112) distinct from the memory (104). Additionally, the memory (104) may store any information, such as state information of the computing system (100), an operating system, and / or program code. Various software components may be loaded into memory (104) from a computer-readable recording medium separate from memory (104). This separate computer-readable recording medium may include computer-readable recording media such as disks, floppy drives, DVD / CD-ROM drives, tapes, and / or memory cards. Alternatively, software components may be loaded into memory (104) via a communication interface (106) that is not a computer-readable recording medium. Memory (104) may also be used to store instructions of a computing system (100) including instructions for increasing camera video frames.

[0035] The communication interface (106) may include interfaces for networks such as the Internet or a local area network. The communication interface (106) may include interfaces for wired or wireless connections. Additionally, the communication interface (106) may include an antenna or a wired / wireless transceiver. Inputs or configuration commands for increasing camera video frames may be received through the communication interface (106).

[0036] The processor (108) may include at least one of a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), microprocessor, digital signal processor, microcontroller, application processor (AP), and logic elements capable of performing similar functions. The processor (108) may be configured to process instructions of a computer program. For example, the processor (108) may include or be part of any device capable of processing a sequence of instructions as a component for increasing camera image frames. A method for increasing camera image frames is described in detail with reference to FIGS. 2 through 9. One or more programs may include one or more computer-executable instructions, and the computer-executable instructions may be configured to cause the computing system (100) to perform operations according to exemplary embodiments when executed by the processor (108). Instructions of the computer program may be provided to the processor (108) by memory (104) or a communication interface (106). The processor (108) can be connected to the memory (120) via the bus (110). For example, the processor (108) can be configured to execute instructions received according to program code stored in a recording device such as memory (104).

[0037] The bus (150) can interconnect various other components of the computing system (100), including the processor (108) and memory (104). For example, it can transport data between the processor (108) and memory (104). The bus (150) may include wireless and / or wired communication media between the components of the computing system (100) and may include parallel, serial, or other topological arrangements.

[0038] The computer device (100) described with reference to FIG. 1 may include more or fewer components than the components of FIG. 1.

[0039] FIG. 2 is a schematic diagram showing a case where a computing system according to one embodiment of the present invention is applied to a vehicle.

[0040] Referring to FIG. 2, the computing system (100) can be mounted on a vehicle (1000). The computing system (100) can be utilized in cases where shorter frame intervals are required during image analysis in addition to the vehicle (1000), but the vehicle (1000) is used as an example to help understand the invention.

[0041] Vehicles (1000) are often equipped with advanced driver assistance systems (ADAS) that perform various functions such as lane keeping assistance, lane departure warning, adaptive cruise control, forward collision avoidance, emergency braking, or parking assistance. These advanced driver assistance systems provide various functions to the driver by sampling images captured by multiple cameras (200) mounted on the vehicle (1000) and analyzing the sampled frames. Since the vehicle's movement distance between frames increases as the vehicle's speed increases, analyzing images with frames at shorter intervals can prevent driver risk and enhance safety.

[0042] The first camera (210) and the second camera (220) may be cameras with the same performance. Additionally, they may be cameras with different performance and different functions.

[0043] For example, the first camera (210) may be a fixed camera and the second camera (220) may be a variable camera. Alternatively, both the first camera (210) and the second camera (220) may be fixed cameras or variable cameras. A variable camera is a camera that can change the focus, angle of view, and / or zoom, etc., depending on the situation or control conditions.

[0044] The camera (200) and the computing system (100) may be connected via wired or wireless communication. The camera (200) may store and transmit captured images. The camera (200) may transmit captured images to the computing system (100). Alternatively, the camera (200) may transmit the captured images to the computing system (100) after performing a preprocessing process. For example, the camera (200) may perform sampling of the captured images according to camera performance and / or sampling control commands, and transmit the sampled frames to the computing system (100). Sampling control commands may be determined according to signals transmitted by the computing system (100).

[0045] The computing system (100) can perform various analyses for driver assistance using images or sampled frames received from the camera (200). The computing system (100) can perform image processing or control commands for image processing to make the captured images or sampled frames suitable for analysis. For example, the computing system (100) can generate frames with shorter intervals to be used for image analysis when the vehicle is in high-speed driving mode. The computing system (100) can transmit control commands related to the camera (200) to generate frames with shorter intervals. The related control commands may be commands regarding changing the sampling point or pivoting commands for adjusting the camera's field of view.

[0046] FIG. 3 is a flowchart relating to a method for increasing camera image frames according to an embodiment of the present invention.

[0047] Referring to FIG. 3, a method for increasing camera image frames according to an embodiment of the present invention will be described. The method for increasing camera image frames according to an embodiment of the present invention can be performed by a computing system (100).

[0048] The computing system (100) receives images captured by cameras (200) (step S310). There may be two or more cameras (200), but for the sake of understanding the invention, the case where there are two cameras is described as an example. The computing system (100) receives a first image, which is an image captured by the first camera. The computing system (100) receives a second image, which is an image captured by the second camera.

[0049] The computing system (100) samples the first image and the second image such that the sampling times intersect each other when performing sampling (step S320). The computing system (100) can combine and store the frames sampled from the first image and the frames sampled from the second image (step S330).

[0050] Specifically, the computing system (100) samples the first image at a specific period (T1). The computing system (100) samples the second image, but the sampling time of the second image may be different from the sampling time of the first image. For example, the computing system (100) may perform sampling such that the sampling time of the second image is approximately an intermediate point between the sampling times of the first image. The specific period (T1) may be set and changed based on the performance of the computing system (100) or period setting conditions.

[0051] For example, if the sampling period of the second image is the same as the sampling period (T1) of the first image, the computing system (100) can set the sampling time of the second image such that it differs from the sampling time of the first image by T / 2.

[0052] FIG. 4 is a schematic diagram illustrating an example of sampling a first image and a second image such that the sampling times may intersect each other.

[0053] Referring to FIG. 4, the computing system (100) performs sampling at a specific period T1 in the image of the first camera (210), such as the first camera sampling clock (41). Additionally, the computing system (100) performs sampling at a period T1 in the image of the second camera (220), such as the second camera sampling clock (43). However, the computing system (100) can control the sampling time so that there is a certain time difference (t) between the first camera sampling time and the second camera sampling time. In FIG. 4, t may be time T1 / 2. As the computing system (100) adjusts the sampling time so that t is close to time T1 / 2, there is an advantage in image analysis because the time interval between frames is constant when combining the sampled frames of the first image and the sampled frames of the second image.

[0054] The fact that the computing system (100) samples such that the sampling times intersect each other may mean that, as shown in FIG. 4, the first image is not sampled when the second image is not sampled, and the first image is sampled when the second image is not sampled. However, if the sampling period of the first image and the sampling period of the second image are different, some of the sampling may occur simultaneously.

[0055] The computing system (100) can combine and store frames (42) sampled from the first image and frames (44) sampled from the second image. The stored frames (45) can be used for image analysis.

[0056] Combining the frames (42) sampled from the first video and the frames (44) sampled from the second video does not mean synthesizing the frames, but rather means collecting them in the order of the time of sampling. If the frames (42) sampled from the first video are sampled at 30 frames per second and the frames (42) sampled from the second video are sampled at 30 frames per second, the frames (45) that combine the frames (42) sampled from the first video and the frames (44) sampled from the second video can achieve the effect of sampling 60 frames per second.

[0057] The frames sampled in step S320 can crop the portion corresponding to the common area and correct image distortion caused by differences in viewing angle.

[0058] Sampled frames can be combined to obtain frames with shorter intervals and used for analysis, but for more accurate analysis, the area corresponding to the common region between the frames sampled from the first image and the frames sampled from the second image can be cropped. Taking an advanced driver assistance system of a vehicle as an example, the area around the driving direction where the field of view of the first camera (210) and the second camera (220), which are mounted on the front of the vehicle and photograph the front, overlap can be the most important area, and in high-speed driving mode, the area around the driving direction can be a more important area. Accordingly, to improve the accuracy of image analysis, the computing system (100) can crop all or part of the area where the field of view of the first camera (210) and the second camera (220) match as a common region. Alternatively, if the first camera (210) and the second camera (220) are fixed cameras, the computing system (100) can pre-define information regarding the common region and perform cropping using the pre-defined common region information.

[0059] The computing system (100) can correct image distortion caused by the difference in field of view between the first camera (210) and the second camera (220). The computing system (100) can correct image distortion using information regarding a common area. For example, the computing system (100) can correct image distortion using feature point information, but is not limited to this method and can utilize various known techniques. However, if the first camera (210) and the second camera (220) are fixed cameras, the computing system (100) can correct the distortion of sampled frames or cropped frames after sampling using pre-set image distortion correction information.

[0060] In the case where at least one of the first camera (210) and the second camera (220) is a variable camera, cropping may be performed using predefined common area information or correction may be performed using pre-set image distortion correction information. However, in the process of performing pivoting to match the viewing angles of the first camera (210) and the second camera (220) to a certain level or higher, the computing system (100) may need to perform an operation to calculate information regarding the common area and information for correcting image distortion due to the difference in viewing angles. This is explained with reference to FIG. 5.

[0061] FIG. 5 is a flowchart relating to a method for increasing camera image frames according to another embodiment of the present invention.

[0062] To aid in understanding the invention, the cameras (200) are described by assuming that a first camera (210) and a second camera (220) exist.

[0063] The computing system (100) can pivot the first camera (210) and the second camera (220) (step S510). The computing system (100) can perform pivoting by transmitting a pivoting request signal to the cameras (200). The computing system (100) pivoting the cameras (200) includes both the process of pivoting the cameras directly and pivoting through a pivoting request signal.

[0064] If both the first camera (210) and the second camera (220) are variable cameras, the computing system (100) may pivot both the first camera (210) and the second camera (220) or pivot at least one of the cameras. If only one of the first camera (210) and the second camera (220) is a variable camera and the other is a fixed camera, the computing system (100) may pivot only the variable camera.

[0065] Figure 6 is a schematic diagram showing an example of performing camera pivoting, using the case of a vehicle as an example.

[0066] Referring to FIG. 6, a first camera (210) and a second camera (220) are mounted on the front of a vehicle (1000) to capture images. The field of view of the first camera (210) is equal to 61a, and the field of view of the second camera (220) is equal to 62a. Taking the vehicle as an example, pivoting can be performed to adjust the field of view of the cameras (200) according to driving conditions such as driving speed or rainy conditions. The conditions and degree of pivoting performed by the computing system (100) may be pre-set. For example, in the case of high-speed driving, the computing system (100) may pivot the first camera (210) and the second camera (220) according to the pre-set conditions to pivot the field of view of the first camera (210) to be equal to 62b and pivot the field of view of the second camera (220) to be equal to 62b. The computing system (100) performs pivoting in this manner so that a certain area of ​​the field of view of the first camera (210) and the field of view of the second camera (220) overlaps, creating a common area (63) between the first image and the second image.

[0067] Referring again to FIG. 5, the computing system (100) can receive a first image from a pivoted first camera and a second image from a pivoted second camera (step S520).

[0068] The computing system (100) performs sampling in the first image and performs sampling in the second image at a time different from the sampling time of the first image (step S530). Specific examples and details regarding the sampling of the computing system (100) such that the sampling times intersect when sampling in the first image and the second image can be easily understood through the explanation with reference to FIGS. 3 and FIGS. 4.

[0069] In order to effectively utilize the cross-sampled frames in step S560 for image analysis, the computing system (100) may crop the portion corresponding to the common area between the frame obtained from the first image and the frame obtained from the second image (step S570). Additionally, the computing system (100) may correct distortions caused by differences in viewing angles between cameras, etc., so that the frames obtained in step S530 or step S570 can be used more effectively for image analysis (step S580).

[0070] The computing system (100) can obtain information regarding the area to be cropped in step S570 by performing steps S540 and S550. The area to be cropped may be an area corresponding to the common area between the first image and the second image. The area to be cropped may be changed according to the user's settings and may be an area corresponding to a part of the common area. That is, the information regarding the common area obtained by the computing system (100) includes information regarding the area to be cropped.

[0071] The computing system (100) performs simultaneous sampling on the first image and the second image (step S540). That is, the computing system (100) can extract frames of the first image and frames of the second image corresponding to a specific point in time. The frame extracted from the first image by simultaneous sampling is called the first simultaneous sampling frame, and the frame extracted from the second image by simultaneous sampling is called the second simultaneous sampling frame.

[0072] The computing system (100) determines a region corresponding to a common region in the first simultaneous sampling frame and the second simultaneous sampling frame. For example, the computing system (100) can determine a region corresponding to a common region by extracting feature points from the first simultaneous sampling frame and the second simultaneous sampling frame. For example, the computing system (100) can determine a region corresponding to a common region by extracting feature points using at least one of the SIFT (Scale-Invariant Feature Transform) algorithm, SURF (Speeded Up Robust Feature) algorithm, Harris corner detection algorithm, FAST (Features from Accelerated Segment Test) algorithm, AKAZE algorithm, and ORB (Oriented FAST and Rotated BRIEF).

[0073] The computing system (100) can obtain information regarding a common area, which is information regarding an area to be cropped (step S550). The computing system (100) can set the information regarding the common area to designate all or part of the common area as a cropping target area.

[0074] Additionally, the computing system (100) can calculate information for distortion correction based on the difference in field of view between the first camera (210) and the second camera (220) using various methods (step S560). For example, the computing system (100) can calculate information for distortion correction using feature point information. Alternatively, the computing system (100) can calculate information for distortion correction using the field of view information of the camera.

[0075] The computing system (100) may repeat steps S540, S550, and S560 multiple times to produce information regarding the optimized common area and information for distortion correction. Additionally, the computing system (100) may repeat steps S540, S550, and S560 multiple times before performing steps S530, S570, and S580 to produce information regarding the optimized common area and information for distortion correction.

[0076] The computing system (100) can crop (step S570) and perform distortion correction (step S580) on the cross-sampled frames in step S530 using information about the common region, and then combine and store them according to the time of shooting or the time of sampling.

[0077] In this specification, identification symbols for each step are used for convenience of explanation and do not indicate the order of the steps; the steps may occur differently from the specified order unless the context clearly indicates a specific order. That is, the steps may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

[0078] In addition, some of the illustrated steps in the method for increasing camera image frames according to one embodiment of the present invention may be omitted or added. Furthermore, the method for increasing camera image frames according to one embodiment of the present invention may not occur in the illustrated order, and certain processes may be processed in parallel depending on the situation. For example, the order of steps S570 and S580 may be reversed, and the image simultaneously sampled in step S540 and the image cross-sampled in step S530 may be identical only in that they are images captured by the first camera and the second camera, respectively, but may not be images with identical captured content.

[0079] Additionally, the computing system (100) may perform steps S560, S570, and S580 when frames with shorter time intervals are required, such as in a high-speed driving mode. When frames with shorter time intervals are required, such as in a high-speed driving mode, the computing system (100) may perform steps S510, S520, S530, S540, and S550 before performing S560, S570, and S580 to produce information regarding a common area and information for distortion correction. Alternatively, the computing system (100) may perform steps S510, S520, S530, S540, and S550 in advance before entering a high-speed driving mode. For example, the computing system (100) may perform the operations of steps S510, S520, S530, S540, and S550 in advance for each pivoting setting condition used by the first camera (210) and the second camera (220) in a high-speed driving mode. In this case, the computing system (100) can obtain information regarding a common area and information for distortion correction in advance for each pivoting setting condition.

[0080] Additionally, the computing system (100) may repeat steps S530, S540, and S550 several times to produce more accurate information regarding the common area and information for distortion correction. That is, the computing system (100) may perform simultaneous sampling in step S530 at multiple specific times rather than at a specific time, and perform steps S540 and S550 for each sampled frame to produce more accurate information regarding the common area and information for distortion correction.

[0081] FIG. 7 is a first flowchart relating to a method for increasing camera image frames according to another embodiment of the present invention. FIG. 8 is a second flowchart relating to a method for increasing camera image frames according to another embodiment of the present invention.

[0082] Referring to FIGS. 7 and 8, a method for increasing camera image frames according to another embodiment of the present invention is described when the operation of sampling frames in an image is performed by a camera. FIGS. 7 and 8 describe the case where the first camera (210) and the second camera (220) are variable cameras, but the invention can also be understood by referring to FIGS. 2 to 6, which describes the case where the cameras are fixed. For example, in the case of a fixed camera, a pivoting process may be unnecessary, and cropping and distortion correction may be possible based on pre-set information regarding a common area or pre-set image distortion correction information without the need to calculate information regarding a common area or information for distortion correction through simultaneous sampling.

[0083] The computing system (100) may perform steps S705 to S760 to produce information regarding a common area and information for distortion correction based on the difference in viewing angle.

[0084] Specifically, when a computer system (100) satisfies a preset condition such as a high-speed driving situation, it transmits a control signal to the first camera (210) and the second camera (220) to pivot so that the viewing angles of the two variable cameras (210, 220) overlap at least partially (steps S710, S720). The pivoting control signal transmitted by the computer system (100) to the first camera (210) and the pivoting control signal transmitted to the second camera (220) may differ depending on the performance of the camera, the position of the camera, preset pivoting control conditions, etc.

[0085] The first camera (210) pivots the camera according to the pivoting control signal (step S705) and captures an image (step S715). The second camera (220) pivots the camera according to the pivoting control signal (step S710) and captures an image (step S715).

[0086] The computing system (100) transmits a simultaneous sampling request signal (step S725) to the first camera (210) and the second camera (220) (steps S725, S730). The computing system (100) may transmit the simultaneous sampling request signal to the first camera (210) and the second camera (220) simultaneously, or, if necessary, transmit it with a slight time difference.

[0087] The first camera (210) performs sampling at a specific point in time among the images captured by the first camera (210) in accordance with the simultaneous sampling request signal (step S725) (step S735). The second camera (220) also performs sampling at a specific point in time among the images captured by the second camera (220) in accordance with the simultaneous sampling request signal (step S730) (step S740). The first camera (210) transmits the frame sampled at a specific point in time to the computing system (100) (step S745). The second camera (220) transmits the frame sampled at a specific point in time to the computing system (100) (step S750).

[0088] The computing system (100) can obtain information regarding the common area by analyzing the area corresponding to the common area captured between the frames received in steps S745 and S750 (step S755). For example, the computing system (100) can extract feature points from the frames and compare the feature points to extract the common area by comparing the frame received in step S745 with the frame received in step S750, but the computing system (100) can utilize various known techniques in addition to the method of comparing feature points.

[0089] The computing system (100) can obtain information regarding the common area using the extracted common area (step S755). The computing system (100) can also calculate information for distortion correction based on the difference in field of view between the first camera and the second camera (step S760).

[0090] Referring to FIG. 8, the computing system (100) can transmit a pivoting control signal to the first camera (210) according to a pre-set pivoting control condition (step S810). For example, the computing system (100) can transmit a pivoting control signal when the vehicle is traveling at high speed. Additionally, the computing system (100) can transmit a signal requesting an image frame to be captured after pivoting to the first camera (210) (step S810). The computing system (100) may include information regarding the sampling time and period in the image frame request signal. The first camera (210) pivots the first camera (210) according to the pivoting control signal, and can transmit frames sampled from the image captured by the first camera (210) at a period according to the image frame request signal or a pre-set period T1 to the computing system (100) (step S830).

[0091] The computing system (100) can transmit a pivoting control signal to the second camera (220) according to a pivoting control condition set in advance (step S820). Additionally, the computing system (100) can transmit a signal requesting a video frame to be captured after pivoting to the second camera (220) (step S820). The computing system (100) may include information regarding the sampling time and period in the video frame request signal. For example, the computing system (100) may transmit a video frame request signal such that the sampling period of the second video captured by the second camera is the same as the sampling period of the first video, but the sampling time is out of sync with the sampling time of the first video.

[0092] The second camera (220) pivots the second camera (220) according to a pivoting control signal, and samples frames from the image captured by the second camera (220) at a period of T1, which is a period according to the image frame request signal, such that the sampling time is offset from the sampling time of the first image, and transmits the sampled frames to the computing system (100) (step S840).

[0093] The computing system (100) may know in advance the sampling period of the first camera (210) and the sampling period of the second camera (220).

[0094] The computing system (100) can crop the common area from the frames received in steps S830 and S840 using information regarding the common area obtained in step S755 of FIG. 7 (step S850). If frames with only the common area cropped from frames received from different cameras are combined and stored, it is more accurate and easier to use in analysis.

[0095] Additionally, the computing system (100) can correct distortion caused by the difference in viewing angles of the cropped frames using information for distortion correction obtained in step S760 of FIG. 7 (step S860). After correcting the distortion caused by the difference in viewing angles, combining and storing the frames makes it more accurate and easier to use in analysis.

[0096] The computing system (100) can combine and store cropped and distortion-corrected frames in chronological order (step S870). If the first camera (210) has the capability to sample 60 frames per second and the second camera (220) has the capability to sample 60 frames per second, the set of frames combined and stored in step S870 can represent a capability to sample 120 frames per second. That is, the time between frames is shortened, so that stability can be improved if used to determine danger during high-speed driving of a vehicle.

[0097] By using the camera image frame increase method according to the embodiments of the present invention, image frames at faster intervals can be provided using an existing camera without adding or changing cameras. In addition, by using the camera image frame increase method according to the embodiments of the present invention, autonomous driving performance or the performance of an advanced driver assistance system can be improved during high-speed driving of a vehicle. Even if effects are not explicitly mentioned herein, the effects and potential effects described in the following specification expected by the technical features of the present invention shall be treated as described in the specification of the present invention.

[0098] Furthermore, the various embodiments described herein may be implemented, for example, in a recording medium readable by a computer or similar device using software, hardware, or a combination thereof.

[0099] According to hardware implementation, the embodiments described herein may be implemented using at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers, microcontrollers, microprocessors, and other electrical units for performing functions. In some cases, the embodiments described herein may be implemented as the control module itself.

[0100] According to software implementation, embodiments such as the procedures and functions described herein may be implemented in separate software modules. Each of the software modules may perform one or more functions and operations described herein. Software code may be implemented as a software application written in a suitable programming language. The software code may be stored in a memory module and executed by a control module.

[0101] The above description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications, changes, and substitutions within the scope of the essential characteristics of the present invention without departing from its nature.

[0102] Accordingly, the embodiments disclosed in this invention and the accompanying drawings are intended to illustrate, not limit, the technical concept of the invention, and the scope of the technical concept of the invention is not limited by such embodiments and accompanying drawings. The scope of protection of this invention shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of this invention.

Claims

Claim 1 A method for processing images captured by two or more cameras, comprising: receiving a first image captured by a first camera and a second image captured by a second camera; and sampling the first image at a specific period T1 and sampling the second image, wherein the sampling time of the second image is different from the sampling time of the first image, and pivoting at least one of the first camera and the second camera such that a portion of the field of view of the first camera overlaps with a portion of the field of view of the second camera. Claim 2 A method for increasing camera image frames according to claim 1, wherein the sampling period in the second image is the T1 period, and the sampling time of the second image is sampled such that a time interval of T1 / 2 occurs with the sampling time of the first image. Claim 3 In claim 2, the camera image frame increase method further comprises: a simultaneous sampling step of extracting a first frame, which is a frame at a specific time in the first image, by simultaneous sampling from the first image and the second image, and extracting a second frame, which is a frame at the specific time in the second image; a step of obtaining information regarding a common area between the first frame and the second frame; and a step of cropping the frames sampled in the step of sampling to cause the time interval to occur using the information regarding the common area. Claim 4 In claim 3, the camera image frame increase method further comprises the step of extracting feature points from the first frame and the second frame; and the step of obtaining information regarding the common area comprises obtaining information regarding the common area of ​​the first frame and the second frame using the feature point information. Claim 5 In claim 3, the camera image frame increase method further comprises: a step of calculating information for correcting distortion due to the difference in viewing angle between the first camera and the second camera; and a step of correcting the cropped frames using the information for correcting distortion. Claim 6 delete Claim 7 A method for increasing camera image frames according to claim 1, wherein the first camera is a fixed camera or a variable camera, and the second camera is a variable camera. Claim 8 In claim 4, the step of extracting the feature points is a camera image frame increase method that extracts the feature points using at least one of SIFT (Scale-Invariant Feature Transform) algorithm, SURF (Speeded Up Robust Feature) algorithm, Harris corner detection algorithm, FAST (Features from Accelerated Segment Test) algorithm, AKAZE algorithm, and ORB (Oriented FAST and Rotated BRIEF). Claim 9 A camera image frame increase method according to claim 1, further comprising a data storage step of combining and storing frames sampled from the first image and frames sampled from the second image. Claim 10 A method for processing images captured by two or more cameras, comprising: receiving from the first camera frames sampled at a specific period T1 from a first image captured by the first camera; receiving from the second camera frames sampled at a period T1 from a second image captured by the second camera; and transmitting a control signal to the second camera such that a time interval between the sampling time of the second image and the sampling time of the first image occurs, wherein the frames received from the second camera are frames sampled using the control signal, and pivoting at least one of the first camera and the second camera such that a portion of the field of view of the first camera and a portion of the field of view of the second camera overlap. Claim 11 In claim 10, the camera image frame increase method further comprises: a step of obtaining information regarding a common area using a first frame, which is a frame at a specific time in the first image, and a second frame, which is a frame at the specific time in the second image, extracted by simultaneous sampling in the first image and the second image; and a step of cropping the frames sampled at the T1 period received from the first camera and the second camera using the information regarding the common area. Claim 12 A computer-readable recording medium storing a program that performs a camera image frame increase method according to any one of claims 1 to 5 and claims 7 to 11. Claim 13 A computing system comprising: a processor; and a memory communicating with the processor, wherein the memory stores instructions that cause the processor to perform operations, and the operations include: receiving a first image, which is an image captured by a first camera, and a second image, which is an image captured by a second camera; and sampling the first image at a specific period T1 and sampling the second image, wherein the sampling time of the second image is different from the sampling time of the first image, and pivoting at least one of the first camera and the second camera such that a portion of the field of view of the first camera and a portion of the field of view of the second camera overlap. Claim 14 In claim 13, the operations further comprise: a simultaneous sampling operation of extracting a first frame, which is a frame at a specific time in the first image, by simultaneous sampling from the first image and the second image, and extracting a second frame, which is a frame at the specific time in the second image; an operation of obtaining information regarding a common area between the first frame and the second frame; an operation of cropping the frames sampled in the step of sampling to create a time interval using the information regarding the common area; an operation of calculating information for correcting distortion due to the difference in field of view between the first camera and the second camera; and an operation of correcting the cropped frames using the information for correcting distortion. Claim 15 A vehicle comprises: a first camera mounted on the vehicle; a second camera mounted on the vehicle; and a computing device, wherein the computing device comprises: a processor; and a memory communicating with the processor, wherein the memory stores instructions that cause the processor to perform operations, and the operations include: receiving a first image, which is an image captured by the first camera, and a second image, which is an image captured by the second camera; and sampling the first image at a specific period T1 and sampling the second image, wherein the sampling time of the second image is different from the sampling time of the first image, and further comprising, when the vehicle enters a preset condition, pivoting at least one of the first camera and the second camera, such that the pivoting overlaps a portion of the field of view of the first camera and a portion of the field of view of the second camera. Claim 16 delete

Images