Electronic device and control method thereof

Abstract

The invention discloses an electronic device and a control method thereof. An electronic device includes a communication means that receives object information of a target captured by a camera and identified based on a deep learning model. The electronic device also includes a memory that stores calibration information of the camera and physical information of the target. The electronic device additionally includes a processor configured to determine movement information of the target based on the object information. The processor is further configured to determine a first physical quantity to the target based on the movement information and the calibration information. The processor is further configured to determine a second physical quantity to the target based on the movement information and the physical information. The processor is additionally configured to compare the first physical quantity to the second physical quantity to determine whether to adjust the calibration information.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to Korean Patent Application No. 10-2023-0101656 filed in the Korean Intellectual Property Office on August 3, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to a method for adjusting calibration information to estimate the distance to an object recognized from a camera, and more particularly, to a method for using deep learning based two-dimensional (2D) bounding box information and key point information when recognizing an object of interest using camera images. Background Art

[0004] The process of using a camera to recognize an object includes the steps of inputting an image received by the camera into an image processing controller used in a vehicle and executing logic for autonomous driving image recognition in the image processing controller, and receiving a result value of an image recognition deep learning model and processing the result value as a signal for vehicle control.

[0005] In the step of receiving the result value of the image recognition deep learning model and processing the result value as a signal for vehicle control, the camera calibration information can be used to estimate the distance value in the real three-dimensional (3D) system based on the information in the image obtained by means of the camera. The calibration information can be a parameter value of the relationship between the table image and the real 3D system, which can be obtained in the vehicle end-of-line process (EOL) vehicle production stage for general mass-produced vehicles.

[0006] However, the position of a camera mounted on a vehicle or the angle at which the camera is mounted may change due to internal and external factors. When an error occurs in the calibration value, an error occurs in the distance and speed calculated for the target object.

[0007] The statements in this Background section merely provide background information related to the present disclosure and may not constitute prior art. Summary of the invention

[0008] The present disclosure has been made to solve the above-mentioned problems occurring in the prior art while keeping the advantages achieved by the prior art intact.

[0009] One aspect of the present disclosure provides an electronic device and a control method thereof, wherein the electronic device is used to obtain object information of a target captured by a camera and recognized based on a deep learning model, determine movement information of the target based on the object information, and determine whether to adjust calibration information according to the movement information of the target.

[0010] Another aspect of the present disclosure provides an electronic device for actively recognizing and compensating for slight warping of a camera, wherein the camera may be slightly warped when a vehicle is traveling, and provides an autonomous driving system having reliable vehicle control data and a control method thereof.

[0011] The technical problems to be solved by the present disclosure are not limited to the above-mentioned technical problems. A person skilled in the art in the field to which the present disclosure belongs should clearly understand other technical problems not mentioned in this article from the following description.

[0012] According to one aspect of the present disclosure, an electronic device is provided. The electronic device may include a communication device configured to receive object information of a target captured by a camera and identified based on a deep learning model. The electronic device may also include a memory storing calibration information of the camera and physical information of the target. The electronic device also includes a processor. The processor is configured to determine movement information of the target based on the object information. The processor is also configured to determine a first physical quantity to the target based on the movement information and the calibration information. The processor is further configured to determine a second physical quantity to the target based on the movement information and the physical information. The processor is further configured to compare the first physical quantity with the second physical quantity to determine whether to adjust the calibration information.

[0013] When the target is determined as a dynamic object based on the movement information, the processor may determine a distance to a lower end of a center of a bounding box included in the object information as the first physical quantity.

[0014] When the target is determined to be a dynamic object based on the movement information, the processor may determine the second physical quantity as a ratio between actual width information of the target and width information of an image about the target, wherein the actual width information is included in the physical information and the width information is included in the object information.

[0015] The processor may determine the second physical quantity by dividing i) a value obtained by multiplying the actual width information by the focal length of the camera by ii) the width information about the image.

[0016] When the target is determined as a static object based on the movement information, the processor may determine the first physical quantity using a movement amount of a key point included in the object information.

[0017] When the target is determined as a static object based on the movement information, the processor may receive a current speed through the communication device and may determine the current speed as the second physical quantity.

[0018] The processor may adjust the calibration information in response to determining that a difference between the first physical quantity and the second physical quantity is greater than or equal to a predetermined reference value.

[0019] The processor may determine an average value of the first physical quantity and an average value of the second physical quantity during a predetermined reference time, and may adjust the calibration information based on determining that a difference between the average value of the first physical quantity and the average value of the second physical quantity is greater than or equal to a predetermined reference value.

[0020] According to another aspect of the present invention, a method for controlling an electronic device is provided. The method may include receiving object information of a target captured by a camera and identified based on a deep learning model. The method may also include: determining movement information of the target based on the object information. The method may further include determining a first physical quantity to the target based on the movement information and calibration information of the target. The method may also include: determining a second physical quantity to the target based on the physical information and movement information of the target. The method may further include comparing the first physical quantity with the second physical quantity to determine whether to adjust the calibration information.

[0021] The determining of the first physical quantity may include determining, when the target is determined as a dynamic object based on the movement information, a distance to a lower end of a center of a bounding box included in the object information as the first physical quantity.

[0022] Determining the second physical quantity may include: when the target is determined to be a dynamic object based on the movement information, determining the second physical information as a ratio between actual width information of the target and width information of an image about the target, wherein the actual width information is included in the physical information and the width information is included in the object information.

[0023] Determining the second physical quantity may include dividing i) a value obtained by multiplying the actual width information by the focal length of the camera by ii) the width information about the image to determine the second physical quantity.

[0024] The determining of the first physical quantity may include determining the first physical quantity using a movement amount of a key point included in the object information when the target is determined as a static object based on the movement information.

[0025] Determining the second physical quantity may include, when the target is determined as a static object based on the movement information, receiving a current speed through a communication device, and determining the current speed as the second physical quantity.

[0026] Determining whether to adjust the calibration information may include adjusting the calibration information in response to determining that a difference between the first physical quantity and the second physical quantity is greater than or equal to a predetermined reference value.

[0027] Determining whether to adjust the calibration information may include: determining an average value of the first physical quantity and an average value of the second physical quantity during a predetermined reference time, and adjusting the calibration information based on a difference between the average value of the first physical quantity and the average value of the second physical quantity being greater than or equal to a predetermined reference value. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0029] Figure 1 is a diagram showing a control block diagram including an electronic device according to an embodiment;

[0030] Figure 2 is a diagram showing a control block diagram of an electronic device according to an embodiment;

[0031] Figure 3 is a diagram showing a bounding box used in an electronic device according to an embodiment;

[0032] Figure 4 is a diagram illustrating reference points of a bounding box in an electronic device according to an embodiment;

[0033] Figure 5 is a diagram showing width information about an image used in an electronic device according to an embodiment;

[0034] Figure 6 is a diagram showing physical information of an object stored in a memory of an electronic device according to an embodiment;

[0035] Figure 7 is a diagram illustrating a method for calculating a distance to an object based on a focal length in an electronic device according to an embodiment;

[0036] Figure 8 A diagram showing key point information used in an electronic device according to an embodiment;

[0037] Fig. 9 is a diagram showing an example of a calibration error in an electronic device according to an embodiment;

[0038] Fig.10 is a diagram showing a normal distribution graph used in an electronic device according to an embodiment;

[0039] Fig.11 is a diagram showing a control flow chart of an electronic device according to an embodiment; and

[0040] Fig.12 FIG. 2 is a diagram showing an electronic device according to an embodiment of the present invention. Fig.11 An illustration of the control flow graph that follows. DETAILED DESCRIPTION

[0041] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, and may include various changes, equivalents, or replacements of the corresponding embodiments.

[0042] With regard to the description of the drawings, like reference numerals may be used to refer to like or related elements.

[0043] The singular form of a noun corresponding to an item may include the item or a plurality of items unless the relevant context clearly indicates otherwise.

[0044] As used herein, each of the expressions "A or B", "at least one of A and B", "at least one of A or B", "at least one of A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include any and all combinations of one or more of the items listed with the corresponding expression in the expression.

[0045] Terms such as “first” and “second” or “first” and “second” may be used to simply distinguish a corresponding component from another component, and do not limit the components in any other aspects (eg, importance or order).

[0046] It should be understood that if any (e.g., first) component is referred to as being "coupled to," "coupled to," "connected to," or "connected to" another (e.g., second) component, with or without the term "operably" or "communicatively," this means that the element can be coupled to another element directly (e.g., via one or more wires), wirelessly, or via a third component.

[0047] The terms “include”, “comprising” and the like specify the existence of stated features, numbers, steps, operations, components, parts or combinations thereof, but do not exclude the existence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0048] When one component is “connected,” “coupled,” “supported” or “contacting” another component, this includes the case where the components are directly connected, coupled, supported or touching each other and the case where the components are indirectly connected, coupled, supported or touching each other through a third component.

[0049] When a component is located "on" another component, this includes the case where the other component exists between the two components as well as the case where one component is adjacent to the other component.

[0050] The term "and / or" includes a plurality of related described components or any combination of the plurality of related described components.

[0051] When a component, device, element, etc. of the present disclosure is described as having a purpose or performing an operation, function, etc., the component, device, or element should be considered herein as being "configured to" satisfy the purpose or perform the operation or function.

[0052] Hereinafter, operation principles and embodiments of the present disclosure are described with reference to the accompanying drawings.

[0053] Figure 1 is a diagram illustrating an electronic device according to an embodiment.

[0054] Reference Figure 1 , when the autonomous driving deep learning model 2 receives an image of the target 10 from the camera 3 and derives object information of the target 10, the electronic device 1 according to the embodiment may receive the object information of the target 10. The electronic device 1 may determine movement information of the target 10 based on the object information of the target 10. The electronic device 1 may determine the distance to the target 10 based on the determined movement information, calibration information, and physical information.

[0055] The autonomous driving deep learning model 2 may receive an image from the camera 3, and may derive object information of the target 10 from the received image. The autonomous driving deep learning model 2 may transmit the object information to the electronic device 1. The autonomous driving deep learning model 2 may include an artificial neural network trained by a computer vision-based learning algorithm. An embodiment of the learning algorithm may be, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0056] The artificial neural network may include multiple neural network layers. Each of the multiple neural network layers may have multiple weight values. Each of the multiple neural network layers may perform neural network operations based on the operation results of the previous layer and the operations in the multiple weight values. The multiple weight values ​​of the multiple neural network layers may be optimized by training the artificial intelligence model. For example, the multiple weight values ​​may be updated to reduce or minimize the loss value or cost value obtained by the artificial intelligence model during the training process.

[0057] The artificial neural network may include a deep neural network (DNN). The artificial neural network may include, for example but not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RMB), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q network, etc.

[0058] The autonomous driving deep learning model 2 may receive a road image including a target 10 (such as a vehicle) from an external device (such as a camera 3). The autonomous driving deep learning model 2 may include a bounding box recognition device 200 and a key point recognition device 210 to analyze the road image with the aid of an artificial neural network. The bounding box recognition device 200 may specify the position of the target 10 included in the road image, and may display the boundary of the specified target 10 as a predetermined type of box. The key point recognition device 210 may recognize key points (e.g., road markings) to calculate the speed of the vehicle based on objects fixed on the road.

[0059] The electronic device 1 according to the embodiment may use two different schemes to determine the distance to the target 10. The distances determined using the two different schemes may be expressed as a first physical quantity and a second physical quantity. The first physical quantity may refer to the distance to the target 10 determined based on the movement information of the target 10 and the calibration information of the camera 3. The second physical quantity may refer to the distance to the target 10 determined based on the movement information of the target 10 and the physical information of the target 10.

[0060] The electronic device 1 may determine whether the calibration information needs to be adjusted based on the error between the first physical quantity and the second physical quantity. When it is determined that the calibration information needs to be adjusted, the electronic device 1 may adjust the calibration information and may transmit vehicle control data for vehicle control to the vehicle controller 4 .

[0061] The electronic device 1 according to the embodiment may include a processor 100, a memory 110 and a communication device 120. It should be clear that the electronic device 1 may also include additional components necessary to implement the present disclosure. Figure 2 The components of the electronic device 1 according to the embodiment are described in more detail.

[0062] Figure 2 is a diagram showing a control block diagram of the electronic device 1 according to the embodiment.

[0063] Reference Figure 2 , the electronic device 1 may include: at least one processor 100, a memory 110, and a communication device 120. The electronic device 1 may communicate with the autonomous driving deep learning model 2 through the communication device 120 to determine whether to adjust the calibration information. The electronic device 1 may also communicate with the vehicle controller 4 through the communication device 120 to adjust the calibration information and transmit vehicle control data.

[0064] The electronic device 1 according to the embodiment may refer to all electronic devices, each of which includes a processor 100 and a memory 110, and may include, for example, a server device, a personal computer, a terminal, a portable phone, a smart phone, a handheld device, and a wearable device. The above electronic devices can be loaded into a vehicle for operation. The components of the electronic device 1 according to the embodiment are described in more detail below.

[0065] The memory 110 may store various information required to drive the electronic device 1. For example, the memory 110 may store an operating system and programs necessary to drive the electronic device 1. In addition or alternatively, the memory 110 may store data required to drive the electronic device 1. For example, the memory 110 may store calibration information of the camera 3 for determining the distance to the target 10 and physical information of the target 10.

[0066] The memory 110 may include a volatile memory, such as a state random access memory (SRAM) or a dynamic random access memory (DRAM), for temporarily storing data. In addition, the memory 110 may include a non-volatile memory, such as a read-only memory (ROM), an erasable programmable ROM (EPROM), or an electrically erasable programmable ROM (EEPROM), for permanently (e.g., for a long time) storing data.

[0067] The communication device 120 may include a wired communication device 122 and a wireless communication device 121 for communicating with the autonomous driving deep learning model 2 and the vehicle controller 4 .

[0068] The wired communication device 122 can access a wired communication network and can communicate with an external device through the wired communication network. For example, the wired communication device 122 can access a wired communication network through Ethernet (IEEE802.3 technical standard). As another embodiment, the wired communication device 122 can access a wired communication network through a controller area network (CAN) communication. The wired communication device 122 can send and receive data with an external device through a wired communication network.

[0069] The wireless communication device 121 may include at least one of a short-range communication module and / or a long-range communication module.

[0070] The short-range communication module may use a short-range communication method to communicate with an external device adjacent to the electronic device 1. The short-range communication module may use one of Bluetooth, Bluetooth Low Energy, Infrared Data Association (IrDA), ZigBee, Wireless Fidelity (Wi-Fi), Wi-Fi Direct, Ultra Wideband (UWB), Near Field Communication (NFC), etc.

[0071] The long-distance communication module may include a communication module for performing various types of long-distance communications, and may include a mobile communication device. The mobile communication device may send a wireless signal to at least one of a base station, an external terminal, or other external devices on a mobile communication network and receive a wireless signal from at least one of a base station, an external terminal, or other external devices on a mobile communication network. In addition, the long-distance communication module may communicate with an external device such as another electronic device 1 through a surrounding access point (AP). The AP may connect a local area network (LAN) connected to the electronic device 1 and a wide area network (WAN) connected to a communication server. Thus, the electronic device 1 may be connected to the communication server via the WAN to communicate with the communication server.

[0072] The processor 100 may output control signals to perform overall control of the electronic device 1. The processor 100 may include one central processing unit (CPU) or multiple CPUs and a graphics processing unit (GPU). The processor 100 may be implemented as an array of multiple logic gates and / or may be implemented as a combination of a general-purpose microprocessor 100 and a memory 110 storing computer-readable instructions (e.g., programs) executable by the microprocessor 100.

[0073] The processor 100 may control the above components to determine the first physical quantity and the second physical quantity to determine whether to adjust the calibration information, and determine that the calibration information needs to be adjusted when the error between the first physical quantity and the second physical quantity is greater than or equal to the reference value. In an embodiment, the processor 100 may determine the movement information of the target 10 based on the object information. The processor 100 may determine the first physical quantity to the target 10 based on the movement information and the calibration information. The processor 100 may determine the second physical quantity to the target 10 based on the movement information and the physical information. The processor 100 may compare the first physical quantity with the second physical quantity to determine whether to adjust the calibration information.

[0074] When the movement information indicates a dynamic object, the processor 100 may determine the distance to the lower end of the center of the bounding box included in the object information as the first physical quantity. The processor 100 may determine the second physical quantity as a ratio between the actual width information of the target 10 included in the physical information and the width information about the image of the target 10 included in the object information. The processor 100 may divide a value obtained by multiplying the focal length of the camera 3 by the actual width information by the width information about the image to determine the second physical quantity.

[0075] When the movement information indicates a static object, the processor 100 may determine the first physical quantity using the movement amount of the key point included in the object information. When the movement information indicates a static object, the processor 100 may receive the current speed through the communication device 120 and may determine the second physical quantity based on the current speed.

[0076] The processor 100 may adjust the calibration information based on determining that the difference between the first physical quantity and the second physical quantity is greater than or equal to a predetermined reference value. The processor 100 may determine an average value of the first physical quantity and an average value of the second physical quantity during a predetermined reference time. The processor 100 may adjust the calibration information based on determining that the difference between the average value of the first physical quantity and the average value of the second physical quantity is greater than or equal to a reference value.

[0077] Therefore, the electronic device 1 according to the embodiment may determine whether to adjust the calibration information based on the calibration information of the camera 3 and the physical information of the object 10 , thereby reducing the time and effort required to calibrate the separate camera 3 after vehicle manufacturing.

[0078] Hereinafter, a process of determining whether to adjust calibration information by means of the processor 100 included in the electronic device 1 according to an embodiment is described below in more detail.

[0079] Figure 3 is a diagram illustrating a bounding box used in an electronic device according to an embodiment.

[0080] refer to Figure 3 , the electronic device 1 according to the embodiment may receive object information of the target 10 recognized from the autonomous driving deep learning model 2. The object information of the target 10 may include information of the target 10 included in the image captured by the camera 3 through image processing analysis. For example, the object information of the target 10 may include type information of the target 10 (e.g., vehicle type information and partial information of the vehicle). The type information may be expressed as a category. The object information of the target 10 may also include location information of the target 10 (e.g., two-dimensional (2D) bounding box information).

[0081] The vehicle type information that may be included in the type information of the target 10 may include specification information of the vehicle. For example, the vehicle type information may include a sedan, a sports utility vehicle (SUV), a two-door car, a hatchback, a truck, a convertible, a limousine, etc. The partial information of the vehicle that may be included in the type information of the target 10 may include information indicating whether the identified target 10 is the entire vehicle or a part of the vehicle. For example, the partial information of the vehicle may include the entire vehicle, a bumper of the vehicle, a taillight of the vehicle, a license plate of the vehicle, etc.

[0082] The autonomous driving deep learning model 2 can obtain multiple points c through edge detection to obtain vehicle type information of the target 10. For example, Figure 3 As shown, the autonomous driving deep learning model 2 can obtain the positions of the ends of the two taillights of the vehicle through edge detection. The autonomous driving deep learning model 2 can also obtain the positions of the left and right ends of the bumper. In addition, the autonomous driving deep learning model 2 can obtain the positions of the ends of the two side mirrors through edge detection and can obtain the positions of the ends of the two fenders. Therefore, the autonomous driving deep learning model 2 can determine the vehicle type of the target 10 based on the relative distances of the positions of the multiple points c obtained.

[0083] The autonomous driving deep learning model 2 can generate a bounding box to obtain partial information of the vehicle in the type information of the target 10 and the position information of the target 10.

[0084] The bounding box may be generated during the image recognition process. The bounding box may include a box shape having a minimum size that includes an object. The bounding box may be generated in all or a portion of the target 10. Figure 3 As shown, a bounding box a including the entire vehicle and a bounding box b including a portion of the vehicle may be generated.

[0085] The electronic device 1 may receive type information (e.g., vehicle type information and partial information of the vehicle) of the target 10 and position information (e.g., 2D bounding box information) of the target 10 from the autonomous driving deep learning model 2. The electronic device 1 may determine a second physical quantity to the target 10 based on the received type information of the target 10 and the received position information of the target 10. The electronic device 1 may compare the first physical quantity determined based on the calibration information of the camera 3 with the second physical quantity to determine whether to adjust the calibration information.

[0086] In an embodiment, the object information of the target 10 may include movement category information for determining movement information of the target 10. The movement category information may include information on whether the object 10 whose image is analyzed is a mobile object or an object fixed to the road. For example, in the movement category information, vehicles, pedestrians, bicycles, etc. may be indicated as dynamic objects, and on the other hand, in the movement category information, road signs, road signs, buildings, etc. may be indicated as static objects.

[0087] Therefore, the electronic device 1 may determine the movement information of the target 10 based on the object information of the target 10. The electronic device 1 may determine whether to adjust the calibration information in different ways according to whether the target 10 is a dynamic object or a static object.

[0088] For example, when the target 10 is determined to be a dynamic object based on the movement information of the target 10, the processor 100 can use the calibration information to determine the distance to the lower end of the center of the bounding box as the first physical quantity. The processor 100 can use the ratio between the actual width information of the target 10 and the width information about the image to determine the second physical quantity.

[0089] On the other hand, when the target 10 is determined to be a static object based on the movement information of the target 10, the processor 100 may determine the first physical quantity using the movement amount of the key point. The processor 100 may determine the second physical quantity based on the current speed received through the communication device 120.

[0090] In this way, the electronic device 1 according to the embodiment can determine whether to adjust the calibration information in different ways according to whether the target 10 included in the image is a moving object or a stationary object. Therefore, since the electronic device 1 can determine whether to adjust the calibration information based on the stationary object, although there is no moving object in the image captured by the camera 3, the accuracy and reliability of determining whether to adjust the calibration information are improved.

[0091] The following reference Figures 4 to 7 An embodiment of determining the first physical quantity and the second physical quantity based on determining that the target 10 is a dynamic object is described in more detail.

[0092] Figure 4is a diagram illustrating reference points of a bounding box in an electronic device according to an embodiment.

[0093] Reference Figure 4 , the processor 100 of the electronic device 1 according to the embodiment may set a reference point for determining the distance to the target 10 based on determining that the target 10 is a dynamic object. The processor 100 may use calibration information to determine the first distance to the target 10. The calibration information may refer to a relationship value between the image coordinate system of the camera 3 and the world coordinate system.

[0094] In an embodiment, because the world coordinate system has (x, y, z) values ​​three-dimensionally and the image coordinate system of the camera 3 has (u, v) values ​​two-dimensionally, the processor 100 may adjust the z value to "0" so that the world coordinate system and the image coordinate system are consistent with each other. For example, the processor 100 may determine a point on the image at which the z value of the three-dimensional (3D) world coordinate system becomes "0". In an embodiment, the processor 100 may determine the point at which the z value of the 3D world coordinate system becomes "0" as point c, such as Figure 4 A point c in FIG. 1 connects from the lower end of the center of a bounding box b including the vehicle bumper to the lower end of the center of a bounding box a including the entire vehicle.

[0095] Therefore, the processor 100 may set Z to "0" based on the model of the pinhole camera 3. It is expressed as the following equation 1. The processor 100 may obtain (X, Y) values ​​to determine the distance to the point c (assumed to be the shortest distance to the target 10) as the first physical quantity.

[0096] [Equation 1]

[0097]

[0098] Therefore, the processor 100 can compare the first physical quantity with the reference Figures 5 to 7 The second physical quantity described is compared to determine whether to adjust the calibration information.

[0099] Figure 5 is a diagram illustrating width information about an image used in an electronic device according to an embodiment.

[0100] See also Figure 5 , the processor 100 of the electronic device 1 according to the embodiment may receive information about a bounding box a corresponding to a bumper of a vehicle from the autonomous driving deep learning model 2. The processor 100 may determine the width information b included in the information about the bounding box a corresponding to the bumper of the vehicle as the width information about the image of the vehicle.

[0101] The processor 100 may perform an arithmetic operation on the width information of the image about the vehicle and the physical information of the object stored in the memory 110 to obtain a second physical quantity which is a distance to the object.

[0102] exist Figure 5 In an embodiment, width information about the image of the vehicle is obtained. However, in another embodiment, height information about the image of the vehicle may be obtained. In other words, the processor 100 may determine the height information of the bounding box a included in the information about the bounding box a corresponding to the bumper of the vehicle as the height information about the image of the vehicle. The processor 100 may perform an arithmetic operation on the height information about the image of the vehicle and the physical information of the object stored in the memory 110 to obtain a second physical quantity that is a distance to the object.

[0103] Figure 6 is a diagram illustrating physical information of an object stored in a memory of an electronic device according to an embodiment.

[0104] The memory 110 of the electronic device 1 according to the embodiment may store information of the target 10 recognized by the autonomous driving deep learning model 2. For example, the physical information of the target 10 may include specification information about the vehicle. The specification information may include the total height a included in the height information of the vehicle, the thread b and the total width c included in the width information, the total length included in the length information, the wheelbase included in the wheelbase information, etc.

[0105] As described above, the processor 100 of the electronic device 1 according to the embodiment may calculate the first physical quantity based on the calibration information of the camera 3 , and may calculate the second physical quantity based on the physical information of the object 10 .

[0106] The processor 100 may calculate the specification information of the vehicle stored in the memory 110 and the specification information of the vehicle measured on the image according to the rate at which the second physical quantity is calculated. Figure 7 The process of determining the second physical information according to one embodiment is described in more detail.

[0107] Figure 7 is a diagram illustrating a method of calculating a distance to an object based on a focal length in an electronic device according to an embodiment.

[0108] Reference Figure 7 , the processor 100 may divide a value obtained by multiplying the actual vehicle width c previously stored in the memory 110 by the focal length of the camera 3 by ii) the vehicle width on the image to determine a second physical quantity.

[0109] In detail, the processor 100 may determine the second physical quantity based on Equation 2 below.

[0110] [Equation 2]

[0111] w (vehicle width on the image): W (actual vehicle width) = f (focal length on camera 3): X (second physical quantity)

[0112] As above reference Figure 5 As described above, the processor 100 may determine the width information included in the bounding box information corresponding to the bumper of the vehicle as w (width information on the image) b of the vehicle.

[0113] The processor 100 may obtain the width information of the vehicle from the specification information of the vehicle stored in the memory 110. The processor 100 may determine the width information of the vehicle as W (actual vehicle width) a. The processor 100 may substitute the focal length of the camera 3 stored in the memory 110 into the above equation 2 to calculate the second physical quantity.

[0114] In other words, the processor 100 may modify the above-mentioned Equation 2 to determine a value obtained by dividing a value obtained by multiplying W (actual vehicle width) by f (focal length of the camera 3 ) by w (vehicle width on the image) as the second physical quantity.

[0115] Therefore, the processor 100 may Figure 4 The first physical quantity determined in Figures 5 to 7 Since the position of the camera 3 or the installation angle of the camera 3 changes due to, for example, the distortion of the camera 3, when the difference between the first physical quantity and the second physical quantity is greater than a predetermined error, the processor 100 may determine that the calibration information needs to be adjusted.

[0116] Figure 8 A diagram for explaining key point information used in an electronic device according to an embodiment.

[0117] refer to Figure 8 When it is determined that the target 10 analyzed by the autonomous driving deep learning model 2 is a static object, the processor 100 may determine the first physical quantity based on the characteristics of the static object. For example, when it is determined that the target 10 analyzed by the autonomous driving deep learning model 2 is an object fixed to the road (e.g., a road sign, a road sign, and a building), the processor 100 may determine the object as a key point and may determine the movement amount of the host vehicle based on the key point.

[0118] In other words, Figure 8 As shown in , the processor 100 may display a sign a displayed on the road as a bounding box, and may determine the bounding box as a key point b.

[0119] The processor 100 may determine the first physical quantity and the second physical quantity using two different methods. For example, the processor 100 may obtain the moving speed of the electronic device 1 in the host vehicle based on the moving amount of the key point b having the same index, and may determine the moving speed of the host vehicle as the first physical quantity. In other words, because the key point has a fixed index on the image and moves according to the movement of the electronic device 1 in the host vehicle, the processor 100 may calculate the moving amount based on the calibration information, and may divide the moving amount by the moving time of the electronic device 1 to determine the moving speed of the electronic device 1 as the first physical quantity.

[0120] Furthermore, the processor 100 may determine the moving speed of the electronic device 1 in the host vehicle whose moving speed is received by the electronic device 1 through CAN communication as the second physical quantity.

[0121] In an embodiment, when a first physical quantity based on the calibration information and a second physical quantity calculated through CAN communication differ from each other by a value greater than a reference error, the processor 100 may determine that the calibration information needs to be adjusted.

[0122] As a result, when the difference between the first physical quantity and the second physical quantity is greater than or equal to the predetermined reference value, the processor 100 can adjust the calibration information. Therefore, when the camera 3 is distorted, although the calibration is not performed in a separate place, the processor 100 can adjust the calibration information to reduce the error in calculating the distance to the target 10.

[0123] Fig. 9 is a diagram illustrating an example of a calibration error in an electronic device according to an embodiment.

[0124] refer to Fig. 9 As described above, when it is determined that the processor 100 should adjust the calibration information, the processor 100 may adjust the calibration information to a candidate value stored in the memory 110 for calibrating the expected error.

[0125] For example, the memory 110 may store a rotation / motion transformation matrix for converting a world coordinate system into a coordinate system of the camera 3 based on the angle of distortion of the camera 3. Fig. 9 As shown in , the processor 100 may match the property values ​​of the corresponding rotation / motion transformation matrix with the first warp angle a, the second warp angle b, and the third warp angle c. The processor 100 may then adjust the calibration information based on the corresponding property values.

[0126] The processor 100 may determine to adjust the calibration information, may select a property value in which the error between the first physical quantity and the second physical quantity is the smallest, and may determine the selected property value as the calibration information. Thus, although the error between the first physical quantity and the second physical quantity is generated due to the distortion of the camera 3, the processor 100 may reduce the error based on the rotation / motion transformation matrix without a separate calibration task, thereby improving the accuracy of the calculated distance.

[0127] Fig.10 is a diagram illustrating a normal distribution graph used in an electronic device according to an embodiment.

[0128] Reference Fig.10 , the processor 100 may assume that the physical quantity determined in the process of determining the physical quantity in the various methods described above follows a normal distribution. Therefore, the electronic device 1 according to the embodiment may estimate the physical quantity based on the normal distribution, and may determine whether to adjust the calibration information based on determining that the estimated physical quantity is greater than or equal to a predetermined reference value, without determining whether to sensitively adjust the calibration information for each error in a single frame of an image captured by the camera 3. Therefore, when the electronic device 1 is used in a vehicle to determine whether to adjust the calibration information, the electronic device 1 may improve reliability.

[0129] In an embodiment, the processor 100 may determine an average value of the first physical quantity and an average value of the second physical quantity during a predetermined reference time. When the difference between the average value of the first physical quantity and the average value of the second physical quantity is greater than or equal to the reference value, the processor 100 may adjust the calibration information. On the other hand, when the difference between the average value of the first physical quantity and the average value of the second physical quantity is less than the reference value, the processor 100 may not adjust the calibration information.

[0130] In other words, if Fig.10 As shown in (a), when the average value of the first physical quantity and the average value of the second physical quantity are consistent with each other, or when the error between the average value of the first physical quantity and the average value of the second physical quantity is less than a predetermined reference error, the processor 100 may transmit the control data to the vehicle controller 4 without adjusting the calibration information.

[0131] On the other hand, Fig.10 As shown in (b), when the error between the average value of the first physical quantity and the average value of the second physical quantity is greater than or equal to a predetermined reference error, the processor 100 may first adjust the calibration information and may transmit control data generated based on the adjusted calibration information to the vehicle controller 4.

[0132] Therefore, although the camera 3 may be distorted due to certain factors, as calibration information is continuously adjusted, the distance to the external object may be provided with high reliability and safe autonomous driving may be achieved.

[0133] Fig.11 is a diagram showing a control flow chart of a method performed by an electronic device according to an embodiment. Fig.12 FIG. 2 is a diagram showing a method of executing a process of performing ... Fig.11 An illustration of a control flow graph of the following method.

[0134] Reference Fig.11 In operation 1100, the electronic device 1 may receive object information of a target 10 recognized by a deep learning model from the outside. The deep learning model may include an autonomous driving deep learning model 2. The entity that provides an image to the autonomous driving deep learning model 2 may include a camera 3.

[0135] At operation 1110, the processor 100 of the electronic device 1 may determine movement information of the recognized target 10 based on the object information of the target 10. At operation 1120, the processor 100 may determine whether the target 10 is a dynamic object or a static object based on the movement information of the target 10.

[0136] When it is determined based on the movement information that the target 10 is a dynamic object ("Yes" in operation 1120), the method may proceed to operation 1130. In operation 1130, the processor 100 may determine a distance from the electronic device 1 to the lower end of the center of the bounding box included in the object information of the target 10 as a first distance. In an embodiment, the first physical quantity and the second physical quantity may include a first distance, a second distance, a first speed, and a second speed.

[0137] In operation 1140, the processor 100 may determine a value obtained by dividing i) a value obtained by multiplying actual width information of the target 10 included in the physical information of the target 10 by the focal length of the camera 3 by ii) width information about the image of the target 10 included in the object information of the target 10 as the second distance.

[0138] On the other hand, when it is determined based on the movement information that the target 10 is a static object ("No" in operation 1120), the method may proceed to operation 1150. In operation 1150, the processor 100 may determine the first speed using the movement amount of the key point included in the object information of the target 10. In operation 1160, the processor 100 may determine the second speed based on the current speed received from the outside. In other words, the processor 100 may determine the current speed of the host vehicle as the second speed, wherein the current speed may be received from the controller of the host vehicle through CAN communication.

[0139] Reference Fig.12 In operation 1170, the processor 100 may determine a difference between the first physical quantity and the second physical quantity as an error. In operation 1180, the processor 100 may determine whether the error is greater than or equal to a predetermined reference value.

[0140] When the error between the first physical quantity and the second physical quantity is greater than or equal to the predetermined reference value ("Yes" in operation 1180), the method may proceed to operation 1190. In operation 1190, the processor 100 may adjust the calibration information. On the other hand, when the error between the first physical quantity and the second physical quantity is less than the predetermined reference value ("No" in operation 1180), the processor 100 may omit the adjustment of the calibration information.

[0141] In operation 1200 , the processor 100 may calculate the distance to the target 10 based on the calibration information and may transmit vehicle control data to the vehicle controller 4 .

[0142] Therefore, according to the embodiment, the electronic device 1 can use the calibration information to obtain the first physical quantity, can obtain the second physical quantity without using the calibration information, can calculate the physical quantity with the same value in different methods, and can periodically identify whether to adjust the calibration information, thereby improving the accuracy of the process of determining the distance to the target 10.

[0143] The disclosed embodiments may be implemented in the form of a storage medium storing instructions executable by a computer. The instructions may be stored in the form of program code and may generate a program module and may perform the operations of the disclosed embodiments when executed by the processor 100. The storage medium may be implemented as a computer-readable storage medium.

[0144] Computer-readable storage media may include different types of storage media that store instructions decoded by a computer. For example, a computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic tape, a disk, a flash memory, an optical data storage device, etc.

[0145] The computer-readable storage medium may be provided in the form of a non-transitory storage medium. In this document, the term "non-transitory storage medium" simply means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves). The term does not necessarily distinguish when data is semi-permanently stored in a storage medium and when data is temporarily stored in a storage medium. For example, a "non-transitory storage medium" may include a buffer that temporarily stores data.

[0146] According to an embodiment, the method according to various embodiments disclosed in the present disclosure may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or via an application store (e.g., PlayStore). TM) online distribution (e.g., download or upload), or directly distributed between two user devices (e.g., smart phones). If distributed online, at least a portion of the computer program product (e.g., a downloadable application) may be temporarily generated or at least temporarily stored in a machine-readable storage medium, such as the memory 110 of a manufacturer's server, a server of an application store, or a relay server.

[0147] According to an embodiment of the present invention, the electronic device can obtain 2D bounding box information and key point information in the image from the deep learning recognition result. When the vehicle is driving, the electronic device can determine and adjust whether the camera is distorted or whether the angle of the camera installation has an error, thereby reducing the time and effort used to physically change the distortion of the camera or the angle of the camera installation.

[0148] According to an embodiment of the present disclosure, the electronic device may have an effect of filtering sample data having high noise through a method for expressing the sample data as a normal distribution and comparing the sample data with reference data based on the normal distribution.

[0149] According to an embodiment of the present disclosure, the electronic device can calculate the distance to the target vehicle and the speed of the target vehicle to predict the movement of the target vehicle, and even when the camera is distorted while driving, the electronic device can automatically identify and compensate for the distortion of the camera, thereby providing accurate and continuous autonomous driving control information for the vehicle's autonomous driving system.

[0150] In the above, specific embodiments are shown and described. However, the present disclosure is not limited to the described embodiments. A person skilled in the art should be able to make various changes without departing from the gist of the technical concept of the present disclosure as set forth in the following claims.

Claims

1. An electronic device, comprising: a communication device configured to receive object information of a target captured by the camera and identified based on the deep learning model; A memory storing calibration information of the camera and physical information of the target; as well as a processor configured to determine movement information of the target based on the object information, determining a first physical quantity to the target based on the movement information and the calibration information, determining a second physical quantity to the target based on the movement information and the physical information, and The first physical quantity is compared with the second physical quantity to determine whether to adjust the calibration information.

2. The electronic device according to claim 1, wherein: The processor is configured to, when it is determined that the target is a dynamic object based on the movement information, determine a distance to a lower end of a center of a bounding box included in the object information as the first physical quantity.

3. The electronic device according to claim 1, wherein: The processor is configured to: when the target is determined to be a dynamic object based on the movement information, determine the second physical quantity as a ratio between actual width information of the target and width information of an image about the target, wherein the actual width information is included in the physical information and the width information is included in the object information.

4. The electronic device according to claim 3, wherein: The processor is configured to determine the second physical quantity by dividing i) a value obtained by multiplying the actual width information by a focal length of the camera by ii) the width information on the image.

5. The electronic device according to claim 1, wherein: The processor is configured to determine the first physical quantity using a movement amount of a key point included in the object information when the target is determined as a static object based on the movement information.

6. The electronic device according to claim 1, wherein: The processor is configured to, when the target is determined as a static object based on the movement information, receive a current speed through the communication device, and determine the current speed as the second physical quantity.

7. The electronic device according to claim 1, wherein: The processor is configured to adjust the calibration information in response to determining that a difference between the first physical quantity and the second physical quantity is greater than or equal to a predetermined reference value.

8. The electronic device according to claim 1, wherein: The processor is configured to: determining an average value of a plurality of the first physical quantities and an average value of a plurality of the second physical quantities during a predetermined reference time; and In response to determining that a difference between the average value of the plurality of first physical quantities and the average value of the plurality of second physical quantities is greater than or equal to a predetermined reference value, the calibration information is adjusted.

9. A method for controlling an electronic device, the method comprising: receiving, by a processor, object information of a target captured by a camera and identified based on a deep learning model; determining, by the processor, movement information of the target based on the object information; determining, by the processor, a first physical quantity of the target based on the movement information and calibration information of the target; determining, by the processor, a second physical quantity to the target based on the movement information and physical information of the target; as well as The first physical quantity is compared with the second physical quantity by the processor to determine whether to adjust the calibration information.

10. The method according to claim 9, wherein: Determining the first physical quantity includes: When the target is determined to be a dynamic object based on the movement information, a distance to a lower end of a center of a bounding box included in the object information is determined as the first physical quantity.

11. The method according to claim 9, wherein: Determining the second physical quantity includes: When the target is determined to be a dynamic object based on the movement information, the second physical quantity is determined as a ratio between actual width information of the target and width information of an image about the target, wherein the actual width information is included in the physical information and the width information is included in the object information.

12. The method according to claim 11, wherein: Determining the second physical quantity includes: The second physical quantity is determined by dividing i) a value obtained by multiplying the actual width information by the focal length of the camera by ii) the width information on the image.

13. The method according to claim 9, wherein: Determining the first physical quantity includes: When the target is determined as a static object based on the movement information, the first physical quantity is determined using a movement amount of a key point included in the object information.

14. The method according to claim 9, wherein: Determining the second physical quantity includes: When the target is determined as a static object based on the movement information, a current speed is received through a communication device, and the current speed is determined as the second physical quantity.

15. The method according to claim 9, wherein: Determining whether to adjust the calibration information includes: Based on determining that a difference between the first physical quantity and the second physical quantity is greater than or equal to a predetermined reference value, the calibration information is adjusted.

16. The method according to claim 9, wherein: Determining whether to adjust the calibration information includes: determining an average value of a plurality of the first physical quantities and an average value of a plurality of the second physical quantities during a predetermined reference time; and In response to determining that a difference between the average value of the plurality of first physical quantities and the average value of the plurality of second physical quantities is greater than or equal to a predetermined reference value, the calibration information is adjusted.

Images