Electronic device and method for identifying object in video

Abstract

To provide an electronic device and a method for identifying an object in a video.SOLUTION: An electronic device includes at least one camera, a gyro sensor, a memory, and at least one processor operatively connected to the at least one camera, the gyro sensor, and the memory, and the at least one processor can be configured to acquire video information using the at least one camera, acquire first rotation data related to the electronic device using the gyro sensor on the basis of acquiring the video information, acquire second rotation data using a specified estimation model, acquire a rotation value related to the video information on the basis of the first rotation data and the second rotation data, and identify at least one object related to the video information on the basis of the acquired rotation value.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an electronic device and method for identifying an object in a video.

Background Art

[0002] An electronic device can acquire a video. The electronic device can identify an object in the acquired video. However, when the video is acquired while the electronic device is rotated, the acquired video includes rotated objects, so the electronic device may not be able to identify the objects.

[0003] The above information can be provided as related art to facilitate understanding of the present disclosure. No claim or judgment is made as to whether any of the above is applicable as prior art related to the present disclosure.

Summary of the Invention

Means for Solving the Problems

[0004] According to one embodiment, an electronic device includes at least one camera, a gyro sensor, a memory, and at least one processor operatively connected to the at least one camera, the gyro sensor, and the memory. The at least one processor acquires video information using the at least one camera, and based on acquiring the video information, acquires first rotation data regarding the electronic device using the gyro sensor, acquires second rotation data using a specified estimation model, acquires a rotation value regarding the video information based on the first rotation data and the second rotation data, and can be configured to identify at least one object regarding the video information based on the acquired rotation value.

[0005] According to one embodiment, a method of an electronic device may include obtaining video information using at least one camera of the electronic device; obtaining first rotation data regarding the electronic device using a gyro sensor of the electronic device based on obtaining the video information; obtaining second rotation data using a specified estimation model; obtaining a rotation value regarding the video information based on the first rotation data and the second rotation data; and identifying at least one object regarding the video information based on the obtained rotation value.

Advantages of the Invention

[0006] According to one embodiment, an electronic device may identify an object based on a rotation value of a video. The electronic device may display a video with a bounding box overlaid on the identified object.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Mode for Carrying Out the Invention

[0008] Hereinafter, various embodiments of the present specification will be described with reference to the accompanying drawings. Regarding the description of the drawings, the same reference numerals can be used for similar or related components.

[0009] Fig. 1 shows an example of an electronic device for rotating an image according to an embodiment.

[0010] Referring to FIG. 1, the electronic device 101 can be configured based on various shapes. For example, the electronic device 101 can include the electronic device 101-1. The electronic device 101 can be configured like the electronic device 101-1. The electronic device 101-1 may correspond to a device (e.g., a black box) attached to a vehicle 110 (e.g., a motorcycle), or may be included in the device. According to an embodiment, the electronic device 101-1 can correspond to or be included in an ECU (electronic control unit) within the vehicle 110. The ECU may sometimes be referred to as an ECM (electronic control module). The electronic device 101-1 can be composed of independent hardware for the purpose of providing the functions according to the embodiments of the present invention in the vehicle 110. For example, the electronic device 101 can include the electronic device 101-2. The electronic device 101 can be configured like the electronic device 101-2. The electronic device 101-2 may be wearable by a user of the electronic device 101-2. The electronic device 101-2 may be used to provide information to the user while being worn by the user.

[0011] Hereinafter, the operation of the electronic device 101 including the electronic device 101-1 and the electronic device 101-2 will be described.

[0012] According to an embodiment, the electronic device 101 can include a camera disposed in one direction of the vehicle 110. FIG. 1 shows the electronic device 101 (or the camera included in the electronic device 101) disposed toward the front direction and / or the driving direction of the vehicle 110, but is not limited thereto. For example, the electronic device 101 may be disposed toward at least one of the rear direction or the side direction of the vehicle 110.

[0013] According to one embodiment, the electronic device 101 can acquire an image 121 of an external vehicle via a camera. The image 121 can rotate according to the rotation of the vehicle 110 or the electronic device 101. Hereinafter, for convenience of explanation, it will be described as if the rotation value of the electronic device 101 corresponds to the rotation value of the vehicle 110. This is only for convenience of explanation, and the rotation value of the electronic device 101 may be different from the rotation value of the vehicle 110.

[0014] For example, the vehicle 110 can rotate based on three axes (for example, the x-axis, the y-axis, and the z-axis). The order and direction of the three illustrated axes can be changed according to the embodiment. The rotation for each of the three axes can be defined as roll, pitch, and yaw. The rotation with respect to the x-axis can be defined as roll. The rotation with respect to the y-axis can be defined as pitch. The rotation with respect to the z-axis can be defined as yaw.

[0015] The x-axis can face the front of the vehicle 110. Since the camera of the electronic device 101 also faces the front of the vehicle 110, the image acquired via the camera of the electronic device 101 can rotate according to the rotation with respect to the x-axis.

[0016] As an example, the electronic device 101 can acquire an image 121. The image 121 can be rotated based on the rotation value of the vehicle 110 (or the electronic device 101) with respect to its roll. For example, the objects 131, 132, and 133 included in the image 121 can be rotated based on the rotation value of the vehicle 110 (or the electronic device 101) with respect to its roll. As the objects 131, 132, and 133 rotate, the electronic device 101 may not be able to recognize the objects 131, 132, and 133. For example, the bounding box for recognizing an object may be set large for the object (e.g., the objects 131, 132, and 133). As the bounding box is set large for the object, the error in object recognition or distance estimation for the actual object with respect to the object may increase.

[0017] Therefore, the electronic device 101 can rotate the image 121 based on the rotation value of the vehicle 110 (or the electronic device 101) with respect to its roll. The electronic device 101 can identify the image 122 by rotating the image 121 based on the rotation value of the vehicle 110 (or the electronic device 101) with respect to its roll. The image 122 can include the objects 141, 142, and 143. As the image 121 rotates, the object 131 can be changed to the object 141. As the image 121 rotates, the object 132 can be changed to the object 142. As the image 121 rotates, the object 133 can be changed to the object 143.

[0018] The image 122 may be an image acquired when the vehicle 110 (or the electronic device 101) is not rotating. Therefore, the electronic device 101 can smoothly recognize the objects 141, 142, and 143.

[0019] Hereinafter, as in the foregoing embodiments, technical features for rotating an image based on a rotation value related to the image (or video information) and recognizing an object through the rotated image will be described.

[0020] FIG. 2 shows an example of a block diagram of an electronic device according to an embodiment. The electronic device 101 in FIG. 2 can correspond to the electronic device 101 in FIG. 1.

[0021] Referring to FIG. 2, the electronic device 101 can include at least one of a processor 210, a camera 220, a sensor 230, or a memory 240. The processor 210, the camera 220, the sensor 230, and the memory 240 can be electronically and / or operably coupled with each other by an electronic component such as a communication bus. Hereinafter, that a device and / or a circuit is operably coupled means that a direct or indirect connection between the devices and / or circuits is established by wire or wirelessly so that the second circuit and / or the second device is controlled by the first circuit and / or the first device. Although shown in different blocks, the embodiments are not limited thereto. A part of the hardware in FIG. 2 may be included in a single integrated circuit such as a system on a chip (SoC). The type and / or number of the hardware included in the electronic device 101 is not limited to that shown in FIG. 2. For example, the electronic device 101 can include only a part of the hardware shown in FIG. 2.

[0022] According to one embodiment, the electronic device 101 can include hardware for processing data based on one or more instructions. The hardware for processing data can include a processor 210. For example, the hardware for processing data can include an ALU (arithmetic and logic unit), an FPU (floating point unit), an FPGA (field programmable gate array), a CPU (central processing unit), and / or an AP (application processor). The processor 210 may have a single-core processor structure, or may have a multi-core processor structure such as a dual core, a quad core, a hexa core, or an octa core.

[0023] According to one embodiment, the camera 220 of the electronic device 101 can include a lens assembly or an image sensor. The lens assembly can collect light emitted from a subject that is the object of image capture. The lens assembly can include one or more lenses. According to one embodiment, the camera 220 can include a plurality of lens assemblies. For example, in the camera 220, some of the plurality of lens assemblies may have the same lens attributes (for example, angle of view, focal length, autofocus, f number, or optical zoom), or at least one lens assembly may have one or more lens attributes different from the lens attributes of other lens assemblies. The said lens attributes may sometimes be called the intrinsic parameters of the camera 220. These intrinsic parameters can be stored in the memory 240 of the electronic device 101.

[0024] In one embodiment, the lens assembly can include a wide-angle lens or a telephoto lens. According to one embodiment, the flash can include one or more light-emitting diodes (e.g., RGB (red-green-blue) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. For example, the image sensor in the camera 220 can obtain an image corresponding to the subject by converting the light radiated or reflected from the subject and transmitted through the lens assembly into an electrical signal. According to one embodiment, the image sensor can include, for example, one selected image sensor among image sensors with different attributes such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same attribute, or a plurality of image sensors having other attributes. Each image sensor included in the image sensor can be realized using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0025] According to one embodiment, the sensor 230 of the electronic device 101 can be used to obtain various external information. The sensor 230 can include a gyro sensor 231 and an acceleration sensor 232. For example, the gyro sensor 231 can identify (or measure, sense) the angular velocity of the electronic device 101 in three directions of the x-axis, y-axis, and z-axis. For example, the acceleration sensor 232 can identify (or measure, detect) the acceleration of the electronic device 101 in three directions of the x-axis, y-axis, and z-axis.

[0026] According to one embodiment, the memory 240 of the electronic device 101 can include hardware components for storing data and / or instructions input to and / or output from the processor 210 of the electronic device 101. For example, the memory 240 can include a volatile memory such as a RAM (random-access memory), and / or a non-volatile memory such as a ROM (read-only memory). For example, the volatile memory can include at least one of a DRAM (dynamic RAM), an SRAM (static RAM), a Cache RAM, and a PSRAM (pseudo SRAM). For example, the non-volatile memory can include at least one of a PROM (programmable ROM), an EPROM (erasable PROM), an EEPROM (electrically erasable PROM), a flash memory, a hard disk, a compact disk, an SSD (solid state drive), and an eMMC (embedded multi-media card).

[0027] Although not shown, the electronic device 101 may further include various components. For example, the electronic device 101 can further include a display for displaying a user interface.

[0028] According to one embodiment, the electronic device 101 may include a neural network. For example, the neural network may include a mathematical model for the neural activity of an organism related to inference and / or recognition and / or hardware (e.g., CPU, GPU (graphic processing unit), and / or NPU (neural processing unit)), software, or any combination thereof for driving the mathematical model. The neural network may be based on a CNN (convolutional neural network) and / or an LSTM (long-short term memory).

[0029] For example, the electronic device 101 may include (or store) a specified estimation model represented by a plurality of parameters based on a neural network. The electronic device 101 may use the specified estimation model to obtain rotation data. For example, the electronic device 101 may set video information as input data of the specified estimation model. The electronic device 101 may obtain rotation data based on the output data of the specified estimation model.

[0030] According to one embodiment, the electronic device 101 may use a gyro sensor 231 and an acceleration sensor 232 to obtain a rotation value regarding video information acquired by a camera 220. The electronic device 101 may identify at least one object regarding the video information based on the obtained rotation value. Examples of the operation of the electronic device 101 for obtaining a rotation value regarding video information using the gyro sensor 231 and the acceleration sensor 232 will be described later with reference to FIGS. 3 to 5b.

[0031] According to an embodiment, the electronic device 101 can obtain a rotation value related to video information acquired using the camera 220 by using the gyro sensor 231 and a designated estimation model. The electronic device 101 can identify at least one object related to the video information based on the obtained rotation value. An example of the operation of the electronic device 101 for obtaining a rotation value related to video information by using the gyro sensor 231 and the designated estimation model will be described later with reference to FIGS. 5 to 10b.

[0032] FIG. 3 shows an exemplary flowchart for explaining the operation of an electronic device according to an embodiment. The electronic device 101 and / or the processor 210 in FIG. 2 can execute at least one of the operations in FIG. 3. In one embodiment, a computer-readable storage medium including a software application and / or instructions for causing the electronic device 101 and / or the processor 210 to execute the operations in FIG. 3 can be provided.

[0033] Referring to FIG. 3, in operation 310, the processor 310 can obtain video information. For example, the electronic device 101 can move together with the vehicle 110. While the electronic device 101 is moving with the vehicle 110, video information can be obtained by using the camera 220 facing the front of the vehicle 110. The first video corresponding to the video information may be in a rotated state based on the rotation value of the vehicle 110 (for example, the rotation value with respect to roll).

[0034] In operation 320, the processor 210 can obtain first rotation data by using the gyro sensor 231 and obtain second rotation data by using the acceleration sensor 232.

[0035] According to one embodiment, the processor 210 can obtain first rotation data using the gyro sensor 231. Based on the three axes shown in FIG. 1, the processor 210 can obtain a first angular velocity ((Ω_X)), a second angular velocity ((Ω_Y)), and a third angular velocity ((Ω_Z)) using the gyro sensor 231. The first angular velocity ((Ω_X)) can be the angular velocity with respect to the x-axis. The second angular velocity ((Ω_Y)) can be the angular velocity with respect to the y-axis. The third angular velocity ((Ω_Z)) can be the angular velocity with respect to the z-axis. Note that each angular velocity is assumed to be a vector.

[0036] The processor 210 can obtain first rotation data based on the first angular velocity ((Ω_X)). The first rotation data can include a rotation value related to the roll of the electronic device 101 (or the vehicle 110). The rotation value related to the roll of the electronic device 101 can be obtained based on the following formula.

Equation

[0037] According to an embodiment, the processor 210 can obtain second rotation data by using the acceleration sensor 232. Based on the three axes shown in FIG. 1, the processor 210 can obtain a first acceleration A_X, a second acceleration A_Y, and a third acceleration A_Z by using the acceleration sensor 232. The first acceleration A_X can be the acceleration with respect to the x-axis. The second acceleration A_Y can be the acceleration with respect to the y-axis. The third acceleration A_Z can be the acceleration with respect to the z-axis.

[0038] The processor 210 can obtain second rotation data based on the first acceleration A_X, the second acceleration A_Y, and the third acceleration A_Z. The second rotation data can include a rotation value related to the roll of the electronic device 101 (or the vehicle 110). The rotation value related to the roll of the electronic device 101 can be obtained based on the following mathematical formula.

Equation

[0039] In operation 330, the processor 210 can obtain a rotation value related to video information. For example, the processor 210 can obtain a rotation value related to video information based on the first rotation data and the second rotation data.

[0040] For example, the first rotation data can be highly accurate within a short time. The second rotation data can be highly accurate within a long time. Therefore, the processor 210 can obtain a rotation value related to video information by using a complementary filter (or a Kalman filter). Specific operations for obtaining a rotation value related to video information by using a complementary filter will be described later with reference to FIG. 4.

[0041] In operation 340, the processor 210 can identify at least one object related to the video information. For example, the processor 210 can identify at least one object related to the video information based on the obtained rotation value.

[0042] For example, the processor 210 can rotate a first video according to the video information based on the obtained rotation value. The processor 210 can obtain a second video by rotating the first video based on the obtained rotation value. The processor 210 can identify at least one bounding box for at least one object included in the second video. The processor 210 can display at least one bounding box by superimposing it on the second video. The processor 210 can obtain a third video by rotating (or reverse-rotating) the second video based on the obtained rotation value. The third video can be a video in which at least one bounding box is displayed by superimposing it on at least one object in the first video.

[0043] FIG. 4 shows an example of the operation of an electronic device according to an embodiment.

[0044] Referring to FIG. 4, the processor 210 of the electronic device 101 can acquire gyro data (e.g., the first angular velocity (Ω_X)) using the gyro sensor 231. The processor 210 can acquire acceleration data (e.g., the first acceleration A_X, the second acceleration A_Y, and the third acceleration A_Z) using the acceleration sensor 232. The processor 210 can acquire a rotation value regarding video information using a complementary filter based on the gyro data and the acceleration data.

[0045] For example, the processor 210 can acquire the first rotation data using the integrator 420 based on the gyro data. For example, the processor 210 can acquire the first rotation data according to Equation 1 based on the gyro data. The processor 210 can acquire the second rotation data based on the acceleration data. For example, the processor 210 can acquire the second rotation data according to Equation 2 based on the acceleration data.

[0046] The acceleration sensor 232 may generate a lot of noise in the high-frequency region (or short time). Therefore, the processor 210 can apply a low-pass filter 410 to the second rotation data to remove the noise. The gyro sensor 231 may have a drift phenomenon in the low-frequency region (or long time). Therefore, the processor 210 can apply a high-pass filter 430 to the first rotation data to remove the drift phenomenon.

[0047] The processor 210 can acquire a rotation value regarding video information by combining the first rotation data to which the high-pass filter 430 is applied and the second rotation data to which the low-pass filter 410 is applied. For example, the processor 210 can acquire a rotation value regarding video information based on Equation 3.

Equation

[0048] The larger the magnitude of α, the greater the influence of the first rotation data, and the smaller the magnitude of α, the greater the influence of the second rotation data. For example, the larger the magnitude of α, the closer the rotation value regarding the video information is obtained to the rotation value obtained using the gyro sensor 231, and the smaller the magnitude of α, the closer the rotation value regarding the video information can be obtained to the rotation value obtained using the acceleration sensor 232.

[0049] FIG. 5a shows an example of the value obtained via the acceleration sensor according to an embodiment.

[0050] FIG. 5b shows an example of the operation of the electronic device according to an embodiment.

[0051] Referring to FIG. 5a, the electronic device 101 can identify the acceleration of the electronic device 101 using the acceleration sensor 232. Hereinafter, for the sake of convenience of explanation, the operation of identifying the acceleration in the state where the electronic device 101 is included in the vehicle 110 will be described. For example, the acceleration of the electronic device 101 may correspond to the acceleration of the vehicle 110.

[0052] According to an embodiment, during the movement of the vehicle 110, a rotation with respect to roll can occur. When a rotation with respect to roll occurs, gravity 501 and centrifugal force 502 may occur.

[0053] The processor 210 can identify the acceleration due to the net force 503 of the gravitational force 501 and the inertial force 502. Although not shown, in the vehicle 110 (or the electronic device 101), various forces can act in addition to the gravitational force 501 and the inertial force 502. Therefore, it may be difficult for the processor 210 to accurately identify the acceleration of the electronic device 101 using the acceleration sensor 232.

[0054] Referring to FIG. 5b, the rotation value for the roll identified according to Equation 2 can be accurately identified when the electronic device 101 is not moving. However, when the acceleration of the electronic device 101 is identified while the electronic device 101 is moving, since the acceleration due to gravity and the acceleration due to movement are combined, the rotation value for the roll identified according to Equation 2 may not be accurate.

[0055] The image 590 can represent one frame of a video according to the video information acquired through the camera 220 of the electronic device 101. The processor 210 can acquire the video information while the electronic device 101 is rotating and moving rapidly.

[0056] The processor 210 can use the gyro sensor 231 to identify first rotation data and use the acceleration sensor 232 to identify second rotation data in order to identify the rotation value regarding the video information. The processor 210 can identify the rotation value regarding the video information based on the first rotation data and the second rotation data.

[0057] The processor 210 can use the gyro sensor 231 to identify first rotation data. The processor 210 can identify an object 530 representing a horizontal line in the image 590 based on the first rotation data.

[0058] The processor 210 can identify the second rotation data by using the acceleration sensor 232. The processor 210 can identify an object 510 representing the horizon in the image 590 based on the second rotation data.

[0059] The processor 210 can identify a rotation value related to the video information based on the first rotation data and the second rotation data. The processor 210 can identify an object 520 representing the horizon in the image 590 based on the rotation value related to the video information.

[0060] The object 530 can represent the horizon identified by using the gyro sensor 231. Since the gyro sensor 231 can identify an accurate rotation value in a short time, the object 530 may be closest to the actual horizon.

[0061] The object 510 can represent the horizon identified by using the acceleration sensor 232. As described in FIG. 5a, while the electronic device 101 (or the vehicle 110) is moving, due to the force acting on the electronic device 101 (or the vehicle 110), the processor 210 may not be able to identify an accurate rotation value. Therefore, the object 510 may not be able to correctly display the actual horizon.

[0062] The object 520 can represent the horizon identified based on the horizon identified by using the gyro sensor 231 and the horizon identified by using the acceleration sensor 232. Since the error between the horizon identified by using the acceleration sensor 232 and the actual horizon is large, the object 520 may also not be able to correctly display the actual horizon.

[0063] According to an embodiment, the processor 210 can identify a value for the roll of the electronic device 101 (or the vehicle 110) as follows in the following mathematical formula.

Equation

[0064] As described above, the processor 210 may have difficulty identifying the accurate rotation value of the video information using only the gyro sensor 231 and the acceleration sensor 232. Therefore, the processor 210 can identify the rotation value of the video information using the gyro sensor 231 and a specified estimation model. Hereinafter, the technical features for identifying the rotation value of the video information using the gyro sensor 231 and the specified estimation model will be described.

[0065] FIG. 6a shows an exemplary flowchart for explaining the operation of an electronic device according to an embodiment. The electronic device 101 and / or the processor 210 of FIG. 2 can execute at least one of the operations of FIG. 6a. In one embodiment, a computer-readable storage medium including a software application and / or instructions for causing the electronic device 101 and / or the processor 210 to execute the operations of FIG. 6a can be provided.

[0066] Referring to FIG. 6a, in operation 601, the processor 210 can acquire video information. For example, the electronic device 101 can move together with the vehicle 110. While the electronic device 101 is moving with the vehicle 110, video information can be acquired using the camera 220 facing the front of the vehicle 110. The first video corresponding to the video information may be in a rotated state based on the rotation value of the vehicle 110 (for example, the rotation value for roll).

[0067] In operation 602, the processor 210 can acquire first rotation data using the gyro sensor 231 and acquire second rotation data using a specified estimation model.

[0068] According to one embodiment, the processor 210 can acquire first rotation data using the gyro sensor 231. The operation of acquiring the first rotation data using the gyro sensor 231 may correspond to the operation of acquiring the first rotation data in operation 320.

[0069] According to one embodiment, the processor 210 can acquire second rotation data using a specified estimation model. For example, the processor 210 can use the specified estimation model to acquire second rotation data in various driving environments including not only flat roads but also inclined roads.

[0070] For example, the specified estimation model may be composed of a combination of a convolution layer and a fully connected (FC) layer. A specific example of the specified estimation model will be described later with reference to FIG. 6b.

[0071] For example, the processor 210 can set video information as input data of a specified estimation model. The processor 210 can obtain second rotation data based on the output data of the specified estimation model. The second rotation data can include a rotation value with respect to the roll of the electronic device 101 (or the vehicle 110). The processor 210 can identify a horizontal line in the video information. The processor 210 can identify a rotation value with respect to the roll of the electronic device 101 (or the vehicle 110) based on identifying the angle by which the horizontal line has rotated in the video information.

[0072] According to one embodiment, the processor 210 can learn a specified estimation model based on a first learning video and a second learning video obtained by rotating the first learning video according to a specified rotation value. A specific example of the operation of learning the specified estimation model will be described later with reference to FIG. 7.

[0073] In operation 603, the processor 210 can obtain a rotation value related to the video information. For example, the processor 210 can obtain a rotation value related to the video information based on the first rotation data and the second rotation data.

[0074] According to one embodiment, since the second rotation data is obtained using a specified estimation model, there may be a case where the rotation value with respect to the roll of the electronic device 101 (or the vehicle 110) cannot be correctly estimated by a scene (or frame, special pattern) not used for learning. Therefore, the processor 210 can obtain a rotation value related to the video information using a complementary filter (or a Kalman filter). Specific operations for obtaining a rotation value related to the video information using a complementary filter will be described later with reference to FIG. 9.

[0075] In operation 604, the processor 210 can identify at least one object related to the video information. For example, the processor 210 can identify at least one object related to the video information based on the acquired rotation value.

[0076] For example, the processor 210 can rotate a first video according to the video information based on the acquired rotation value. The processor 210 can obtain a second video by rotating the first video based on the acquired rotation value. The processor 210 can identify at least one bounding box for at least one object included in the second video. The processor 210 can display at least one bounding box superimposed on the second video. The processor 210 can obtain a third video by rotating (or reverse rotating) the second video based on the acquired rotation value. The third video may be a video in which at least one bounding box is displayed superimposed on at least one object in the first video.

[0077] The operation of the electronic device 101 (or the processor 210) for identifying at least one object related to the video information will be described later with reference to FIGS. 10a and 10b.

[0078] FIG. 6b shows an example of a specified estimation model according to an embodiment.

[0079] Referring to FIG. 6b, the specified estimation model 600 may be composed of a combination of a convolutional layer 610 and an FC layer 620. For example, the convolutional layer 610 can maintain spatial information regarding the image 651 and can be used to extract features. The FC layer 620 can be used to output data within a specified range.

[0080] For example, the number of convolutional layers 610 can be configured with 13 layers. For example, the number of FC layers 620 can be configured with 3 layers. For example, the specified estimation model 600 can be configured based on the VGG16 model. The specified estimation model 600 may include at least a part of the VGG16 model. For example, the specified estimation model 600 may be a model in which an FC layer for regression is added to the final classification stage of the VGG16 model.

[0081] The image 651 can be set as the input data of the convolutional layer 610. For example, the image 651 may be one frame of a video corresponding to video information acquired using the camera 220. The image 651 can be composed of a three-dimensional vector of 222×224×3. Based on the output data of the convolutional layer 610, a three-dimensional vector of 7×7×512 can be identified. The three-dimensional vector of 7×7×512 can be set as the input data of the FC layer 620. Through the FC layer 620, a rotation value 652 can be output within a specified range (for example, 0 or more and 25 or less).

[0082] Therefore, the processor 210 can learn the specified estimation model 600 using driving videos (or driving images) for various environments. The processor 210 can use the specified estimation model 600 to acquire second rotation data. The second rotation data may include a rotation value with respect to the roll of the electronic device 101 (or the vehicle 110).

[0083] FIG. 7 shows an example of learning data of a specified estimation model according to an embodiment.

[0084] Referring to FIG. 7, the processor 210 can learn a specified estimation model. For example, in order to learn the specified estimation model, it may be necessary to obtain images and rotation values acquired during the movement of the electronic device 101 (or the vehicle 110). Considerable resources may be required to obtain the images and rotation values acquired during the movement of the electronic device 101 (or the vehicle 110). Therefore, the processor 210 can identify the image 710, the image 720, and the image 730 in order to augment the learning data.

[0085] The processor 210 may be an image obtained by rotating the image 710 by 0 degrees. The processor 210 can obtain the image 710 in a state where the rotation value with respect to roll is 0 degrees while the vehicle 110 is running. The processor 210 can identify the image 710 as a ground truth. The processor 210 can learn the specified estimation model 600 based on the image 710.

[0086] The processor 210 can obtain the image 720 by rotating the image 710 according to the specified rotation value. The processor 210 can crop the region 721 including the object within the image 720. The processor 210 can obtain (or identify) the image 730 by cropping the region 721 including the object within the image 720. The processor 210 can identify the image 730 as a ground truth for the specified rotation value. For example, the processor 210 can remove the outer region of the image by cropping the region 721 including the object within the image 720. According to an embodiment, the processor 210 can obtain the image 730 by cropping a specified region (for example, the central region) within the image 720 regardless of the object.

[0087] For example, the processor 210 can set a specified rotation value as any value within a specified range. The processor 210 can obtain a ground truth image (e.g., image 730) by obtaining an image rotated according to various rotation values.

[0088] Therefore, the processor 210 can obtain the image 730 based on rotating the image 710 obtained while the rotation value of the electronic device 101 is 0 degrees based on the specified rotation value. The processor 210 can identify the image 730 as the ground truth of the specified rotation value.

[0089] In the foregoing embodiments, for the sake of convenience of explanation, an example of learning the specified estimation model 600 using images (e.g., image 710, image 720, and image 730) is shown, but it is not limited thereto. The processor 210 can identify a first learning image as a ground truth. The processor 210 can obtain a second learning image based on rotating the first learning image according to a specified rotation value. The processor 210 can identify the second learning image as the ground truth of the specified rotation value.

[0090] FIG. 8a shows an example of the operation of an electronic device according to an embodiment.

[0091] FIG. 8b shows an example of the operation of an electronic device according to an embodiment.

[0092] Referring to FIG. 8a, the processor 210 can obtain video information using the camera 220. The processor 210 can identify a first video corresponding to the video information.

[0093] The processor 210 can obtain (or identify) the rotation value using the specified estimation model 600. The processor 210 can set the video information (or the first video) as the input data of the specified estimation model 600. The processor 210 can obtain (or identify) the rotation value based on the output data of the specified estimation model 600.

[0094] In operation 802, the processor 210 can rotate the first video based on the obtained rotation value. The processor 210 can obtain (or identify) the second video based on rotating the first video. The second video can be a video with the rotation removed from the first video. The processor 210 can set the second video as the input data of the object detection model 803. The processor 210 can obtain the third video based on the output data of the object detection model 803. The processor 210 can identify at least one object in the second video via the object detection model 803. The processor 210 can obtain the third video with at least one bounding box related to the at least one object overlaid on the second video.

[0095] Referring to FIG. 8b, the processor 210 can obtain video information using the camera 220. The processor 210 can identify the first video corresponding to the video information.

[0096] The processor 210 can set video information (or the first video) as input data for the object detection model 811. The processor 210 can obtain a second video based on the output data of the object detection model 811. The processor 210 can identify at least one object in the first video via the object detection model 811. The processor 210 can obtain a second video in which at least one bounding box related to at least one object is displayed superimposed on the first video.

[0097] The processor 210 can obtain (or identify) a rotation value using the specified estimation model 600. The processor 210 can set video information (or the first video) as input data for the specified estimation model 600. The processor 210 can obtain (or identify) a rotation value based on the output data of the specified estimation model 600.

[0098] In operation 813, based on the obtained rotation value, the processor 210 can rotate at least one bounding box. For example, since at least one object in the second video is in a rotated state, at least one bounding box can be configured to be larger than at least one object. The processor 210 can obtain a third video in which at least one bounding box is corrected and displayed by rotating at least one bounding box.

[0099] Referring to FIG. 8a again, after rotating the first video, the processor 210 can identify at least one object using the object identification model 803. Since the unrotated video (or image) is set as the input data for the object identification model 803, a general object identification model 803 can be used. However, latency may occur due to the rotation of the first video.

[0100] Referring to FIG. 8b again, the processor 210 can identify at least one object using the object identification model 811 without rotating the first video. The processor 210 can rotate at least one bounding box identified based on the object identification model 811. Since the first video is not rotated, at least one object can be identified with low latency. However, the processor 210 may need to configure the rotated video as learning data for the object identification model 811.

[0101] FIG. 9 shows an example of the operation of an electronic device according to an embodiment.

[0102] Referring to FIG. 9, the processor 210 of the electronic device 101 can acquire gyro data (e.g., the first angular velocity (Ω_X)) using the gyro sensor 231. The processor 210 can acquire video information using the camera 220. The processor 210 can acquire a rotation value related to the video information using the complementary filter 900 based on the gyro data and the video information.

[0103] For example, the processor 210 can acquire first rotation data using the integrator 920 based on the gyro data. For example, the processor 210 can acquire first rotation data according to Equation 1 based on the gyro data. The processor 210 can acquire second rotation data using the specified estimation model 600 based on the video information. For example, the processor 210 can set the video information as the input data of the specified estimation model 600. The processor 210 can acquire second rotation data based on the output data of the specified estimation model 600.

[0104] Since the second rotation data is obtained using the specified estimation model 600, the rotation value with respect to the roll of the electronic device 101 (or the vehicle 110) may not be correctly estimated by a scene (or frame, special pattern) not used for learning. Therefore, the processor 210 can obtain a rotation value regarding the video information using a complementary filter (or Kalman filter) based on the first rotation data and the second rotation data. For example, the processor 210 can obtain a rotation value regarding the video information based on Equation 5.

Equation

[0105] For example, within a short period of time, the rotation value regarding the video information may be close to the rotation value included in the first rotation data obtained using the gyro sensor 231. For example, within a long period of time, the rotation value regarding the video information may be close to the rotation value included in the second rotation data obtained using the specified estimation model 600.

[0106] FIG. 10a shows an example of the operation of an electronic device according to an embodiment.

[0107] FIG. 10b shows an example of the operation of an electronic device according to an embodiment.

[0108] Referring to FIG. 10a, the processor 210 can acquire video information using the camera 220. The processor 210 can identify a first video corresponding to the video information.

[0109] The processor 210 can identify at least one object using the specified estimation model 600. The processor 210 can set the first video (or video information) as the input data of the specified estimation model 600. The processor 210 can acquire second rotation data based on the output data of the specified estimation model 600. The processor 210 can acquire first rotation data using gyro data. The processor 210 can set the first rotation data and the second rotation data as the input values of the complementary filter 900. The processor 210 can acquire a rotation value regarding the video information using the complementary filter 900 described in FIG. 9. For example, the processor 210 can acquire a rotation value regarding the video information using the number 5.

[0110] In operation 1001, the processor 210 can rotate the first video based on the acquired rotation value. The processor 210 can acquire (or identify) a second video based on rotating the first video. The second video can be a video with rotation removed from the first video. The processor 210 can set the second video as the input data of the object detection model 1003. The processor 210 can acquire a third video based on the output data of the object detection model 1003. The processor 210 can identify at least one object in the second video via the object detection model 1003. The processor 210 can acquire a third video in which at least one bounding box regarding at least one object is superimposed and displayed on the second video.

[0111] Referring to FIG. 10b, the processor 210 can acquire video information using the camera 220. The processor 210 can identify a first video according to the video information.

[0112] The processor 210 can set the video information (or the first video) as input data of the object detection model 1011. The processor 210 can acquire a second video based on the output data of the object detection model 1011. The processor 210 can identify at least one object in the first video via the object detection model 1011. The processor 210 can acquire a second video in which at least one bounding box related to at least one object is superimposed and displayed on the first video.

[0113] The processor 210 can acquire (or identify) second rotation data using the specified estimation model 600. The processor 210 can set the video information (or the first video) as input data of the specified estimation model 600. The processor 210 can acquire (or identify) a rotation value based on the output data of the specified estimation model 600. The processor 210 can acquire first rotation data using gyro data. The processor 210 can set the first rotation data and the second rotation data as input values of the complementary filter 900. The processor 210 can acquire a rotation value related to the video information using the complementary filter 900 described in FIG. 9. For example, the processor 210 can acquire a rotation value related to the video information using Equation 5.

[0114] In operation 1013, based on the acquired rotation value, the processor 210 can rotate at least one bounding box. For example, since at least one object in the second video is in a rotated state, at least one bounding box can be configured to be larger than at least one object. The processor 210 can obtain a third video in which at least one bounding box is corrected and displayed by rotating at least one bounding box.

[0115] Referring to FIG. 10a again, after rotating the first video, the processor 210 can identify at least one object using the object identification model 1003. Since the non-rotated video (or image) is set as the input data of the object identification model 1003, a general object identification model 1003 can be used. However, latency may occur according to the rotation of the first video.

[0116] Referring to FIG. 10b again, the processor 210 can identify at least one object using the object identification model 1011 without rotating the first video. The processor 210 can rotate at least one bounding box identified based on the object identification model 1011. Since the first video is not rotated, at least one object can be identified with low latency. However, the processor 210 may need to configure the rotated video as the learning data of the object identification model 1011.

[0117] FIG. 11 shows an example of the operation of an electronic device according to an embodiment.

[0118] Referring to FIG. 11, the processor 210 can acquire video information using the camera 220. The image 1100 can be acquired based on one frame of the first video corresponding to the video information. The processor 210 can display the image 1100 using the display included in the electronic device 101 or the display of an external device.

[0119] The processor 210 can acquire first rotation data using the gyro sensor 231. The object 1110 can be displayed based on the rotation value identified by the first rotation data. The object 1110 can represent the horizon identified based on the first rotation data within the image 1100.

[0120] The processor 210 can acquire second rotation data using the specified estimation model 600. The object 1120 can be displayed based on the rotation value identified by the second rotation data. The object 1120 can represent the horizon identified based on the second rotation data within the image 1100.

[0121] The processor 210 can identify the rotation value of the video information based on the first rotation data and the second rotation data. The processor 210 can display the object 1130 within the image 1100 based on the rotation value of the video information. For example, the processor 210 can acquire the rotation value regarding the video information using the complementary filter 900 based on the first rotation data and the second rotation data. The object 1130 can be displayed based on the rotation value regarding the video information. The object 1130 can represent the horizon identified based on the rotation value regarding the video information.

[0122] Based on the first rotation data and the second rotation data, the processor 210 can obtain a rotation value regarding the video information. Based on the rotation value regarding the video information, the processor 210 can display an object 1130 representing a horizon close to the actual horizon within the image 1110.

[0123] According to one embodiment, the processor 210 can identify at least one object within the image 1100 based on the rotation value regarding the video information. The processor 210 can rotate the image 1100 based on the rotation value. After rotating the image 1100, the processor 210 can identify at least one object using an object detection model.

[0124] The processor 210 can display at least one bounding box regarding the identified at least one object within the image 1100. The at least one bounding box may be in a rotated state based on the rotation value regarding the video information. For example, the processor 210 can identify an object 1101, an object 1102, an object 1103, and an object 1104 within the image 1100. The processor 210 can display a bounding box 1111 corresponding to the shape of the object 1101 within the image 1100. The processor 210 can display a bounding box 1112 corresponding to the shape of the object 1102 within the image 1100. The processor 210 can display a bounding box 1113 corresponding to the shape of the object 1103 within the image 1100. The processor 210 can display a bounding box 1114 corresponding to the shape of the object 1104 within the image 1100.

[0125] For example, the bounding box 1111, the bounding box 1112, the bounding box 1113, and the bounding box 1114 may be in a rotated state based on the rotation value regarding the video information.

[0126] According to one embodiment, the processor 210 can display text 1121 indicating information about the object 1101 within the image 1100 together with the bounding box 1111. The information about the object 1101 can represent the distance between the actual object corresponding to the object 1101 and the electronic device 101. For example, the text 1121 can represent the distance in the first direction (e.g., left or right direction) (i.e., -5 m), the distance in the second direction (e.g., forward direction) (i.e., 3 m), and the shortest distance (i.e., 5.83 m).

[0127] The processor 210 can display text 1122 indicating information about the object 1102 within the image 1100 together with the bounding box 1112. The information about the object 1102 can represent the distance between the actual object corresponding to the object 1102 and the electronic device 101. For example, the text 1121 can represent the distance in the first direction (e.g., left or right direction) (i.e., -3 m), the distance in the second direction (e.g., forward direction) (i.e., 5 m), and the shortest distance (i.e., 5.83 m).

[0128] The processor 210 can display text 1123 indicating information about the object 1103 within the image 1100 together with the bounding box 1113. The information about the object 1103 can represent the distance between the actual object corresponding to the object 1103 and the electronic device 101. For example, the text 1123 can represent the distance in the first direction (e.g., left or right direction) (i.e., 2 m), the distance in the second direction (e.g., forward direction) (i.e., 4 m), and the shortest distance (i.e., 4.47 m).

[0129] The processor 210 can display text 1124 indicating information about the object 1104 within the image 1100 together with the bounding box 1114. The information about the object 1104 can represent the distance between the actual object corresponding to the object 1104 and the electronic device 101. For example, the text 1124 can represent the distance with respect to the first direction (e.g., left or right direction) (i.e., 3 m), the distance with respect to the second direction (e.g., forward direction) (i.e., 4 m), and the shortest distance (i.e., 5 m).

[0130] As described above, the processor 210 can configure the bounding box according to the size of the object and display it within the image (or video). The processor 210 can rotate and display the bounding box according to the rotation value of the video.

[0131] FIG. 12 shows an example of a block diagram showing an autonomous driving system of a vehicle according to an embodiment.

[0132] The autonomous driving system 1200 of a vehicle according to FIG. 12 can be a deep learning network including a sensor 1203, an image pre-processor 1205, a deep learning network 1207, an artificial intelligence (AI) processor 1209, a vehicle control module 1211, a network interface 1213, and a communication unit 1215. In various embodiments, each element can be connected via various interfaces. For example, the sensor data sensed and output by the sensor 1203 can be fed to the image pre-processor 1205. The sensor data processed by the image pre-processor 1205 may be fed to the deep learning network 1207 run by the AI processor 1209. The output of the deep learning network 1207 run by the AI processor 1209 may be fed to the vehicle control module 1211. The intermediate result of the deep learning network 1207 run by the AI processor 1209 can be fed to the AI processor 1209. In various embodiments, the network interface 1213 transmits autonomous driving route information and / or autonomous driving control commands for the autonomous driving of the vehicle to the internal block configuration by executing communication with in-vehicle electronic devices. In one embodiment, the network interface 1213 can be used to transmit the sensor data acquired via the sensor 1203 to an external server. In some embodiments, the autonomous driving control system 1200 may appropriately include additional or fewer components. For example, in some embodiments, the image pre-processor 1205 can be an optional component. In another example, a post-processing component (not shown) may be included in the autonomous driving control system 1200 to perform post-processing on the output of the deep learning network 1207 before the output is provided to the vehicle control module 1211.

[0133] In some embodiments, sensor 1203 can include one or more sensors. In various embodiments, sensor 1203 can be attached at different positions of the vehicle. Sensor 1203 may be directed in one or more different directions. For example, sensor 1203 can be attached to the front, sides, rear, and / or roof of the vehicle so as to face directions such as forward-facing, rear-facing, side-facing, etc. In some embodiments, sensor 1203 may be an image sensor such as high dynamic range cameras. In some embodiments, sensor 1203 includes non-visual sensors. In some embodiments, sensor 1203 includes RADAR (Radio Detection and Ranging), LiDAR (Light Detection And Ranging), and / or ultrasonic sensors in addition to the image sensor. In some embodiments, sensor 1203 is not mounted on a vehicle having vehicle control module 1211. For example, sensor 1203 may be included as part of a deep learning system for capturing sensor data and may be attached to the environment or road and / or attached to surrounding vehicles.

[0134] In some embodiments, the Image pre-processor 1205 can be used to pre-process the sensor data of the sensor 1203. For example, the Image pre-processor 1205 can be used to split the sensor data into one or more components, and / or to post-process one or more components, for pre-processing the sensor data. In some embodiments, the Image pre-processor 1205 may be a graphics processing unit (GPU), a central processing unit (CPU), an image signal processor, or a specialized image processor. In various embodiments, the Image pre-processor 1205 can be a tone-mapper processor for processing high dynamic range data. In some embodiments, the Image pre-processor 1205 may be a component of the AI processor 1209.

[0135] In some embodiments, the Deep learning network 1207 can be a deep learning network for implementing control instructions for controlling an autonomous vehicle. For example, the Deep learning network 1207 can be an artificial neural network such as a convolutional neural network (CNN) trained using sensor data, and the output of the Deep learning network 1207 is provided to the vehicle control module 1211.

[0136] In some embodiments, the artificial intelligence (AI) processor 1209 can be a hardware processor for running the deep learning network 1207. In some embodiments, the AI processor 1209 is a specialized AI processor for performing inference on sensor data via a convolutional neural network (CNN). In some embodiments, the AI processor 1209 may be optimized for the bit depth of the sensor data. In some embodiments, the AI processor 1209 may be optimized for deep learning operations such as neural network operations including convolution, inner product, vector, and / or matrix operations. In some embodiments, the AI processor 1209 may be implemented via a plurality of graphics processing units (GPUs) that can effectively perform parallel processing.

[0137] In various embodiments, the AI processor 1209 can be coupled via an input / output interface to a memory configured to provide an AI processor having instructions that, while the AI processor 1209 is executing, perform deep learning analysis on sensor data received from the sensor 1203 and determine the results of machine learning used to at least partially autonomously operate the vehicle. In some embodiments, a Vehicle Control Module 1211 can process instructions for vehicle control output from the artificial intelligence (AI) processor 1209 and use the output of the AI processor 1209 to convert (translate) it into instructions for controlling the various modules of the vehicle. In some embodiments, the vehicle control module 1211 is used to control a vehicle for autonomous driving. In some embodiments, the vehicle control module 1211 can adjust the steering and / or speed of the vehicle. For example, the vehicle control module 1211 can be used to control the driving of the vehicle such as deceleration, acceleration, steering, lane change, lane keeping, etc. In some embodiments, the vehicle control module 1211 can generate control signals for controlling vehicle lighting such as brake lights, turn signals, headlights, etc. In some embodiments, the vehicle control module 1211 can be used to control vehicle audio-related systems such as the vehicle's sound system, the vehicle's audio warnings, the vehicle's microphone system, the vehicle's horn system, etc.

[0138] In some embodiments, the vehicle control module 1211 can be used to control notification systems, including a warning system for informing passengers and / or drivers of driving events such as access to an intended destination or a potential collision. In some embodiments, the vehicle control module 1211 may be used to adjust sensors such as the vehicle's sensor 1203. For example, the vehicle control module 1211 can modify the orientation of the sensor 1203, change the output resolution and / or format type of the sensor 1203, increase or decrease the capture rate, adjust the dynamic range, and adjust the focus of the camera. Additionally, the vehicle control module 1211 can turn sensors on / off individually or in a group.

[0139] In some embodiments, the vehicle control module 1211 can be used to change parameters of the image pre-processor 1205 in ways such as changing the frequency range of a filter, adjusting edge detection parameters for feature and / or object detection, and adjusting channels and bit depth. In various embodiments, the vehicle control module 1211 may be used to control the autonomous driving of the vehicle and / or the driver assistance functions of the vehicle.

[0140] In some embodiments, the network interface 1213 can be responsible for the internal interface between the block configuration of the autonomous driving control system 1200 and the communication unit 1215. Specifically, the network interface 1213 can be a communication interface for receiving and / or transmitting data including voice data. In various embodiments, the network interface 1213 can connect a voice call via the communication unit 1215, receive and / or transmit text messages, transmit sensor data, update the software of the vehicle in the autonomous driving system, or connect to an external server to update the software of the autonomous driving system of the vehicle.

[0141] In various embodiments, the communication unit 1215 can include various wireless interfaces of cellular or WiFi type. For example, the network interface 1213 can be used to receive updates for the operating parameters and / or instructions for the sensor 1203, the image pre-processor 1205, the deep learning network 1207, the AI processor 1209, and the vehicle control module 1211 from an external server connected via the communication unit 1215. For example, the machine learning model of the deep learning network 1207 can be updated using the communication unit 1215. According to yet another example, the communication unit 1215 may be used to update the operating parameters of the image pre-processor 1205 such as image processing parameters and / or the firmware of the sensor 1203.

[0142] In another embodiment, the communication unit 1215 can be used to activate communication for emergency services and emergency contact in the event of an accident or a near-accident event. For example, in a collision event, the communication unit 1215 can be used to call for emergency services for assistance and can be used to notify external emergency services of the details of the collision and the position of the vehicle. In various embodiments, the communication unit 1215 can update or obtain the expected arrival time and / or the location of the destination.

[0143] According to one embodiment, the autonomous driving system 1200 shown in FIG. 12 may be composed of electronic devices of a vehicle. According to one embodiment, when an autonomous driving cancellation event occurs from a user during autonomous driving of the autonomous driving system 1200, the AI processor 1209 of the autonomous driving system 1200 can be controlled to input autonomous driving cancellation event-related information into the training set data of the deep learning network, thereby controlling the autonomous driving software of the vehicle to learn.

[0144] FIGS. 13 and 14 show an example of a block diagram showing an autonomous driving mobile body according to one embodiment. Referring to FIG. 13, the autonomous driving mobile body 1300 according to the present embodiment can include a control device 1400, sensing modules 1304a, 1304b, 1304c, 1304d, an engine 1306, and a user interface 1308.

[0145] The autonomous driving mobile body 1300 can be provided with an autonomous driving mode or a manual mode. As an example, it may be switched from the manual mode to the autonomous driving mode or from the autonomous driving mode to the manual mode according to a user input received via the user interface 1308.

[0146] When the mobile body 1300 is operating in the autonomous driving mode, the autonomous driving mobile body 1300 can operate under the control of the control device 1400.

[0147] In this embodiment, the control device 1400 can include a controller 1420 including a memory 1422 and a processor 1424, a sensor 1410, a communication device 1430, and an object detection device 1440.

[0148] Here, the object detection device 1440 can execute all or part of the functions of the distance measurement device (for example, the electronic device 101).

[0149] That is, in this embodiment, the object detection device 1440 is a device for detecting an object located outside the moving body 1300, and the object detection device 1440 can detect an object located outside the moving body 1300 and generate object information according to the detection result.

[0150] The object information can include information on the presence or absence of an object, the position information of the object, the distance information between the moving body and the object, and the relative speed information between the moving body and the object.

[0151] The object can include various objects located outside the moving body 1300, such as a lane, other vehicles, pedestrians, traffic signals, light, roads, structures, speed bumps, terrain objects, animals, etc. Here, the traffic signal can be a concept including traffic lights, traffic signs, patterns or texts drawn on the road surface. And the light can be light generated from a lamp provided on another vehicle, light generated by a street lamp, or sunlight.

[0152] And the structure can be an object located around the road and fixed to the ground. For example, the structure can include street lamps, street trees, buildings, utility poles, signal lights, bridges. The terrain object can include mountains, hills, etc.

[0153] Such an object detection device 1440 can include a camera module. The controller 1420 can extract object information from an external image captured by the camera module and cause the controller 1420 to process the information.

[0154] Also, the object detection device 1440 may further include an imaging device for recognizing the external environment. In addition to LIDAR, RADAR, GPS devices, odometry devices, and other computer vision devices, ultrasonic sensors, infrared sensors, etc. can be used, and these devices can operate selectively or simultaneously as needed to enable more accurate sensing.

[0155] On the other hand, the distance measurement device according to an embodiment of the present invention can calculate the distance between the autonomous driving vehicle 1300 and an object, and based on the calculated distance in cooperation with the control device 1400 of the autonomous driving vehicle 1300, control the operation of the vehicle.

[0156] As an example, when there is a possibility of a collision according to the distance between the autonomous driving vehicle 1300 and an object, the autonomous driving vehicle 1300 can control the brakes to reduce or stop the speed. As another example, when the object is a moving object, the autonomous driving vehicle 1300 can control the driving speed of the autonomous driving vehicle 1300 to maintain a distance of a predetermined distance or more from the object.

[0157] The distance measurement device according to an embodiment of the present invention can be constituted by a module within the control device 1400 of the autonomous driving vehicle 1300. That is, the memory 1422 and the processor 1424 of the control device 1400 can realize the collision prevention method according to the present invention in software.

[0158] In addition, the sensor 1410 can be connected to the internal / external environment sensing modules 1304a, 1304b, 1304c, and 1304d of the moving body to obtain various sensing information. Here, the sensor 1410 can include an attitude sensor (e.g., a yaw sensor, a roll sensor, a pitch sensor), a collision sensor, a wheel sensor, a speed sensor, an inclination sensor, a weight sensing sensor, a heading sensor, a gyro sensor, a position module, a moving body forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor by handle rotation, a moving body internal temperature sensor, a moving body internal humidity sensor, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, and the like.

[0159] Thereby, the sensor 1410 can obtain sensing signals for moving body attitude information, moving body collision information, moving body direction information, moving body position information (GPS information), moving body angle information, moving body speed information, moving body acceleration information, moving body inclination information, moving body forward / backward information, battery information, fuel information, tire information, moving body lamp information, moving body internal temperature information, moving body internal humidity information, steering wheel rotation angle, moving body external illuminance, pressure applied to the accelerator pedal, pressure applied to the brake pedal, and the like.

[0160] In addition, the sensor 1410 may further include an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake air temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), and the like.

[0161] In this way, the sensor 1410 can generate moving body state information based on the sensing data.

[0162] The wireless communication device 1430 is configured to perform wireless communication between the autonomous driving mobile body 1300. For example, it enables communication between the user's mobile phone, or other wireless communication devices 1430, other mobile bodies, a central device (traffic control device), a server, etc. and the autonomous driving mobile body 1300. The wireless communication device 1430 can transmit and receive wireless signals according to the connected wireless protocol. The wireless communication protocol includes Wi-Fi, Bluetooth, LTE (Long-Term Evolution), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), GSM (Global Systems for Mobile Communications), and the communication protocol is not limited to this.

[0163] Also, in this embodiment, the autonomous driving mobile body 1300 can also realize communication between mobile bodies via the wireless communication device 1430. That is, the wireless communication device 1430 can communicate with other mobile bodies on the road and other mobile bodies through vehicle-to-vehicle (V2V) communication. The autonomous driving mobile body 1300 can transmit and receive information such as driving warnings and traffic information through vehicle-to-vehicle communication, and can also request information from other mobile bodies or receive requests. For example, the wireless communication device 1430 can perform V2V communication with a dedicated short-range communication (DSRC) device or a Cellular-V2V (C-V2V) device. In addition to communication between vehicles, communication between a vehicle and other objects (for example, an electronic device carried by a pedestrian, etc.) (V2X, Vehicle to Everything communication) can also be realized via the wireless communication device 1430.

[0164] In addition, the wireless communication device 1430 can obtain, via a non-terrestrial network instead of a terrestrial network, information generated from various mobilities including infrastructure (such as traffic lights, CCTV, RSU, eNode B, etc.) located on the road or other autonomous driving / non-autonomous driving vehicles as information for executing the autonomous driving of the autonomous mobile body 1300.

[0165] For example, the wireless communication device 1430 can perform wireless communication via a non-terrestrial network dedicated antenna mounted on the autonomous mobile body 1300 with a low earth orbit (LEO) satellite system, a medium earth orbit (MEO) satellite system, a geostationary orbit (GEO) satellite system, a high altitude platform (HAP) system, etc., that constitute the non-terrestrial network.

[0166] For example, the wireless communication device 1430 can currently TH perform wireless communication with various platforms constituting the NTN according to a wireless connection standard compliant with the 5G NR NTN (5 Generation New Radio Non-Terrestrial Network) standard being discussed in 3GPP, etc., and is not limited thereto.

[0167] In this embodiment, the controller 1420 can select a platform that can appropriately perform NTN communication in consideration of various information such as the position, current time, and available power of the autonomous mobile body 1300, and control the wireless communication device 1430 to execute wireless communication with the selected platform.

[0168] In this embodiment, the controller 1420 is a unit that controls the overall operation of each unit within the mobile body 1300. It may be configured during manufacturing by the manufacturer of the mobile body, or may be further configured after manufacturing to execute the function of autonomous driving. Alternatively, through the upgrade of the controller 1420 configured during manufacturing, it can include a configuration for executing continuous additional functions. Such a controller 1420 may be referred to as an ECU (Electronic Control Unit).

[0169] The controller 1420 can collect various data from the connected sensors 1410, object detection device 1440, communication device 1430, etc., and based on the collected data, transmit a control signal to the sensors 1410, engine 1306, user interface 1308, communication device 1430, object detection device 1440 including other components within the mobile body. Also, although not shown, a control signal can also be transmitted to an acceleration device, braking system, steering device, or navigation device related to the driving of the mobile body.

[0170] In this embodiment, the controller 1420 can control the engine 1306. For example, when the autonomous driving mobile body 1300 detects the speed limit of the road during driving, it can control the engine 1306 so that the driving speed does not exceed the speed limit, or control the engine 1306 to accelerate the driving speed of the autonomous driving mobile body 1300 within a range not exceeding the speed limit.

[0171] Also, while the autonomous mobile body 1300 is in motion, when the autonomous mobile body 1300 approaches a lane or leaves a lane, the controller 1420 determines whether such approach and departure from the lane are in accordance with normal driving conditions or other driving conditions, and controls the engine 1306 to control the driving of the mobile body according to the determination result. Specifically, the autonomous mobile body 1300 can detect lanes formed on both sides of the road on which the mobile body is traveling. In this case, the controller 1420 determines whether the autonomous mobile body 1300 is approaching a lane or leaving a lane. When it is determined that the autonomous mobile body 1300 is approaching a lane or leaving a lane, it can be determined whether such driving is due to accurate driving conditions or other driving conditions. Here, as an example of normal driving conditions, it can be a situation where a lane change of the mobile body is necessary. And as an example of other driving conditions, it can be a situation where a lane change of the mobile body is not necessary. When the controller 1420 determines that the autonomous mobile body 1300 is approaching a lane or leaving a lane in a situation where a lane change of the mobile body is not required, the controller 1420 can control the driving of the autonomous mobile body 1300 so that the autonomous mobile body 1300 does not leave the lane and travels normally with that mobile body.

[0172] When there is another mobile body or an obstacle in front of the mobile body, the engine 1306 or the braking system can be controlled to decelerate the traveling mobile body, and in addition to the speed, the trajectory, the operation route, and the steering angle can be controlled. Alternatively, the controller 1420 may generate necessary control signals according to recognition information of other external environments such as the traveling lane and the traveling signal of the mobile body, and control the driving of the mobile body.

[0173] In addition to generating its own control signals, the controller 1420 can also control the driving of the mobile body by communicating with surrounding mobile bodies or a central server and sending commands for controlling peripheral devices via the received information.

[0174] In addition, when the position of the camera module 1450 is changed or the viewing angle is changed, the controller 1420 may generate a control signal to control the execution of calibration of the camera module 1450 in order to prevent the accurate recognition of the moving object or the lane according to the present embodiment from being difficult. Therefore, in the present embodiment, the controller 1420 can continuously maintain the normal mounting position, direction, viewing angle, etc. of the camera module 1450 even if the mounting position of the camera module 1450 is changed due to vibrations, impacts, etc. generated during the movement of the autonomous driving vehicle 1300 by generating a calibration control signal in the camera module 1450. The controller 1420 can generate a control signal to execute calibration of the camera module 1450 when the initial mounting position, direction, viewing angle information of the camera module 1450 stored in advance and the initial mounting position, direction, viewing angle information, etc. of the camera module 1450 measured during the running of the autonomous driving vehicle 1300 change by more than a critical value.

[0175] In the present embodiment, the controller 1420 may include a memory 1422 and a processor 1424. The processor 1424 can execute the software stored in the memory 1422 according to the control signal of the controller 1420. Specifically, the controller 1420 stores data and instructions for performing the lane detection method according to the present invention in the memory 1422, and these instructions can be executed by the processor 1424 to implement one or more methods disclosed in this specification.

[0176] At this time, the memory 1422 may be stored in a non-volatile recording medium executable by the processor 1424. The memory 1422 can store software and data via appropriate internal and external devices. The memory 1422 can be composed of a RAM (random access memory), a ROM (read only memory), a hard disk, and a memory 1422 device connected to a dongle.

[0177] Memory 1422 can store at least an operating system (OS), user applications, and executable instructions. Memory 1422 can also store application data and array data structures.

[0178] Processor 1424 is a microprocessor or a suitable electronic processor and can be a controller, microcontroller, or state machine.

[0179] Processor 1424 can be implemented with a combination of computing devices, and the computing devices can be composed of a digital signal processor, a microprocessor, or a suitable combination thereof.

[0180] On the other hand, the autonomous mobile 1300 may further include a user interface 1308 for user input to the above-described control device 1400. The user interface 1308 can enable the user to input information through appropriate interactions. For example, it can be implemented with a touch screen, keypad, operation buttons, etc. The user interface 1308 transmits an input or command to the controller 1420, and the controller 1420 can execute the control operation of the mobile body in response to the input or command.

[0181] Also, the user interface 1308 can be a device external to the autonomous mobile 1300 and communicate with the autonomous mobile 1300 via the wireless communication device 1430. For example, the user interface 1308 can be made interoperable with a mobile phone, tablet, or other computer device.

[0182] Furthermore, in this embodiment, the autonomous driving vehicle 1300 has been described as including the engine 1306, but it is also possible to include other types of propulsion systems. For example, the vehicle can be driven by electric energy or through a hydrogen energy or a hybrid system combining them. Therefore, the controller 1420 includes a propulsion mechanism by the propulsion system of the autonomous driving vehicle 1300, and can provide a control signal according to this to the configuration of each propulsion mechanism.

[0183] Hereinafter, with reference to FIG. 14, the detailed configuration of the control device 1400 according to this embodiment will be described in more detail.

[0184] The control device 1400 includes a processor 1424. The processor 1424 can be a general-purpose single-chip or multi-chip microprocessor, a dedicated microprocessor, a microcontroller, a programmable gate array, or the like. The processor may be called a central processing unit (CPU). Also, in this embodiment, the processor 1424 can be used in combination with a plurality of processors.

[0185] The control device 1400 also includes a memory 1422. The memory 1422 can be any electronic component capable of storing electronic information. The memory 1422 can also include a combination of memories 1422 in addition to a single memory.

[0186] Data and instructions 1422a for executing the distance measurement method of the distance measurement device according to the present invention may be stored in the memory 1422. When the processor 1424 executes the instructions 1422a, all or part of the instructions 1422a and the data 1422b necessary for the execution of the instructions may be loaded (1424a, 1424b) onto the processor 1424.

[0187] The control device 1400 may include a transmitter 1430a, a receiver 1430b, or a transceiver 1430c to enable signal transmission and reception. One or more antennas 1432a, 1432b may be electrically connected to the transmitter 1430a, the receiver 1430b, or each transceiver 1430c, and may further include an antenna.

[0188] The control device 1400 may include a digital signal processor (DSP) 1470. Through the DSP 1470, a mobile body can be enabled to quickly process digital signals.

[0189] The control device 1400 may include a communication interface 1480. The communication interface 1480 may include one or more ports and / or communication modules for connecting other devices to the control device 1400. The communication interface 1480 can enable interaction between the user and the control device 1400.

[0190] The various components of the control device 1400 may be connected together by one or more buses 1490, and the buses 1490 may also include a power bus, a control signal bus, a status signal bus, a data bus, etc. According to the control of the processor 1424, the components can transmit mutual information via the buses 1490 and execute desired functions.

[0191] On the other hand, in various embodiments, the control device 1400 may be associated with a gateway for communication with a security cloud.

[0192] FIG. 15 shows an example of a gateway related to a user device according to various embodiments.

[0193] Referring to FIG. 15, the control device 1400 may be associated with a gateway 1505 for providing information obtained from at least one of the components 1501 to 1504 of the vehicle 1500 to the security cloud 1506. For example, the gateway 1505 may be included within the control device 1400. In another example, the gateway 1505 may be composed of separate devices within the vehicle 1500 that are distinct from the control device 1400. The gateway 1505 communicably connects the networks within the vehicle 1500 that are secured by the software management cloud 1509, the security cloud 1506, and the in-vehicle security software 1510, which have different networks.

[0194] For example, the component 1501 can be a sensor. For example, this sensor can be used to obtain information regarding at least one of the state of the vehicle 1500 or the state around the vehicle 1500. For example, the component 1501 can include the sensor 1410.

[0195] For example, the component 1502 may be an ECU (electronic control unit). For example, the ECU can be used for engine control, transmission control, airbag control, and tire air pressure management.

[0196] For example, component 1503 can be an instrument cluster. For example, an instrument cluster may mean a panel arranged in front of the driver's seat on the dashboard. For example, the instrument cluster may be configured to display information necessary for driving to the driver (or passenger). For example, the instrument cluster can be used to display at least one of a visual element for indicating the number of revolutions per minute (RPM) of the engine, a visual element for indicating the speed of the vehicle 1500, a visual element for indicating the remaining fuel level, a visual element for indicating the gear state, or a visual element for indicating information obtained via component 1501.

[0197] For example, component 1504 can be a telematics device. For example, this telematics device can mean a device that combines wireless communication technology and GPS (global positioning system) technology to provide various mobile communication services such as location information and safe driving within the vehicle 1500. For example, the telematics device can be used to connect the driver, the cloud (e.g., security cloud 1506), and / or the surrounding environment to the vehicle 1500. For example, the telematics device may be configured to support high bandwidth and low latency for 5G NR standard technologies (e.g., 5G NR's V2X technology, 5G NR's Non-Terrestrial Network (NTN) technology). For example, the telematics device may be configured to support the autonomous driving of the vehicle 1500.

[0198] For example, the gateway 1505 can be used to connect the network within the vehicle 1500 to the software management cloud 1509 and the security cloud 1506, which are networks outside the vehicle. For example, the software management cloud 1509 can be used to update or manage at least one software necessary for the driving and management of the vehicle 1500. For example, the software management cloud 1509 can interact with the in-car security software 1510 installed in the vehicle. For example, the in-car security software 1510 can be used to provide security functions within the vehicle 1500. For example, the in-car security software 1510 can encrypt data transmitted and received via the in-vehicle network using an encryption key obtained from an external authorized server for the encryption of the in-vehicle network. In various embodiments, the encryption key used by the in-car security software 1510 can be generated corresponding to the identification information of the vehicle (vehicle number plate, vehicle VIN (vehicle identification number)) or information uniquely assigned to each user (such as user identification information).

[0199] In various embodiments, the gateway 1505 can transmit data encrypted by the in-vehicle security software 1510 to the software management cloud 1509 and / or the security cloud 1506 based on the encryption key. The software management cloud 1509 and / or the security cloud 1506 can decrypt the data encrypted by the encryption key of the in-vehicle security software 1510 using a decryption key, thereby identifying which vehicle or which user the data is received from. For example, since this decryption key is a unique key corresponding to the encryption key, the software management cloud 1509 and / or the security cloud 1506 can identify the sender of the data (e.g., the vehicle or the user) based on the data decrypted via the decryption key.

[0200] For example, the gateway 1505 is configured to support the in-vehicle security software 1510 and may be associated with the control device 2100. For example, the gateway 1505 may be associated with the control device 1400 to support the connection between the client device 1507 connected to the security cloud 1506 and the control device 1400. In another example, the gateway 1505 may be associated with the control device 1400 to support the connection between the third-party cloud 1508 connected to the security cloud 1506 and the control device 1400. However, it is not limited thereto.

[0201] In various embodiments, the gateway 1505 can be used to connect the software management cloud 1509 for managing the operating software of the vehicle 1500 and the vehicle 1500. For example, the software management cloud 1509 can monitor whether an update of the operating software of the vehicle 1500 is requested, and based on monitoring that an update of the operating software of the vehicle 1500 is requested, provide data for updating the operating software of the vehicle 1500 via the gateway 1505. As another example, the software management cloud 1509 can receive a user request for updating the operating software of the vehicle 1500 from the vehicle 1500 via the gateway 1505, and based on this reception, provide data for updating the operating software of the vehicle 1500. However, it is not limited thereto.

[0202] FIG. 16 is a block diagram of an electronic device according to an embodiment. The electronic device 101 in FIG. 16 may include the electronic device 101 in FIGS. 1-11.

[0203] Referring to FIG. 16, the processor 1610 of the electronic device 101 can execute computations related to the neural network 1630 stored in the memory 1620. The processor 1610 may include at least one of a CPU (central processing unit), a GPU (graphics processing unit), or an NPU (neural processing unit). The NPU may be implemented as a separate chip from the CPU, or may be integrated into a chip such as the CPU in the form of an SoC (system on a chip). The NPU integrated into the CPU may be called a neural core and / or an AI (artificial intelligence) accelerator.

[0204] Referring to FIG. 16, the processor 1610 can identify the neural network 1630 stored in the memory 1620. The neural network 1630 can include a combination of an input layer 1632, one or more hidden layers 1634 (or intermediate layers), and an output layer 1636. The aforementioned layers (e.g., the input layer 1632, one or more hidden layers 1634, and the output layer 1636) can include a plurality of nodes. The number of hidden layers 1634 may vary depending on the embodiment, and a neural network 1630 including a plurality of hidden layers 1634 may sometimes be referred to as a deep neural network. The operation of training a deep neural network may sometimes be referred to as deep learning.

[0205] In one embodiment, when the neural network 1630 has the structure of a feed forward neural network, a first node included in a particular layer can be connected to all of the second nodes included in another layer previous to the particular layer. In the memory 1620, the parameters stored for the neural network 1630 can include the weights assigned to the connections between the second nodes and the first node. In the neural network 1630 having the structure of a feed forward neural network, the value of the first node can correspond to the weighted sum of the values assigned to the second nodes, based on the weights assigned to the connections connecting the second nodes and the first node.

[0206] In one embodiment, when the neural network 1630 has a structure of a convolutional neural network, a first node included in a specific layer can correspond to a weighted sum for a part of a second node included in another layer previous to the specific layer. Some of the second nodes corresponding to the first node may be identified by a filter corresponding to the specific layer. In the memory 1620, the parameters stored for the neural network 1630 can include the weights representing the filter. The filter can include one or more nodes among the second nodes that are used to calculate the weighted sum of the first node, and the weights corresponding to each of the one or more nodes.

[0207] According to one embodiment, the processor 1610 of the electronic device 101 can perform training on the neural network 1630 using the learning data set 1640 stored in the memory 1620. Based on the learning data set 1640, the processor 1610 can adjust one or more parameters stored in the memory 1620 for the neural network 1630.

[0208] According to one embodiment, the processor 1610 of the electronic device 101 can perform object detection, object recognition, and / or object classification using the neural network 1630 trained based on the learning dataset 1640. The processor 1610 can input an image (or video) acquired via the camera 1650 into the input layer 1632 of the neural network 1630. Based on the input layer 1632 into which the image is input, the processor 1610 can sequentially obtain the values of the nodes of the layers included in the neural network 1630 and obtain a set of values of the nodes of the output layer 1636 (for example, output data). The output data can be used as a result of estimating the information included in the image using the neural network 1630. The embodiment is not limited thereto, and the processor 1610 can input an image (or video) acquired from an external electronic device connected to the electronic device 101 via the communication circuit 1660 into the neural network 1630.

[0209] In one embodiment, the neural network 1630 trained to process an image can be used to identify a region corresponding to a subject in the image (object detection) and / or to identify the class of the subject represented in the image (object recognition and / or object classification). For example, the electronic device 101 can use the neural network 1630 to segment a region corresponding to the subject in the image based on a rectangular shape such as a bounding box. For example, the electronic device 101 can use the neural network 1630 to identify at least one class that matches the subject among a plurality of specified classes.

[0210] According to one embodiment, the electronic device includes at least one camera, a gyro sensor, a memory, and at least one processor operatively connected to the at least one camera, the gyro sensor, and the memory. The at least one processor acquires video information using the at least one camera, and based on acquiring the video information, acquires first rotation data regarding the electronic device using the gyro sensor, acquires second rotation data using a specified estimation model, acquires a rotation value regarding the video information based on the first rotation data and the second rotation data, and can be set to identify at least one object regarding the video information based on the acquired rotation value.

[0211] According to one embodiment, the at least one processor can be further set to acquire the rotation value regarding the video information using a complementary filter based on the first rotation data and the second rotation data.

[0212] According to one embodiment, the at least one processor can be further set to set the video information as input data of the specified estimation model and acquire the second rotation data based on output data of the specified estimation model.

[0213] According to one embodiment, the at least one processor can be further set to learn the specified estimation model based on a first learning video and a second learning video obtained by rotating the first learning video according to a specified rotation value.

[0214] According to one embodiment, the at least one processor is configured to obtain a second video obtained by rotating a first video corresponding to the video information based on the obtained rotation value, identify the at least one object using an object detection model within the second video, obtain a third video in which at least one bounding box related to the at least one object is superimposed and displayed on the second video, and further obtain, based on the third video and the obtained rotation value, a fourth video in which the at least one bounding box is superimposed and displayed within the first video.

[0215] According to one embodiment, the at least one processor is configured to identify a first video corresponding to the video information, identify the at least one object within the first video using an object detection model, obtain a second video in which at least one bounding box related to the at least one object is superimposed and displayed on the first video, and further obtain, based on the obtained rotation value, a third video in which the at least one bounding box within the second video is corrected and displayed.

[0216] According to one embodiment, the first rotation data may include a rotation value identified based on an axis corresponding to the direction in which the at least one camera is facing.

[0217] According to one embodiment, the electronic device further includes a display, and the at least one processor is configured to superimpose and display at least one bounding box for identifying the at least one object on the video while the first video corresponding to the video information is being displayed via the display, and further superimpose and display information related to the at least one object on the first video based on the at least one bounding box.

[0218] According to one embodiment, the electronic device is disposed in a vehicle, and the video information can be acquired using the at least one camera disposed facing forward of the vehicle.

[0219] According to one embodiment, the electronic device is configured to be wearable on a part of the user's body, and the video information can be acquired using the at least one camera disposed facing forward of the user.

[0220] According to one embodiment, a method of an electronic device includes: obtaining video information using at least one camera of the electronic device; obtaining first rotation data regarding the electronic device using a gyro sensor of the electronic device based on obtaining the video information, and obtaining second rotation data using a specified estimation model; obtaining a rotation value regarding the video information based on the first rotation data and the second rotation data; and identifying at least one object regarding the video information based on the obtained rotation value.

[0221] According to one embodiment, the method may further include obtaining the rotation value regarding the video information using a complementary filter based on the first rotation data and the second rotation data.

[0222] According to one embodiment, the method may further include setting the video information as input data of the specified estimation model; and obtaining the second rotation data based on output data of the specified estimation model.

[0223] According to one embodiment, the method may further include training the specified estimation model based on a first training video and a second training video obtained by rotating the first training video according to a specified rotation value.

[0224] According to one embodiment, the method may further include: obtaining a second video obtained by rotating a first video corresponding to the video information based on the obtained rotation value; identifying the at least one object using an object detection model within the second video; obtaining a third video in which at least one bounding box related to the at least one object is superimposed and displayed on the second video; and obtaining a fourth video in which the at least one bounding box is superimposed and displayed within the first video based on the third video and the obtained rotation value.

[0225] According to one embodiment, the method may further include: identifying a first video corresponding to the video information; identifying the at least one object within the first video using an object detection model; obtaining a second video in which at least one bounding box related to the at least one object is superimposed and displayed on the first video; and obtaining a third video in which the at least one bounding box within the second video is corrected and displayed based on the obtained rotation value.

[0226] According to one embodiment, the first rotation data may include a rotation value identified based on an axis corresponding to a direction in which the at least one camera is facing.

[0227] According to one embodiment, the method may further include: while the first video corresponding to the video information is being displayed via a display of the electronic device, superimposing and displaying at least one bounding box for identifying the at least one object on the video; and based on the at least one bounding box, superimposing and displaying information related to the at least one object on the first video.

[0228] According to one embodiment, the electronic device is disposed in a vehicle, and the video information can be acquired using the at least one camera disposed facing forward of the vehicle.

[0229] According to one embodiment, the electronic device is configured to be wearable on a part of the user's body, and the video information can be acquired using the at least one camera disposed facing forward of the user.

[0230] According to one embodiment, an electronic device (e.g., electronic device 101) can identify a rotation value of the electronic device (or vehicle) using at least one of a gyro sensor and a specified estimation model without a high-performance IMU (inertial measurement unit). The electronic device can rotate the video based on the rotation value. The electronic device can identify at least one object in the rotated video. The electronic device can set a bounding box for each of the at least one object in the rotated video according to the size of the at least one object. Further, the electronic device may be configured for means of movement including not only vehicles but also bicycles and kick scooters.

[0231] One embodiment of this specification and the terms used therein are not intended to limit the technical features described in this specification to a specific embodiment, but should be understood to include various modifications, equivalents, or alternatives of that embodiment. Regarding the description of the drawings, similar or related components may be assigned similar reference numerals. The singular form of a noun corresponding to an item can include one or more items, unless otherwise clearly indicated in the relevant context. In this specification, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase of that phrase, or all possible combinations thereof. Terms such as "first", "second", or "the first" or "the second" may be used simply to distinguish one component from other corresponding components, and do not limit the component in other aspects (such as importance or order). When a certain (e.g., first) component is referred to as "coupled" or "connected" to another (e.g., second) component, either in combination with the terms "functionally" or "communicatively" or without such terms, it means that one component can be connected to another component directly (e.g., by wire), wirelessly, or via a third component.

[0232] In the specific embodiments of the present disclosure described above, the components included in the disclosure are represented in singular or plural according to the specific embodiments presented. However, the singular or plural expressions are selected for convenience of description in the presented situation, and the present disclosure is not limited to singular or plural components. Components represented in plural may be composed of a single one, and components represented in singular may be composed of a plurality.

[0233] According to various embodiments, one or more of the foregoing corresponding components or operations may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., modules or programs) can be integrated into one component. In this case, the integrated component can perform one or more functions of each of the plurality of components in the same or similar manner as those performed by the corresponding component among the plurality of components before integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, repeatedly, or empirically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0234] Although specific embodiments have been described in the detailed description of the present disclosure, it goes without saying that various modifications are possible without departing from the scope of the present disclosure.

Description of Reference Numerals

[0235] 101 Electronic device 210, 1424, 1610 Processor 220, 1650 Camera 230, 1203, 1410, 1501 Sensor 231 Gyro sensor 232 Acceleration sensor 240, 1422, 1620 Memory 410, 910 Low-pass filter 420, 920 Integrator 430, 930 High-pass filter 600 Specified estimation model 652 Rotation value 803, 811, 1003 Object detection model 900 Complementary filter 1205 Image pre-processor 1207 Deep learning network 1209 AI processor 1211 Vehicle Control Module 1213 Network Interface 1215 Communication Unit 1420 Controller 1430 Wireless Communication Device 1440 Object Detection Device 1306 Engine 1400 Control Device 1422a, 1424a Commands 1422b, 1424b Data 1430a Transmitter 1430b Receiver 1480 Communication Interface 1503 Instrument Cluster 1504 Telematics Device 1505 Gateway 1506 Security Cloud 1508 Third-Party Cloud 1509 Software Management Cloud 1510 In-Vehicle Security Software 1630 Neural Network 1640 Learning Dataset 1660 Communication Circuit

Claims

1. 1. An electronic device comprising: At least one camera; Gyro sensor; memory; and At least one processor operatively connected to the at least one camera, the gyro sensor, and the memory, the at least one processor comprising: acquiring video information using the at least one camera; acquiring first rotation data related to the electronic device using the gyro sensor based on acquiring the image information, and acquiring second rotation data using a specified estimation model; obtaining a rotation value for the image information based on the first rotation data and the second rotation data; An electronic device configured to identify at least one object relating to the video information based on the obtained rotation value.

2. The at least one processor The electronic device of claim 1 , further configured to obtain the rotation value for the video information using a complementary filter based on the first rotation data and the second rotation data.

3. The at least one processor Setting the video information as input data for the specified estimation model; The electronic device of claim 1 , further configured to obtain the second rotation data based on output data of the specified estimation model.

4. The at least one processor 2. The electronic device of claim 1, further configured to train the specified estimation model based on a first training video and a second training video in which the first training video is rotated according to a specified rotation value.

5. The at least one processor acquiring a second image obtained by rotating a first image according to the image information based on the acquired rotation value; identifying the at least one object within the second image using an object detection model; obtaining a third image in which at least one bounding box for the at least one object is displayed superimposed on the second image; The electronic device of claim 1 , further configured to obtain a fourth image based on the third image and the obtained rotation value, the fourth image including the at least one bounding box superimposed within the first image.

6. The at least one processor identifying a first image according to the image information; identifying the at least one object in the first video using an object detection model; obtaining a second image in which at least one bounding box for the at least one object is displayed superimposed on the first image; The electronic device of claim 1 , further configured to obtain a third image in which the at least one bounding box in the second image is displayed with correction based on the obtained rotation value.

7. The first rotation data is The electronic device of claim 1 , further comprising a rotation value identified based on an axis corresponding to a direction in which the at least one camera is pointing.

8. The electronic device comprises: Further comprising a display; The at least one processor displaying at least one bounding box for identifying the at least one object superimposed on a first image corresponding to the image information while the first image is being displayed on the display; The electronic device of claim 1 , further configured to display information about the at least one object based on the at least one bounding box overlaid on the first image.

9. The electronic device comprises: Located in a vehicle, The video information is The electronic device of claim 1 , wherein the image is captured using the at least one camera positioned toward the front of the vehicle.

10. The electronic device comprises: The device is configured to be wearable on a part of a user's body; The video information is The electronic device of claim 1 , wherein the image is captured using the at least one camera positioned toward a front of the user.

11. 1. A method of an electronic device comprising: acquiring video information using at least one camera of said electronic device; acquiring first rotation data related to the electronic device using a gyro sensor of the electronic device based on acquiring the video information, and acquiring second rotation data using a specified estimation model; obtaining a rotation value for the video information based on the first rotation data and the second rotation data; and The method includes an act of identifying at least one object related to the video information based on the obtained rotation value.

12. 12. The method of claim 11, further comprising the act of obtaining the rotation value for the video information using a complementary filter based on the first rotation data and the second rotation data.

13. setting the video information as input data for the specified estimation model; and The method of claim 11 , further comprising an operation of obtaining the second rotation data based on output data of the specified estimation model.

14. 12. The method of claim 11, further comprising: training the specified estimation model based on a first training video and a second training video in which the first training video is rotated according to a specified rotation value.

15. acquiring a second image obtained by rotating a first image corresponding to the image information based on the acquired rotation value; identifying, within the second video, the at least one object using an object detection model; acquiring a third image in which at least one bounding box for the at least one object is displayed superimposed on the second image; and 12. The method of claim 11, further comprising the act of obtaining a fourth image based on the third image and the obtained rotation value, the fourth image including the at least one bounding box superimposed within the first image.

16. identifying a first image according to the image information; identifying the at least one object in the first video using an object detection model; obtaining a second image in which at least one bounding box for the at least one object is displayed superimposed on the first image; and The method of claim 11 , further comprising the act of acquiring a third image in which the at least one bounding box in the second image is displayed with correction based on the acquired rotation value.

17. The first rotation data is The method of claim 11 , including identifying a rotation value based on an axis corresponding to a direction in which the at least one camera is pointing.

18. while a first image responsive to the image information is being displayed via a display of the electronic device, displaying at least one bounding box identifying the at least one object superimposed on the first image; and The method of claim 11 , further comprising the act of displaying information about the at least one object based on the at least one bounding box overlaid on the first image.

19. The electronic device comprises: Located in a vehicle, The video information is The method of claim 11 , wherein the at least one camera is positioned toward the front of the vehicle.

20. The electronic device comprises: The device is configured to be wearable on a part of a user's body; The video information is The method of claim 11 , wherein the image is captured with the at least one camera positioned toward a front of the user.

Images