Electronic device and method for identifying object in image

Abstract

According to one embodiment, an electronic device includes: at least one camera; a gyroscope sensor; a memory; and at least one processor operably connected with the at least one camera, the gyroscope sensor and the memory. The at least one processor is configured to obtain image information by using the at least one camera; on the basis of acquiring the image information, acquiring first rotation data related to the electronic device by using the gyroscope sensor, and acquiring second rotation data by using a specified presumption model; acquiring a rotation value related to the image information based on the first rotation data and the second rotation data; and a recognition unit configured to recognize at least one object related to the image information based on the acquired rotation value.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and method for identifying an object in an image. Background Art

[0002] An electronic device may acquire an image. The electronic device may identify an object in the acquired image. However, when an image is acquired while the electronic device is in a rotated state, the acquired image includes a rotated object, and thus the electronic device may not be able to identify the object.

[0003] The above information may be provided as related art to assist in understanding the present disclosure. No assertion or determination is made as to whether any of the foregoing may be applied as prior art to the present disclosure. Summary of the Invention

[0004] According to an embodiment, an electronic device may include: at least one camera; a gyro sensor; a memory; and at least one processor operatively connected to the at least one camera, the gyro sensor, and the memory. The at least one processor may be configured to: acquire image information using the at least one camera; based on the acquired image information, acquire first rotation data related to the electronic device using the gyro sensor, and acquire second rotation data using a specified estimation model; based on the first rotation data and the second rotation data, acquire a rotation value related to the image information; and based on the acquired rotation value, identify at least one object related to the image information.

[0005] According to an embodiment, a method of an electronic device may include: an operation of acquiring image information using at least one camera of the electronic device; an operation of, based on the acquired image information, acquiring first rotation data related to the electronic device using the gyro sensor of the electronic device, and acquiring second rotation data using a specified estimation model; an operation of acquiring a rotation value related to the image information based on the first rotation data and the second rotation data; and an operation of identifying at least one object related to the image information based on the acquired rotation value.

[0006] According to an embodiment, an electronic device may identify an object based on a rotation value of an image. The electronic device may display an image in which a bounding box is displayed overlapping the identified object. Brief Description of the Drawings

[0007] Figure 1 An example of an electronic device for rotating an image according to an embodiment is shown.

[0008] Figure 2 An example of a block diagram of an electronic device according to an embodiment is shown.

[0009] Figure 3 A schematic flowchart for explaining the operation of an electronic device according to an embodiment is shown.

[0010] Figure 4 An example of the operation of an electronic device according to an embodiment is shown.

[0011] Figure 5A An example of a value obtained by an acceleration sensor according to an embodiment is shown.

[0012] Figure 5B An example of the operation of an electronic device according to an embodiment is shown.

[0013] Figure 6A A schematic flowchart for explaining the operation of an electronic device according to an embodiment is shown.

[0014] Figure 6B An example of a specified estimation model according to an embodiment.

[0015] Figure 7 An example of learning data of a specified estimation model according to an embodiment is shown.

[0016] Figure 8A An example of the operation of an electronic device according to an embodiment is shown.

[0017] Figure 8B An example of the operation of an electronic device according to an embodiment is shown.

[0018] Figure 9 An example of the operation of an electronic device according to an embodiment is shown.

[0019] Figure 10A An example of the operation of an electronic device according to an embodiment is shown.

[0020] Figure 10B An example of the operation of an electronic device according to an embodiment is shown.

[0021] Figure 11 An example of the operation of an electronic device according to an embodiment is shown.

[0022] Figure 12 An example of a block diagram of an autonomous driving system of a vehicle according to an embodiment is shown.

[0023] Figure 13 and Figure 14 An example of a block diagram showing an autonomous driving vehicle according to an embodiment is shown.

[0024] Figure 15 An example of a gateway related to a user device according to various embodiments is shown.

[0025] Figure 16 A block diagram of an electronic device according to an embodiment is shown. Detailed Description

[0026] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the description with reference to the accompanying drawings, like reference numerals may be used for like or related components.

[0027] Figure 1 An example of an electronic device for rotating an image according to an embodiment is shown.

[0028] Referring to Figure 1 , the electronic device 101 may be configured based on various shapes. For example, the electronic device 101 may include the electronic device 101-1. The electronic device 101 may be configured together with the electronic device 101-1. The electronic device 101-1 may correspond to a device (e.g., a blackbox) attached to a vehicle 110 (e.g., a motorcycle), or may be disposed in the device. According to an embodiment, the electronic device 101-1 may correspond to an electronic control unit (ECU) within the vehicle 110, or may be disposed in the ECU. The ECU may be referred to as an electronic control module (ECM). The electronic device 101-1 may be configured as independent hardware for providing functions according to embodiments of the present invention in the vehicle 110. For example, the electronic device 101 may include the electronic device 101-2. The electronic device 101 may be configured together with the electronic device 101-2. The electronic device 101-2 may be worn by a user of the electronic device 101-2. The electronic device 101-2 may be used to provide information to the user while being worn by the user.

[0029] Hereinafter, the operation of the electronic device 101 including the electronic device 101-1 and the electronic device 101-2 will be described.

[0030] According to an embodiment, the electronic device 101 may include a camera disposed in a direction toward the vehicle 110. Figure 1It is shown that the electronic device 101 (or a camera included in the electronic device 101) is disposed toward the front (front direction and / or driving direction) of the vehicle 110, but is not limited thereto. For example, the electronic device 101 may be disposed toward at least one of the rear direction or the side direction of the vehicle 110.

[0031] According to an embodiment, the electronic device 101 may acquire an image 121 of an external vehicle through a camera. The image 121 may be rotated according to the rotation of the vehicle 110 or the electronic device 101. Hereinafter, for ease of description, the rotation value of the electronic device 101 will be described as corresponding to the rotation value of the vehicle 110. This is only for ease of description, and the rotation value of the electronic device 101 may also be different from the rotation value of the vehicle 110.

[0032] For example, the vehicle 110 may rotate based on three axes (e.g., the x-axis, the y-axis, and the z-axis). The order and direction of the three axes shown may be changed according to the embodiment. The rotation for each of the three axes may be defined as roll, pitch, and yaw. The rotation for the x-axis may be defined as roll. The rotation for the y-axis may be defined as pitch. The rotation for the z-axis may be defined as yaw.

[0033] The x-axis may point to the front of the vehicle 110. The camera of the electronic device 101 also faces the front of the vehicle 110, so the image acquired through the camera of the electronic device 101 may be rotated according to the rotation for the x-axis.

[0034] As an example, the electronic device 101 may acquire the image 121. The image 121 may be rotated based on a rotation value related to the roll of the vehicle 110 (or the electronic device 101). For example, the objects 131, 132, and 133 in the image 121 may be rotated based on a rotation value related to the roll of the vehicle 110 (or the electronic device 101). Since the objects 131, 132, and 133 are rotated, the electronic device 101 may not be able to identify the objects 131, 132, and 133. For example, a bounding box for identifying an object may be set larger for the objects (e.g., the objects 131, 132, and 133). When the bounding box is set larger for the objects, the error in object recognition or the estimated distance of the real object for the object may increase.

[0035] Accordingly, the electronic device 101 may rotate the image 121 based on the rotation value related to the roll of the vehicle 110 (or the electronic device 101). The electronic device 101 may rotate the image 121 based on the rotation value related to the roll of the vehicle 110 (or the electronic device 101), thereby identifying the image 122. The image 122 may include the object 141, the object 142, and the object 143. When the image 121 rotates, the object 131 may change to the object 141. When the image 121 rotates, the object 132 may change to the object 142. When the image 121 rotates, the object 133 may change to the object 143.

[0036] The image 122 may be an image acquired in a state where the vehicle 110 (or the electronic device 101) is not rotated. Accordingly, the electronic device 101 may smoothly identify the object 141, the object 142, and the object 143.

[0037] As in the above embodiment, the technical features of rotating an image based on the rotation value of the image (or image information) and identifying an object through the rotated image will be described below.

[0038] Figure 2 An example of a block diagram of an electronic device according to an embodiment is shown. Figure 2 The electronic device 101 may correspond to Figure 1 the electronic device 101.

[0039] Referring to Figure 2 , the electronic device 101 may include at least one of a processor 210, a camera 220, a sensor 230, and a memory 240. The processor 210, the camera 220, the sensor 230, and the memory 240 may be electrically connected and / or operably connected to each other through an electronic component such as a communication bus. Hereinafter, an operable coupling (connection) of a device and / or a circuit means establishing a direct or indirect wired or wireless connection between the device and / or the circuit such that the second circuit and / or the second device is controlled by the first circuit and / or the first device. Although shown in different blocks, the embodiment is not limited thereto. Figure 2 Some of the hardware in Figure 2 may be provided in a single integrated circuit, such as a system on a chip (SoC). The type and / or quantity of the hardware in the electronic device 101 are not limited to Figure 2Some of the hardware shown.

[0040] An electronic device 101 according to an embodiment may include hardware that processes data based on one or more instructions. The hardware for processing data may include a processor 210. For example, the hardware for processing data may include an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). The processor 210 may have a structure of a single-core processor, or may have a structure of a multi-core processor such as a dual core, a quad core, a hexa core, or an octa core.

[0041] According to an embodiment, a camera 220 of the electronic device 101 may include a lens assembly or an image sensor. The lens assembly may collect light emitted from a subject that is an object to be imaged. The lens assembly may include one or more lenses. A camera 220 according to an embodiment may include a plurality of lens assemblies. For example, in the camera 220, some of the plurality of lens assemblies have the same lens properties (e.g., field of view angle, focal length, autofocus, f number, or optical zoom), or at least one lens assembly may have one or more lens properties different from those of another lens assembly. The lens properties may be referred to as intrinsic parameters of the camera 220. The intrinsic parameters may be stored in a memory 240 of the electronic device 101.

[0042] In one embodiment, the lens assembly may include a wide-angle lens or a telephoto lens. According to one embodiment, the flash may include more than one light-emitting diode (LED) (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED) or a xenon lamp. For example, the image sensor in the camera 220 may obtain an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly into an electrical signal. According to one embodiment, the image sensor may include one image sensor selected from a plurality of image sensors having different attributes (e.g., RGB sensor, black and white (BW) sensor, IR sensor, or UV sensor), a plurality of image sensors having the same attribute, or a plurality of image sensors having different attributes. Each image sensor in the image sensor may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

[0043] According to one embodiment, the sensor 230 of the electronic device 101 may be used to obtain various external information. The sensor 230 may include a gyro sensor 231 and an acceleration sensor 232. For example, the gyro sensor 231 may identify (or measure or detect) the angular velocity of the electronic device 101 in three directions (x-axis, y-axis, and z-axis). For example, the acceleration sensor 232 may identify (or measure, detect) the acceleration of the electronic device 101 in three directions (x-axis, y-axis, and z-axis).

[0044] The memory 240 of the electronic device 101 according to an embodiment may include hardware components for storing data and / or instructions input to the processor 210 of the electronic device 101 and / or output from the processor 120 of the electronic device 101. For example, the memory 240 may include a volatile memory such as a random-access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). For example, the volatile memory may include at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). For example, the non-volatile memory may include at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, an optical disk, a solid state drive (SSD), and an embedded multi-media card (eMMC).

[0045] Although not shown, the electronic device 101 may further include various components. For example, the electronic device 101 may further include a display for displaying a user interface.

[0046] According to an embodiment, the electronic device 101 may include a neural network. For example, a neural network is a mathematical model of biological neural activities related to inference and / or cognition and / or hardware (e.g., a CPU, a graphic processing unit (GPU), and / or a neural processing unit (NPU)), software, or any combination thereof for driving the mathematical model. The neural network may be based on a convolutional neural network (CNN) and / or a long-short term memory (LSTM).

[0047] For example, the electronic device 101 may include (or store) a specified estimation model indicated by multiple parameters based on a neural network. The electronic device 101 may utilize the specified estimation model to obtain rotation data. For example, the electronic device 101 may set image information as the input data of the specified estimation model. The electronic device 101 may obtain rotation data based on the output data of the specified estimation model.

[0048] According to an embodiment, the electronic device 101 may utilize the gyro sensor 231 and the acceleration sensor 232 to obtain rotation values related to the image information obtained by using the camera 220. The electronic device 101 may identify at least one object related to the image information based on the obtained rotation values. Examples of the operation of the electronic device 101 obtaining rotation values related to the image information by using the gyro sensor 231 and the acceleration sensor 232 will be described later in Figures 3 to 5B Examples of the operation of the electronic device 101 obtaining rotation values related to the image information by using the gyro sensor 231 and the acceleration sensor 232 will be described later in

[0049] According to an embodiment, the electronic device 101 may utilize the gyro sensor 231 and the specified estimation model to obtain rotation values related to the image information obtained by using the camera 220. The electronic device 101 may identify at least one object related to the image information based on the obtained rotation values. Examples of the operation of the electronic device 101 obtaining rotation values related to the image information by using the gyro sensor 231 and the specified estimation model will be described later in FIGS. 5 to Figure 10B Examples of the operation of the electronic device 101 obtaining rotation values related to the image information by using the gyro sensor 231 and the specified estimation model will be described later in FIGS. 5 to

[0050] Figure 3 A schematic flowchart for explaining the operation of an electronic device according to an embodiment is shown. Figure 2 The electronic device 101 and / or the processor 210 of Figure 3 may perform at least one of the operations of Figure 3 In an embodiment, a computer-readable storage medium may be provided, which includes software applications and / or instructions to cause the electronic device 101 and / or the processor 210 to perform

[0051] Referring to Figure 3 , in operation 310, the processor 210 may obtain image information. For example, the electronic device 101 may move together with the vehicle 110. While the electronic device 101 is moving together with the vehicle 110, the camera 220 facing the front of the vehicle 110 may be used to obtain image information. The first image according to the image information may be in a state rotated based on the rotation value of the vehicle 110 (e.g., the rotation value related to roll).

[0052] In operation 320, the processor 210 may obtain first rotation data by using the gyro sensor 231 and obtain second rotation data by using the acceleration sensor 232.

[0053] According to one embodiment, the processor 210 may obtain first rotation data by using the gyro sensor 231. Based on Figure 1 the three axes shown, the processor 210 calculates a first angular velocity by using the gyro sensor 231 a second angular velocity and a third angular velocity The first angular velocity may be the angular velocity with respect to the x-axis. The second angular velocity may be the angular velocity with respect to the y-axis. The third angular velocity may be the angular velocity with respect to the z-axis.

[0054] The processor 210 may obtain first rotation data based on the first angular velocity The first rotation data may include a rotation value related to the roll of the electronic device 101 (or the vehicle 110). The rotation value related to the roll of the electronic device 101 may be obtained based on the following mathematical formula.

[0055]

First Mathematical Formula

[0056]

[0057] Referring to the first mathematical formula, φ GYR represents the rotation value related to the roll of the electronic device 101. represents the angular velocity with respect to the x-axis. The x-axis may correspond to the direction in which the camera 220 faces (or the direction facing the front of the vehicle 110). t represents time.

[0058] The rotation value related to the roll of the electronic device 101 obtained by using the gyro sensor 231 may be accurate in a short period of time. As shown in the first mathematical formula, the rotation value related to the roll of the electronic device 101 is based on integral calculation, so errors may occur due to drift over time. Therefore, over time, the accuracy of the rotation value related to the roll of the electronic device 101 obtained by using the gyro sensor 231 may decrease.

[0059] According to one embodiment, the processor 210 may obtain second rotation data by using the acceleration sensor 232. Based on Figure 1 the three axes shown, the processor 210 may obtain a first acceleration (A X ), a second acceleration (A Y ), and a third acceleration (A z ) by using the acceleration sensor 232. The first acceleration (A X ) may be the acceleration with respect to the x-axis. The second acceleration (A Y )) may be the acceleration with respect to the y-axis. The third acceleration (A Z) may be the acceleration with respect to the z-axis.

[0060] The processor 210 may obtain second rotation data based on the first acceleration (A X ), the second acceleration (A Y ), and the third acceleration (A Z ). The second rotation data may include rotation values related to the roll of the electronic device 101 (or the vehicle 110). The rotation values related to the roll of the electronic device 101 may be obtained based on the following mathematical formula.

[0061]

Second Mathematical Formula

[0062]

[0063] Referring to the second mathematical formula, φ ACC represents the rotation value related to the roll of the electronic device 101. A X represents the acceleration with respect to the x-axis. A Y represents the acceleration with respect to the y-axis. A Z represents the acceleration with respect to the z-axis. The x-axis may correspond to the direction in which the camera 220 faces (or the direction facing the front of the vehicle 110). The y-axis is parallel to the ground and may correspond to the direction perpendicular to the x-axis. The z-axis is perpendicular to the ground and may correspond to the direction perpendicular to the x-axis.

[0064] The rotation value related to the roll of the electronic device 101 obtained by using the acceleration sensor 232 may be accurate for a long time. Due to the noise of the acceleration sensor 232 in a short time, the rotation value related to the roll of the electronic device 101 obtained by using the acceleration sensor 232 may not be accurate. Therefore, the shorter the measurement time, the lower the accuracy of the rotation value related to the roll of the electronic device 101 obtained by using the acceleration sensor 232.

[0065] In operation 330, the processor 210 may obtain rotation values related to the image information. For example, the processor 210 may obtain rotation values related to the image information based on the first rotation data and the second rotation data.

[0066] For example, the first rotation data may have high accuracy in a short time. The second rotation data may have high accuracy in a long time. Therefore, the processor 210 may use a complementary filter (or a Kalman filter) to obtain rotation values related to the image information. Specific operations of obtaining rotation values related to the image information by using a complementary filter will be described later in conjunction with Figure 4 the description of obtaining rotation values related to the image information by using a complementary filter.

[0067] In operation 340, the processor 210 may identify at least one object related to the image information. For example, the processor 210 may identify at least one object related to the image information based on the acquired rotation value.

[0068] For example, the processor 210 may cause the first image according to the image information to rotate based on the acquired rotation value. The processor 210 causes the first image to rotate based on the acquired rotation value, thereby obtaining a second image. The processor 210 may identify at least one bounding box related to at least one object in the second image. The processor 210 may display at least one bounding box overlapping the second image. The processor 210 causes the second image to rotate (or reverse rotate) based on the acquired rotation value, thereby obtaining a third image. The third image may be an image in which at least one bounding box is displayed overlapping at least one object in the first image.

[0069] Figure 4 An example of the operation of an electronic device according to an embodiment is shown.

[0070] Referring to Figure 4 , the processor 210 of the electronic device 101 may acquire gyro data (e.g., a first angular velocity The processor 210 may acquire acceleration data (e.g., a first acceleration (A X ), a second acceleration (A Y ), and a third acceleration (A Z )) using the acceleration sensor 232. The processor 210 may acquire a rotation value related to the image information using a complementary filter based on the gyro data and the acceleration data.

[0071] For example, the processor 210 may acquire first rotation data using an integrator 420 based on the gyro data. For example, the processor 210 may acquire first rotation data according to a first mathematical formula based on the gyro data. The processor 210 may acquire second rotation data based on the acceleration data. For example, the processor 210 may acquire second rotation data according to a second mathematical formula based on the acceleration data.

[0072] The acceleration sensor 232 may generate a large amount of noise in a high-frequency region (or a short time). Therefore, the processor 210 may apply a low pass filter 410 to the second rotation data to remove the noise. The gyro sensor 231 may generate a drift phenomenon in a low-frequency region (or a long time). Therefore, the processor 210 may apply a high pass filter 430 to the first rotation data to remove the drift phenomenon.

[0073] The processor 210 may obtain a rotation value related to the image information by adding the first rotation data to which the high-pass filter 430 is applied and the second rotation data to which the low-pass filter 410 is applied. For example, the processor 210 may obtain a rotation value related to the image information based on the third mathematical formula.

[0074]

Third Mathematical Formula

[0075]

[0076] Referring to the third mathematical formula, α is a constant. φ t represents the rotation value related to the image information. φ t-1 represents the previously obtained rotation value. represents the angular velocity with respect to the x-axis. φ ACC represents the rotation value according to the second rotation data.

[0077] can be obtained based on applying the high-pass filter 430 to the first rotation data can be obtained based on applying the low-pass filter 410 to the second rotation data α·(φ t-1 )+(1-α)·φ ACC .

[0078] As the magnitude of α increases, the influence of the first rotation data increases, and as the magnitude of α decreases, the influence of the second rotation data may increase. For example, as the magnitude of α increases, the obtained rotation value related to the image information can approach the rotation value obtained using the gyro sensor 231, and as the magnitude of α decreases, the obtained rotation value related to the image information can approach the rotation value obtained using the acceleration sensor 232.

[0079] Figure 5A Shows an example of a value obtained by the acceleration sensor according to an embodiment.

[0080] Figure 5B Shows an example of the operation of the electronic device according to an embodiment.

[0081] Referring to Figure 5A , the electronic device 101 may identify the acceleration of the electronic device 101 using the acceleration sensor 232. Hereinafter, for convenience of description, the operation of identifying the acceleration in a state where the electronic device 101 is disposed inside the vehicle 110 will be described. For example, the acceleration of the electronic device 101 may correspond to the acceleration of the vehicle 110.

[0082] According to an embodiment, when the vehicle 110 moves, a rotation related to roll may occur. When a rotation related to roll occurs, a gravity 501 and an inertial force (centrifugal force) 502 may be generated.

[0083] The processor 210 may identify the acceleration based on the net force 503 of the gravitational force 501 and the inertial force 502. Although not shown, the vehicle 110 (or the electronic device 101) may be subject to various forces other than the gravitational force 501 and the inertial force 502. Accordingly, it may be difficult for the processor 210 to accurately identify the acceleration of the electronic device 101 using the acceleration sensor 232.

[0084] Referring Figure 5B , in a state where the electronic device 101 is not moving, the rotation value related to roll identified according to the second mathematical formula can be accurately identified. However, since when identifying the acceleration of the electronic device 101 during the movement of the electronic device 101, the acceleration generated by gravity and the acceleration generated by the movement are superimposed, the rotation value related to roll identified according to the second mathematical formula may not be accurate.

[0085] The image 590 may represent a frame of image information acquired through the camera 220 of the electronic device 101. During the rapid rotation and movement of the electronic device 101, the processor 210 may acquire the image information.

[0086] To identify the rotation value related to the image information, the processor 210 uses the gyro sensor 231 to identify the first rotation data and uses the acceleration sensor 232 to identify the second rotation data. The processor 210 may identify the rotation value related to the image information based on the first rotation data and the second rotation data.

[0087] The processor 210 may use the gyro sensor 231 to identify the first rotation data. The processor 210 may identify the object 530 representing the horizontal line in the image 590 based on the first rotation data.

[0088] The processor 210 may use the acceleration sensor 232 to identify the second rotation data. The processor 210 may identify the object 510 representing the horizontal line in the image 590 based on the second rotation data.

[0089] The processor 210 may identify the rotation value related to the image information based on the first rotation data and the second rotation data. The processor 210 may identify the object 520 representing the horizontal line in the image 590 based on the rotation value related to the image information.

[0090] The object 530 may represent the horizontal line identified using the gyro sensor 231. Since the gyro sensor 231 can identify accurate rotation values in a short time, the object 530 may be closest to the actual horizontal line.

[0091] The object 510 may represent the horizon recognized using the acceleration sensor 232. As Figure 5A described, during the movement of the electronic device 101 (or the vehicle 110), due to the force applied to the electronic device 101 (or the vehicle 110), the processor 210 may not be able to recognize the accurate rotation value. Therefore, the object 510 may not be able to correctly represent the actual horizon.

[0092] The object 520 may represent the horizon recognized based on the horizon recognized using the gyro sensor 231 and the horizon recognized using the acceleration sensor 232. Since the error between the horizon recognized using the acceleration sensor 232 and the actual horizon is large, the object 520 may also not be able to correctly represent the actual horizon.

[0093] According to an embodiment, the processor 210 may recognize the value related to the roll of the electronic device 101 (or the vehicle 110) as the following mathematical formula.

[0094]

Fourth Mathematical Formula

[0095]

[0096] Referring to the fourth mathematical formula, θ (theta) may represent the value related to the roll of the electronic device 101 (or the vehicle 110). v may represent the forward speed of the electronic device 101 (or the vehicle 110). r may represent the radius of rotation. g may represent the acceleration due to gravity.

[0097] In order to use the fourth mathematical formula to recognize the value related to the roll of the electronic device 101 (or the vehicle 110), the radius of rotation of the electronic device 101 (or the vehicle 110) is required. An additional component must be included in the electronic device 101 to recognize the radius of rotation of the electronic device 101 (or the vehicle 110). Therefore, in the case where the additional component is not provided in the electronic device 101, the processor 210 cannot use the fourth mathematical formula to recognize the value related to the roll of the electronic device 101 (or the vehicle 110). In addition, since the fourth mathematical formula can be used under ideal conditions such as a flat road (rather than an inclined road), a large error may occur in the value related to the roll of the electronic device 101 (or the vehicle 110).

[0098] As described above, it may be difficult for the processor 210 to recognize the accurate rotation value of the image information only using the gyro sensor 231 and the acceleration sensor 232. Therefore, the processor 210 may use the gyro sensor 231 and a specified estimation model to recognize the rotation value of the image information. Hereinafter, the technical features of recognizing the rotation value of the image information using the gyro sensor 231 and the specified estimation model will be described.

[0099] Figure 6AA schematic flowchart is shown for explaining the operation of an electronic device according to an embodiment. Figure 2 The electronic device 101 and / or the processor 210 may perform Figure 6A at least one of the operations. In an embodiment, a computer-readable storage medium may be provided, which includes software applications and / or instructions to cause the electronic device 101 and / or the processor 210 to perform Figure 6A the operations.

[0100] Referring to Figure 6A , in operation 601, the processor 210 may acquire image information. For example, the electronic device 101 may move together with the vehicle 110. While the electronic device 101 is moving together with the vehicle 110, the camera 220 facing the front of the vehicle 110 may be used to acquire image information. The first image according to the image information may be in a state rotated based on the rotation value of the vehicle 110 (e.g., the rotation value related to roll).

[0101] In operation 602, the processor 210 may acquire first rotation data using the gyro sensor 231 and acquire second rotation data using a specified estimation model.

[0102] According to an embodiment, the processor 210 may acquire first rotation data using the gyro sensor 231. The operation of acquiring first rotation data using the gyro sensor 231 may correspond to the operation of acquiring first rotation data in operation 320.

[0103] According to an embodiment, the processor 210 may acquire second rotation data using a specified estimation model. For example, the processor 210 may acquire second rotation data in various driving environments including flat roads and inclined roads using a specified estimation model.

[0104] For example, the specified estimation model may be composed of a combination of a convolution layer and a fully connected (FC) layer. Specific examples of the specified estimation model will be described later in Figure 6B .

[0105] For example, the processor 210 may set the image information as the input data of the specified estimation model. The processor 210 may acquire second rotation data based on the output data of the specified estimation model. The second rotation data may include a rotation value related to the roll of the electronic device 101 (or the vehicle 110). The processor 210 may identify the horizontal line in the image information. The processor 210 may identify the rotation value related to the roll of the electronic device 101 (or the vehicle 110) based on the angle of rotation of the identified horizontal line in the image information.

[0106] According to one embodiment, the processor 210 may cause a specified estimation model to learn based on a first learning image and a second learning image obtained by rotating the first learning image according to a specified rotation value. Specific examples of operations for causing the specified estimation model to learn will be described later in Figure 7 the operations for causing the specified estimation model to learn will be described later in

[0107] In operation 603, the processor 210 may obtain a rotation value related to the image information. For example, the processor 210 may obtain a rotation value related to the image information based on first rotation data and second rotation data.

[0108] According to one embodiment, since the second rotation data is obtained using a specified estimation model, it may not be possible to correctly estimate the rotation value related to the roll of the electronic device 101 (or the vehicle 110) through a scene (or frame, special pattern) not used for learning. Therefore, the processor 210 may use a complementary filter (or a Kalman filter) to obtain a rotation value related to the image information. Specific operations for obtaining a rotation value related to the image information using a complementary filter will be described later in conjunction with Figure 9 the operations for obtaining a rotation value related to the image information using a complementary filter will be described later in conjunction with

[0109] In operation 604, the processor 210 may identify at least one object related to the image information. For example, the processor 210 may identify at least one object related to the image information based on the obtained rotation value.

[0110] For example, the processor 210 may cause a first image according to the image information to be rotated based on the obtained rotation value. By causing the first image to be rotated based on the obtained rotation value, the processor 210 may obtain a second image. The processor 210 may identify at least one bounding box related to at least one object in the second image. The processor 210 may display at least one bounding box overlapping the second image. By causing the second image to be rotated (or rotated in the opposite direction) based on the obtained rotation value, the processor 210 may obtain a third image. The third image may be an image in which at least one bounding box is displayed overlapping at least one object in the first image.

[0111] will be described later with reference to Figure 10A and Figure 10B the operations for the electronic device 101 (or the processor 210) to identify at least one object related to the image information.

[0112] Figure 6B is an example of a specified estimation model according to one embodiment.

[0113] Refer to Figure 6B, the specified estimation model 600 may be composed of a combination of a convolutional layer 610 and a fully-connected layer 620. For example, the convolutional layer 610 may be used to maintain spatial information related to the image 651 and extract features. The fully-connected layer 620 may be used to output data within a specified range.

[0114] For example, the number of layers of the convolutional layer 610 may be 13. For example, the number of layers of the fully-connected layer 620 may be 3. For example, the specified estimation model 600 may be configured based on the VGG16 model. The specified estimation model 600 may include at least a part of the VGG16 model. For example, the specified estimation model 600 may be a model obtained by adding a fully-connected layer for regression to the final classification end in the VGG16 model.

[0115] The image 651 may be set as the input data of the convolutional layer 610. For example, the image 651 may be a frame of video according to the video information acquired by the camera 220. The image 651 may be composed of a three-dimensional vector of 222x224x3. A three-dimensional vector of 7x7x512 is identified based on the output data related to the convolutional layer 610. The three-dimensional vector of 7x7x512 may be set as the input data of the fully-connected layer 620. Through the fully-connected layer 620, a rotation value 652 within a specified range (e.g., 0 or more to 25 degrees or less) can be output.

[0116] Therefore, the processor 210 may use driving images (or driving pictures) for various environments to train the specified estimation model 600. The processor 210 may use the specified estimation model 600 to obtain second rotation data. The second rotation data may include a rotation value related to the roll of the electronic device 101 (or the vehicle 110).

[0117] Figure 7 An example of learning data of the specified estimation model according to an embodiment is shown.

[0118] Refer to Figure 7 , the processor 210 may train the specified estimation model. For example, in order to train the specified estimation model, it may be necessary to obtain images and rotation values during the movement of the electronic device 101 (or the vehicle 110). It may require a large amount of resources to obtain images and rotation values during the movement of the electronic device 101 (or the vehicle 110). Therefore, the processor 210 may identify the images 710, 720, and 730 for augmentation of learning data.

[0119] The processor 210 may be an image that rotates the image 710 by 0 degrees. The processor 210 may acquire the image 710 in a state where the rotation value related to roll is 0 degrees while the vehicle 110 is traveling. The processor 210 may recognize the image 710 as the correct answer (ground truth). The processor 210 may cause the specified estimation model 600 to learn based on the image 710.

[0120] The processor 210 may rotate the image 710 according to the specified rotation value, thereby acquiring the image 720. The processor 210 may crop the region 721 including the object in the image 720. The processor 210 may acquire (or recognize) the image 730 by cropping the region 721 including the object in the image 720. The processor 210 may recognize the image 730 as the correct answer for the specified rotation value. For example, the processor 210 may remove the outer region of the image by cropping the region 721 including the object in the image 720. According to an embodiment, the processor 210 may acquire the image 730 by cropping a specified region (e.g., the central region) in the image 720 regardless of the object.

[0121] For example, the processor 210 may set the specified rotation value to any value within the specified range. The processor 210 may acquire the correct answer image (e.g., the image 730) by acquiring images rotated according to various rotation values.

[0122] Therefore, the processor 210 may rotate the image 710 acquired during the rotation value of the electronic device 101 being 0 degrees based on the specified rotation value to acquire the image 730. The processor 210 may recognize the image 730 as the correct answer for the specified rotation value.

[0123] In the above embodiment, for ease of description, an example of causing the specified estimation model 600 to learn using images (e.g., the image 710, the image 720, and the image 730) is shown, but it is not limited thereto. The processor 210 may recognize the first learning image as the correct answer (ground truth). The processor 210 may acquire the second learning image based on rotating the first learning image according to the specified rotation value. The processor 210 may recognize the second learning image as the correct answer for the specified rotation value.

[0124] Figure 8A An example of the operation of an electronic device according to an embodiment is shown.

[0125] Figure 8B An example of the operation of an electronic device according to an embodiment is shown.

[0126] Refer to Figure 8A, the processor 210 can use the camera 220 to obtain image information. The processor 210 can identify a first image based on the image information.

[0127] The processor 210 can use a specified estimation model 600 to obtain (or identify) a rotation value. The processor 210 can set the image information (or the first image) as the input data of the specified estimation model 600. The processor 210 can obtain (or identify) a rotation value based on the output data of the specified estimation model 600.

[0128] In operation 802, the processor 210 can rotate the first image based on the obtained rotation value. The processor 210 can obtain (or identify) a second image based on rotating the first image. The second image can be an image with the rotation removed from the first image. The processor 210 can set the second image as the input data of an object detection model 803. The processor 210 can obtain a third image based on the output data of the object detection model 803. The processor 210 can identify at least one object in the second image through the object detection model 803. The processor 210 can obtain a third image in which at least one bounding box related to at least one object is displayed overlapping the second image.

[0129] Refer to Figure 8B , the processor 210 can use the camera 220 to obtain image information. The processor 210 can identify a first image based on the image information.

[0130] The processor 210 can set the image information (or the first image) as the input data of an object detection model 811. The processor 210 can obtain a second image based on the output data of the object detection model 811. The processor 210 can identify at least one object in the first image through the object detection model 811. The processor 210 can obtain a second image in which at least one bounding box related to at least one object is displayed overlapping the first image.

[0131] The processor 210 can use a specified estimation model 600 to obtain (or identify) a rotation value. The processor 210 can set the image information (or the first image) as the input data of the specified estimation model 600. The processor 210 can obtain (or identify) a rotation value based on the output data of the specified estimation model 600.

[0132] In operation 813, the processor 210 can rotate at least one bounding box based on the obtained rotation value. For example, since at least one object in the second image is in a rotated state, at least one bounding box is larger than at least one object. The processor 210 can obtain a third image in which at least one bounding box is corrected and displayed by rotating at least one bounding box.

[0133] Referring again to Figure 8A , after the processor 210 rotates the first image, the object recognition model 803 can be used to recognize at least one object. Since the unrotated image (or picture) is set as the input data of the object recognition model 803, the ordinary object recognition model 803 can be used. However, due to the rotation of the first image, there may be a delay.

[0134] Referring again to Figure 8B , the processor 210 can use the object recognition model 811 to recognize at least one object without rotating the first image. The processor 210 can rotate at least one bounding box recognized based on the object recognition model 811. Since the first image is not rotated, at least one object can be recognized with low latency. However, the processor 210 may need to configure the rotated image as the learning data of the object recognition model 811.

[0135] Figure 9 An example of the operation of an electronic device according to an embodiment is shown.

[0136] Referring to Figure 9 , the processor 210 of the electronic device 101 can use the gyro sensor 231 to obtain gyro data (e.g., the first angular velocity The processor 210 can use the camera 220 to obtain image information. The processor 210 can use the complementary filter 900 to obtain a rotation value related to the image information based on the gyro data and the image information.

[0137] For example, the processor 210 can use the integrator 920 to obtain the first rotation data based on the gyro data. For example, the processor 210 can obtain the first rotation data based on the gyro data according to the first mathematical formula. The processor 210 can use the specified estimation model 600 to obtain the second rotation data based on the image information. For example, the processor 210 can set the image information as the input data of the specified estimation model 600. The processor 210 can obtain the second rotation data based on the output data of the specified estimation model 600.

[0138] Since the second rotation data is obtained using the specified estimation model 600, the rotation value related to the roll of the electronic device 101 (or the vehicle 110) may not be correctly estimated through a scene (or frame, special pattern) not used for learning. Therefore, the processor 210 can use the complementary filter (or Kalman filter) to obtain the rotation value related to the image information based on the first rotation data and the second rotation data. For example, the processor 210 can obtain the rotation value related to the image information based on the fifth mathematical formula.

[0139]

Fifth Mathematical Formula

[0140]

[0141] Referring to the fifth mathematical formula, α is a constant. φ t represents a rotation value related to the image information. φ t-1 represents a previously obtained rotation value. represents the angular velocity with respect to the x-axis. φ IMG represents a rotation value based on the second rotation data.

[0142] can be obtained based on applying a high-pass filter 930 to the first rotation data can be obtained based on applying a low-pass filter 910 to the second rotation data to obtain α·(φ t-1 )+(1-α)·φ IMG .

[0143] For example, in a short period of time, the rotation value related to the image information can be close to the rotation value in the first rotation data obtained using the gyro sensor 231. For example, in a long period of time, the rotation value related to the image information can be close to the rotation value in the second rotation data obtained using the specified estimation model 600.

[0144] Figure 10A Illustrates an example of the operation of an electronic device according to an embodiment.

[0145] Figure 10B Illustrates an example of the operation of an electronic device according to an embodiment.

[0146] Referring to Figure 10A , the processor 210 can obtain image information using the camera 220. The processor 210 can identify a first image according to the image information.

[0147] The processor 210 can identify at least one object using the specified estimation model 600. The processor 210 can set the first image (or image information) as the input data of the specified estimation model 600. The processor 210 can obtain second rotation data based on the output data of the specified estimation model 600. The processor 210 can obtain first rotation data using gyro data. The processor 210 can set the first rotation data and the second rotation data as the input values of the complementary filter 900. The processor 210 can use Figure 9 the complementary filter 900 described in to obtain a rotation value related to the image information. For example, the processor 210 can obtain a rotation value related to the image information using the fifth mathematical formula.

[0148] In operation 1001, the processor 210 may rotate the first image based on the acquired rotation value. The processor 210 may acquire (or identify) a second image based on rotating the first image. The second image may be an image in which the rotation has been removed from the first image. The processor 210 may set the second image as input data for the object detection model 1003. The processor 210 may acquire a third image based on the output data of the object detection model 1003. The processor 210 may identify at least one object in the second image through the object detection model 1003. The processor 210 may acquire a third image in which at least one bounding box associated with at least one object is displayed overlapping the second image.

[0149] Referring Figure 10B , the processor 210 may acquire image information using the camera 220. The processor 210 may identify a first image based on the image information.

[0150] The processor 210 may set the image information (or the first image) as input data for the object detection model 1011. The processor 210 may acquire a second image based on the output data of the object detection model 1011. The processor 210 may identify at least one object in the first image through the object detection model 1011. The processor 210 may acquire a second image in which at least one bounding box associated with at least one object is displayed overlapping the first image.

[0151] The processor 210 may acquire (or identify) second rotation data using the specified estimation model 600. The processor 210 may set the image information (or the first image) as input data for the specified estimation model 600. The processor 210 may acquire (or identify) a rotation value based on the output data of the specified estimation model 600. The processor 210 may acquire first rotation data using gyroscope data. The processor 210 may set the first rotation data and the second rotation data as input values for the complementary filter 900. The processor 210 may use Figure 9 the complementary filter 900 described in to acquire a rotation value related to the image information. For example, the processor 210 may acquire a rotation value related to the image information using the fifth mathematical formula.

[0152] In operation 1013, the processor 210 may rotate at least one bounding box based on the acquired rotation value. For example, since at least one object in the second image is in a rotated state, at least one bounding box is larger than at least one object. The processor 210 may acquire a third image in which at least one bounding box is corrected and displayed by rotating at least one bounding box.

[0153] Referring again to Figure 10A, after the processor 210 rotates the first image, it can use the object recognition model 1003 to recognize at least one object. Since the unrotated image (or picture) is set as the input data of the object recognition model 1003, the ordinary object recognition model 1003 can be used. However, the rotation of the first image may cause delay.

[0154] Referring again to Figure 10B , the processor 210 can use the object recognition model 1011 to recognize at least one object without rotating the first image. The processor 210 can rotate at least one bounding box recognized based on the object recognition model 1011. Since the first image is not rotated, at least one object can be recognized with low latency. However, the processor 210 may need to configure the rotated image as the learning data of the object recognition model 1011.

[0155] Figure 11 Shows an example of the operation of an electronic device according to an embodiment.

[0156] Referring to Figure 11 , the processor 210 can use the camera 220 to obtain image information. The image 1100 can be obtained based on a frame of the first image according to the image information. The processor 210 can use the display in the electronic device 101 or the display of an external device to display the image 1100.

[0157] The processor 210 can use the gyro sensor 231 to obtain the first rotation data. The object 1110 can be displayed based on the rotation value identified by the first rotation data. The object 1110 can represent the horizon identified based on the first rotation data in the image 1100.

[0158] The processor 210 can use the specified estimation model 600 to obtain the second rotation data. The object 1120 can be displayed based on the rotation value identified by the second rotation data. The object 1120 can represent the horizon identified based on the second rotation data in the image 1100.

[0159] The processor 210 can identify the rotation value related to the image information based on the first rotation data and the second rotation data. The processor 210 can display the object 1130 in the image 1100 based on the rotation value related to the image information. For example, the processor 210 can use the complementary filter 900 to obtain the rotation value related to the image information based on the first rotation data and the second rotation data. The object 1130 can be displayed based on the rotation value related to the image information. The object 1130 can represent the horizon identified based on the rotation value related to the image information.

[0160] The processor 210 may obtain a rotation value related to the image information based on the first rotation data and the second rotation data. The processor 210 may display, in the image 1110, an object 1130 representing a horizon close to the actual horizon based on the rotation value related to the image information.

[0161] According to an embodiment, the processor 210 may identify at least one object in the image 1100 based on the rotation value related to the image information. The processor 210 may rotate the image 1100 based on the rotation value. After the image 1100 is rotated, the processor 210 may use an object detection model to identify at least one object.

[0162] The processor 210 may display, in the image 1100, at least one bounding box related to the identified at least one object. The at least one bounding box may be in a state rotated based on the rotation value related to the image information. For example, the processor 210 may identify an object 1101, an object 1102, an object 1103, and an object 1104 in the image 1100. The processor 210 may display a bounding box 1111 in the image 1100 according to the shape of the object 1101. The processor 210 may display a bounding box 1112 in the image 1100 according to the shape of the object 1102. The processor 210 may display a bounding box 1113 in the image 1100 according to the shape of the object 1103. The processor 210 may display a bounding box 1114 in the image 1100 according to the shape of the object 1104.

[0163] For example, the bounding box 1111, the bounding box 1112, the bounding box 1113, and the bounding box 1114 may be in a state rotated based on the rotation value related to the image information.

[0164] According to an embodiment, the processor 210 may display, in the image 1100, the text 1121 for representing the information related to the object 1101 together with the bounding box 1111. The information related to the object 1101 may represent the distance between the real object corresponding to the object 1101 and the electronic device 101. For example, the text 1121 may represent the distance in the first direction (e.g., the left direction or the right direction) (i.e., -5 m), the distance in the second direction (e.g., the forward direction) (i.e., 3 m), and the shortest distance (i.e., 5.83 m).

[0165] The processor 210 may display the text 1122 for representing information related to the object 1102 together with the bounding box 1112 in the image 1100. The information related to the object 1102 may represent the distance between the real object corresponding to the object 1102 and the electronic device 101. For example, the text 1121 may represent the distance in the first direction (e.g., the left direction or the right direction) (i.e., -3 m), the distance in the second direction (e.g., the forward direction) (i.e., 3 m), and the shortest distance (i.e., 5.83 m).

[0166] The processor 210 may display the text 1123 for representing information related to the object 1103 together with the bounding box 1113 in the image 1100. The information related to the object 1103 may represent the distance between the real object corresponding to the object 1103 and the electronic device 101. For example, the text 1121 may represent the distance in the first direction (e.g., the left direction or the right direction) (i.e., 2 m), the distance in the second direction (e.g., the forward direction) (i.e., 3 m), and the shortest distance (i.e., 5.83 m).

[0167] The processor 210 may display the text 1124 for representing information related to the object 1104 together with the bounding box 1114 in the image 1100. The information related to the object 1104 may represent the distance between the real object corresponding to the object 1104 and the electronic device 101. For example, the text 1121 may represent the distance in the first direction (e.g., the left direction or the right direction) (i.e., 3 m), the distance in the second direction (e.g., the forward direction) (i.e., 3 m), and the shortest distance (i.e., 5.83 m).

[0168] As described above, the processor 210 may configure the bounding box to fit the size of the object and display it in the image (or video). The processor 210 may rotate and display the bounding box according to the rotation value of the video.

[0169] Figure 12 is an example of a block diagram showing an autonomous driving system of a vehicle according to an embodiment.

[0170] According to Figure 12The autonomous driving system 1200 of a vehicle can be a deep learning network including a plurality of sensors 1203, an image pre-processor 1205, a deep learning network 1207, an artificial intelligence (AI) processor 1209, a vehicle control module 1211, a network interface 1213, and a communication unit 1215. In various embodiments, each element can be connected through various interfaces. For example, the sensor data sensed and output by the plurality of sensors 1203 can be fed to the image pre-processor 1205. The sensor data processed by the image pre-processor 1205 can be fed to the deep learning network 1207 running on the AI processor 1209. The output of the deep learning network 1207 running on the AI processor 1209 can be fed to the vehicle control module 1211. The intermediate result of the deep learning network 1207 running on the AI processor 1209 can be fed to the AI processor 1209. In various embodiments, the network interface 1213 communicates with the electronic devices within the vehicle, thereby transferring the autonomous driving path information and / or the autonomous driving control commands for the autonomous driving of the vehicle to a plurality of internal block configurations. In one embodiment, the network interface 1213 can be used to send the sensor data obtained through the sensor(s) 1203 to an external server. In some embodiments, the autonomous driving control system 1200 can appropriately include additional or fewer components. For example, in some embodiments, the image pre-processor 1205 can be an optional component. As another example, a post-processing component (not shown) can be included within the autonomous driving control system 1200 to perform post-processing on the output of the deep learning network 1207 before being provided to the vehicle control module 1211.

[0171] In some embodiments, the plurality of sensors 1203 may include more than one sensor. In various embodiments, the plurality of sensors 1203 may be attached to different locations of the vehicle. The plurality of sensors 1203 may be oriented in more than one different direction. For example, the plurality of sensors 1203 may be attached to the front, sides, rear, and / or roof of the vehicle in a manner that is forward-facing, rear-facing, and side-facing, etc. In some embodiments, the plurality of sensors 1203 may be image sensors such as a plurality of high dynamic range cameras. In some embodiments, the plurality of sensors 1203 includes a plurality of non-visual sensors. In some embodiments, in addition to image sensors, the plurality of sensors 1203 further includes RADAR, Light Detection And Ranging (LiDAR), and / or ultrasonic sensors. In some embodiments, the plurality of sensors 1203 is not mounted to a vehicle having a vehicle control module 1211. For example, the plurality of sensors 1203 is included as part of a deep learning system for capturing sensor data and may be attached to the environment or road and / or mounted on a plurality of surrounding vehicles.

[0172] In some embodiments, an Image pre-processor 1205 may be used to preprocess the sensor data of the plurality of sensors 1203. For example, the Image pre-processor 1205 may be used to preprocess the sensor data, split the sensor data into more than one component, and / or post-process more than one component. In some embodiments, the Image pre-processor 1205 may be a graphics processing unit (GPU), a central processing unit (CPU), an image signal processor, or a specialized image processor. In various embodiments, the Image pre-processor 1205 may be a tone-mapper processor for processing high dynamic range data. In some embodiments, the Image pre-processor 1205 may be a component of the AI processor 1209.

[0173] In some embodiments, the deep learning network 1207 can be a deep learning network for implementing control commands for an autonomous vehicle. For example, the deep learning network 1207 can be an artificial neural network such as a convolutional neural network (CNN) trained using sensor data, and the output of the deep learning network 1207 is provided to the vehicle control module 1211.

[0174] In some embodiments, the artificial intelligence (AI) processor 1209 can be a hardware processor for running the deep learning network 1207. In some embodiments, the AI processor 1209 is a dedicated AI processor for performing inference on sensor data through a convolutional neural network (CNN). In some embodiments, the AI processor 1209 can be optimized for the bit depth of sensor data. In some embodiments, the AI processor 1209 can be optimized for deep learning computations such as computations of neural networks including convolution, dot product, vector, and / or matrix calculations. In some embodiments, the AI processor 1209 can be implemented by multiple graphics processing units (GPUs) capable of effectively performing parallel processing.

[0175] In various embodiments, the AI processor 1209 performs deep learning analysis on sensor data received from the sensor(s) 1203 during the runtime of the AI processor 1209, and can be coupled through an input / output interface to a memory configured to provide the AI processor, which triggers instructions for determining machine learning results for at least partially autonomously actuating the vehicle. In some embodiments, the Vehicle Control Module 1211 processes a plurality of commands for controlling the vehicle output from the artificial intelligence (AI) processor 1209, and can be used to translate the output of the AI processor 1209 into a plurality of instructions for modules for controlling the vehicle to control various modules of the vehicle. In some embodiments, the Vehicle Control Module 1211 is used to control the vehicle for autonomous driving. In some embodiments, the Vehicle Control Module 1211 can adjust the steering and / or speed of the vehicle. For example, the Vehicle Control Module 1211 can be used to control vehicle driving such as decelerating, accelerating, steering, changing lanes, and maintaining lanes. In some embodiments, the Vehicle Control Module 1211 can generate control signals for controlling vehicle lighting, such as brake lights, turn signals, headlights, etc. In some embodiments, the Vehicle Control Module 1211 can be used to control vehicle audio-related systems, such as the vehicle's sound system, the vehicle's audio warnings, the vehicle's microphone system, the vehicle's horn system, etc.

[0176] In some embodiments, the Vehicle Control Module 1211 is used to control notification systems, which include systems for notifying passengers and / or drivers of driving events, such as approaching a predetermined destination or a potential collision. In some embodiments, the Vehicle Control Module 1211 can be used to adjust sensors, such as the vehicle's sensor 1203. For example, the Vehicle Control Module 1211 can modify the orientation of the sensor 1203, change the output resolution and / or format type of the sensor 1203, increase or decrease the capture rate, adjust the dynamic range, and adjust the focus of the camera. Additionally, the Vehicle Control Module 1211 can turn on / off the operation of multiple sensors individually or collectively.

[0177] In some embodiments, the vehicle control module 1211 may be used to change the parameters of the image pre-processor 1205 by modifying the frequency range of the filter, or by adjusting the edge detection parameter for features and / or object detection, or by adjusting channels and bit depth, etc. In various embodiments, the vehicle control module 1211 may be used to control the autonomous driving of the vehicle and / or the driver assistance function of the vehicle.

[0178] In some embodiments, the network interface 1213 may be responsible for the internal interface between the block structure of the autonomous driving control system 1200 and the communication unit 1215. Specifically, the network interface 1213 may be a communication interface for receiving and / or sending data (including voice data). In various embodiments, the network interface 1213 may be connected to an external server to make a voice call through the communication unit 1215, or receive and / or send text messages, or send sensor data, or update the software of the vehicle through the autonomous driving system, or update the software of the autonomous driving system of the vehicle.

[0179] In various embodiments, the communication unit 1215 may include various wireless interfaces in a cellular or WiFi manner. For example, the network interface 1213 may be used to receive action parameters and / or updates for instructions for the plurality of sensors 1203, the image pre-processor 1205, the deep learning network 1207, the AI processor 1209, and the vehicle control module 1211 from an external server connected through the communication unit 1215. For example, the machine learning model of the deep learning network 1207 may use the communication unit 1215 to update. According to another example, the communication unit 1215 may be used to update the action parameters of the image pre-processor 1205 such as image processing parameters and / or the firmware of the sensors 1203.

[0180] In another embodiment, the communication unit 1215 may be used to activate communication for emergency services and emergency contacts in the event of an accident or near-accident. For example, in the event of a collision, the communication unit 1215 may be used to call for emergency services for help and may be used to notify the emergency services of the collision details and the location of the vehicle. In various embodiments, the communication unit 1215 may update or obtain the expected arrival time and / or the destination location.

[0181] According to one embodiment, Figure 12The illustrated autonomous driving system 1200 may also be composed of the electronic devices of a vehicle. According to an embodiment, when an event of a user canceling autonomous driving occurs during the autonomous driving of the vehicle, the AI processor 1209 of the autonomous driving system 1200 may control the learning of the autonomous driving software for controlling the vehicle by controlling the input of information related to the autonomous driving cancellation event into the training set data of the deep learning network.

[0182] Figure 13 and Figure 14 Shows an example of a block diagram showing an autonomous driving mobile body according to an embodiment. Refer to Figure 13 , the autonomous driving mobile body 1300 according to the present embodiment may include a control device 1400, sensing modules 1304a, 1304b, 1304c, 1304d, an engine 1306, and a user interface 1308.

[0183] The autonomous driving mobile body 1300 may have an autonomous driving mode or a manual mode. As an example, it may be switched from the manual mode to the autonomous driving mode or from the autonomous driving mode to the manual mode according to a user input received through the user interface 1308.

[0184] When the mobile body 1300 is operating in the autonomous driving mode, the autonomous driving mobile body 1300 may operate under the control of the control device 1400.

[0185] In the present embodiment, the control device 1400 may include a controller 1420 (including a memory 1422 and a processor 1424), a sensor 1410, a communication device 1430, and an object detection device 1440.

[0186] Among them, the object detection device 1440 may perform all or part of the functions of a distance measurement device (for example, the electronic device 101).

[0187] That is, in the present embodiment, the object detection device 1440 is a device for detecting an object located outside the mobile body 1300, and the object detection device 1440 may detect an object located outside the mobile body 1300 and generate object information according to the detection result.

[0188] The object information may include information on the presence or absence of an object, the position information of the object, the distance information between the mobile body and the object, and the relative speed information between the mobile body and the object.

[0189] The object may include various objects located outside the moving body 1300, such as lanes, other vehicles, pedestrians, traffic signals, light, roads, structures, speed bumps, landmarks, and animals. Among them, the traffic signal may be a concept including traffic lights, traffic signs, patterns or texts drawn on the road surface. In addition, the light may be light generated from a lamp provided in another vehicle, or light generated from a street lamp, or sunlight.

[0190] In addition, the structure may be an object located around the road and fixed to the ground. For example, the structure may include street lamps, roadside trees, buildings, utility poles, traffic lights, and bridges. The terrain may include mountains, hills, etc.

[0191] The object detection device 1440 may include a camera module. The controller 1420 may extract object information from an external image captured by the camera module and allow the controller 1420 to process this information.

[0192] In addition, the object detection device 1440 may further include an imaging device for identifying the external environment. In addition to LIDAR, radar (RADAR), GPS devices, travel distance measurement devices (odometry), and other computer vision devices, ultrasonic sensors, and infrared sensors may be used. These devices may be selected or operate simultaneously as needed to achieve more accurate sensing.

[0193] On the other hand, the distance measurement device according to an embodiment of the present invention calculates the distance between the autonomous driving moving body 1300 and the object, and controls the movement of the moving body based on the calculated distance in association with the control device 1400 of the autonomous driving moving body 1300.

[0194] As an example, when there is a possibility of collision according to the distance between the autonomous driving moving body 1300 and the object, the autonomous driving moving body 1300 may control the brake to decelerate or stop. As another example, when the object is a moving object, the autonomous driving moving body 1300 may control the traveling speed of the autonomous driving moving body 1300 to maintain a distance greater than a specified distance from the object.

[0195] The distance measurement device according to an embodiment of the present invention may be configured as a module within the control device 1400 of the autonomous driving moving body 1300. That is, the memory 1422 and the processor 1424 of the control device 1400 may implement the collision prevention method according to the present invention in software.

[0196] In addition, the sensor 1410 can obtain various sensing information by connecting the internal / external environment of the moving body to the sensing modules 1304a, 1304b, 1304c, and 1304d. Among them, the sensor 1410 can include a posture sensor (e.g., a yaw sensor, a roll sensor, a pitch sensor), a collision sensor, a wheel sensor, a speed sensor, an inclination sensor, a weight sensing sensor, a heading sensor, a gyro sensor, a position module, a moving body forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor rotated by the steering wheel, an internal temperature sensor of the moving body, an internal humidity sensor of the moving body, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, etc.

[0197] Thus, the sensor 1410 can obtain sensing signals for the moving body posture information, moving body collision information, moving body direction information, moving body position information (GPS information), moving body angle information, moving body speed information, moving body acceleration information, moving body inclination information, moving body forward / backward information, battery information, fuel information, tire information, moving body light information, internal temperature information of the moving body, internal humidity information of the moving body, steering wheel rotation angle, external illuminance of the moving body, pressure applied to the accelerator pedal, pressure applied to the brake pedal, etc.

[0198] In addition, the sensor 1410 can include an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake air temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a top dead center (TDC) sensor, a crankshaft angle sensor (CAS), etc.

[0199] In this way, the sensor 1410 can generate moving body state information based on the sensed data.

[0200] The wireless communication device 1430 is configured for wireless communication between the autonomous mobile body 1300. For example, the autonomous mobile body 1300 can communicate with the user's mobile phone, or another wireless communication device 1430, another mobile body, a central device (traffic control device), a server, etc. The wireless communication device 1430 can send and receive wireless signals according to the connected wireless protocol. The wireless communication protocol can be Wi-Fi, Bluetooth, Long-Term Evolution (LTE), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Global Systems for Mobile Communications (GSM), and the communication protocol is not limited to these.

[0201] In addition, in this embodiment, the autonomous mobile body 1300 can also achieve communication between mobile bodies through the wireless communication device 1430. That is, the wireless communication device 1430 can communicate with other mobile bodies on the road through vehicle-to-vehicle (V2V) communication. The autonomous mobile body 1300 can send and receive information such as driving warnings and traffic information through vehicle-to-vehicle communication, and can also request information from other mobile bodies or receive requests from other mobile bodies. For example, the wireless communication device 1430 can perform V2V communication as a dedicated short-range communication (DSRC) device or a cellular V2V (C-V2V) device. In addition, in addition to vehicle-to-vehicle communication, communication between the vehicle and other objects (such as electronic devices carried by pedestrians, etc.) can also be achieved through the wireless communication device 1430 (V2X, Vehicle to Everything communication).

[0202] In addition, the wireless communication device 1430 can obtain information generated from infrastructure on the road (traffic lights, closed-circuit television (CCTV), Road Side Unit (RSU), Evolved Node B (eNode B), etc.) or various mobility entities including autonomous driving / non-autonomous driving vehicles, etc., through a Non-Terrestrial Network instead of a Terrestrial Network, as information for performing autonomous driving of the autonomous driving mobility entity 1300.

[0203] For example, the wireless communication device 1430 can perform wireless communication with a Low Earth Orbit (LEO) satellite system, a Medium Earth Orbit (MEO) satellite system, a Geostationary Orbit (GEO) satellite system, a High Altitude Platform (HAP) system, etc., that constitute a Non-Terrestrial Network (NTN), through a non-terrestrial network dedicated antenna mounted on the autonomous driving mobility entity 1300.

[0204] For example, the wireless communication device 1430 performs wireless communication with various platforms constituting the NTN according to a radio access standard of the 5TH Generation New Radio Non-Terrestrial Network (5G NR NTN) standard currently being discussed in 3GPP, but is not limited thereto.

[0205] In this embodiment, the controller 1420 can consider various information such as the position of the autonomous driving mobility entity 1300, the current time, available power, etc., select an appropriate NTN communication platform, and control the wireless communication device 1430 to perform wireless communication with the selected platform.

[0206] In this embodiment, the controller 1420 is a unit that controls the overall operation of each unit within the moving body 1300, and can be configured by the manufacturer of the moving body during manufacturing or can be additionally configured after manufacturing to perform the function of autonomous driving. Alternatively, it can include a configuration for continuously executing additional functions by upgrading the controller 1420 configured during manufacturing. This controller 1420 can also be referred to as an Electronic Control Unit (ECU).

[0207] The controller 1420 collects various data from the connected sensors 1410, object detection device 1440, communication device 1430, etc., and based on the collected data, transmits control signals to the sensors 1410, engine 1306, user interface 1308, communication device 1430, and object detection device 1440 provided in other configurations within the moving body. Additionally, although not shown, control signals can be sent to an acceleration device, braking system, steering device, or navigation device related to the driving of the moving body.

[0208] In this embodiment, the controller 1420 can control the engine 1306. For example, it can control the engine 1306 so that the autonomous driving moving body 1300 senses the speed limit of the road it is traveling on and controls the traveling speed not to exceed the speed limit, or it can control the engine 1306 to accelerate the traveling speed of the autonomous driving moving body 1300 within the range not exceeding the speed limit.

[0209] Furthermore, if the autonomous driving moving body 1300 approaches or deviates from the lane during the driving of the autonomous driving moving body 1300, the controller 1420 determines whether this approach or deviation from the lane is caused by normal driving conditions or other driving conditions, and can control the engine 1306 according to the determination result to control the driving of the moving body. Specifically, the autonomous driving moving body 1300 can detect the lane lines formed on both sides of the lane for the moving body to travel. In this case, the controller 1420 determines whether the autonomous driving moving body 1300 is approaching or deviating from the lane, and if it determines that the autonomous driving moving body 1300 is approaching or deviating from the lane, it can determine whether this driving is caused by normal driving conditions or other driving conditions. Among them, taking normal driving conditions as an example, it can be a situation where the lane of the moving body needs to be changed. Additionally, as an example of other driving conditions, it can be a situation where the lane of the moving body does not need to be changed. If the controller 1420 determines that the autonomous driving moving body 1300 is approaching or deviating from the lane when the lane of the moving body does not need to be changed, it controls the driving of the autonomous driving moving body 1300 so that the autonomous driving moving body 1300 does not deviate from the lane and travels normally on this lane.

[0210] If there is another moving object or an obstacle in front of the moving object, the engine 1306 or the braking system can be controlled to decelerate the moving object in motion, and in addition to the speed, the trajectory, the travel path, and the steering angle can also be controlled. Alternatively, the controller 1420 can generate necessary control signals based on the recognition information of other external environments such as the travel lane of the moving object and the driving information to control the travel of the moving object.

[0211] In addition to generating its own control signals, the controller 1420 can also communicate with surrounding moving objects or a central server, and send commands for controlling peripheral devices through the received information, thereby also controlling the travel of the moving object.

[0212] In addition, when the position of the camera module 1450 changes or the field of view angle changes, it is difficult for the controller 1420 to accurately recognize the moving object or the lane according to this embodiment. Therefore, to prevent this situation, control signals can be generated to control the calibration of the camera module 1450. Therefore, in this embodiment, the controller 1420 generates a calibration control signal for the camera module 1450, so that even if the installation position of the camera module 1450 changes due to vibrations or impacts generated by the movement of the autonomous driving moving object 1300, the normal installation position, orientation, field of view angle, etc. of the camera module 1450 can be continuously maintained. When the difference between the initially stored installation position, orientation, and field of view angle information of the camera module 1450 and the installation position, orientation, and field of view angle information of the camera module 1450 measured during the travel of the autonomous driving moving object 1300 is above a threshold value, the controller 1420 can generate a control signal to perform the calibration of the camera module 1450.

[0213] In this embodiment, the controller 1420 can include a memory 1422 and a processor 1424. The processor 1424 can execute the software stored in the memory 1422 according to the control signal of the controller 1420. Specifically, the controller 1420 stores the data and commands for executing the lane detection method according to the present invention in the memory 1422, and the commands can be executed by the processor 1424 to implement one or more methods disclosed herein.

[0214] At this time, the memory 1422 can be stored in a recording medium executable by the non-volatile processor 1424. The memory 1422 can store software and data through appropriate internal or external devices. The memory 1422 can be composed of a memory 1422 device connected to a random access memory (RAM), a read only memory (ROM), a hard disk, and a dongle.

[0215] The memory 1422 can store at least an operating system (OS), user application programs, and executable commands. The memory 1422 can also store application data and array data structures.

[0216] The processor 1424 can be a microprocessor or any suitable electronic processor, such as a controller, microcontroller, or state machine.

[0217] The processor 1424 can be implemented as a combination of computing devices, and the computing devices can be digital signal processors, microprocessors, or a suitable combination thereof.

[0218] On the other hand, the autonomous mobile body 1300 can also include a user interface 1308 for user input to the above control device 1400. The user interface 1308 can allow the user to input information through appropriate interactions. For example, it can be implemented through a touch screen, keyboard, operation buttons, etc. The user interface 1308 sends the input or command to the controller 1420, and the controller 1420 can perform control actions of the mobile body in response to the input or command.

[0219] In addition, the user interface 1308 can enable devices external to the autonomous mobile body 1300 to communicate with the autonomous mobile body 1300 through the wireless communication device 1430. For example, the user interface 1308 can be linked with a mobile phone, tablet computer, or other computer devices.

[0220] Furthermore, in this embodiment, the autonomous mobile body 1300 has been described as including an engine 1306, but it can also include other types of propulsion systems. For example, the mobile body can be operated by electric energy, hydrogen energy, or a hybrid power system combining them. Therefore, the controller 1420 can include a propulsion mechanism according to the propulsion system of the autonomous mobile body 1300, and accordingly provide control signals to the configuration of each propulsion mechanism.

[0221] Hereinafter, reference will be made to Figure 14 The detailed configuration of the control device 1400 according to this embodiment will be described in more detail.

[0222] The control device 1400 includes a processor 1424. The processor 1424 can be a general-purpose single-chip or multi-chip microprocessor, a dedicated microprocessor, a microcontroller, a programmable gate array, etc. The processor can also be referred to as a central processing unit (CPU). In addition, in this embodiment, the processor 1424 can be used as a combination of multiple processors.

[0223] Moreover, the control device 1400 includes a memory 1422. The memory 1422 can also be any electronic component capable of storing electronic information. In addition to a single memory, the memory 1422 can also include a combination of memories 1422.

[0224] Data and instructions 1422a for performing the distance measurement method of the distance measurement device according to the present invention can also be stored in the memory 1422. When the processor 1424 executes the instructions 1422a, all or part of the instructions 1422a and the data 1422b required for execution commands can also be loaded 1424a, 1424b onto the processor 1424.

[0225] The control device 1400 can also include a transmitter 1430a, a receiver 930b, or a transceiver 930c to allow the transmission and reception of signals. One or more antennas 1432a, 1432b can also be electrically connected to the transmitter 1430a, the receiver 930b, or each transceiver 930c, and can additionally include antennas.

[0226] The control device 1400 can also include a digital signal processor (DSP) 1470. Through the digital signal processor 1470, the mobile body can quickly process digital signals.

[0227] The control device 1400 can also include a communication interface 1480. The communication interface 1480 can also include one or more ports and / or communication modules for connecting other devices to the control device 1400. The communication interface 1480 can allow a user to interact with the control device 1400.

[0228] Various configurations of the control device 1400 can also be connected together by one or more buses 1490, and the buses 1490 can also include a power bus, a control signal bus, a status signal bus, a data bus, etc. Under the control of the processor 1424, multiple configurations can transmit information to each other through the bus 1490 and perform desired functions.

[0229] On the other hand, in various embodiments, the control device 1400 can be associated with a gateway to communicate with a security cloud.

[0230] Figure 15 An example of a gateway related to a user device according to various embodiments is shown.

[0231] Refer to Figure 15, the control device 1400 can be associated with a gateway 1505 that is configured to provide information obtained from at least one of the components (1501 to 1504) of the vehicle 1500 to the security cloud 1506. For example, the gateway 1505 can be provided within the control device 1400. As another example, the gateway 1505 can also be configured as a separate device disposed within the vehicle 1500 that is distinct from the control device 1400. The gateway 1505 connects the software management cloud 1509, the security cloud 1506, and the network within the vehicle 1500 protected by the in-vehicle security software 1510, which have different networks, so as to enable communication.

[0232] For example, the component 1501 can be a sensor. For example, the sensor can be used to obtain information related to at least one of the state of the vehicle 1500 or the state around the vehicle 1500. For example, the component 1501 can include the sensor 1410.

[0233] For example, the component 1502 can be an electronic control unit (ECU). For example, the ECU can be used for engine control, transmission control, airbag control, and tire pressure management.

[0234] For example, the component 1503 can be an instrument cluster. For example, the instrument cluster can refer to the panel located in front of the driver's seat on the dashboard. For example, the instrument cluster can be configured to display information required for driving to the driver (or passenger). For example, the instrument cluster can be used to display at least one of a visual element for indicating the revolutions per minute (RPM) of the engine, or a visual element for indicating the speed of the vehicle 1500, or a visual element for indicating the remaining fuel level, or a visual element for indicating the gear state, or a visual element for indicating the information obtained by the component 1501.

[0235] For example, the component 1504 may be a telematics device. For example, the telematics device may refer to a device that provides various mobile communication services (such as location information and safe driving, etc.) within the vehicle 1500 by combining wireless communication technology and global positioning system (GPS) technology. For example, the telematics device may be used to connect the vehicle 1500 with the driver, the cloud (such as, the security cloud 1506) and / or the surrounding environment. For example, the telematics device may be configured to support the high bandwidth and low latency of the new radio standard (5G NR) in the fifth generation of mobile communication technology (such as, the vehicle-to-everything (V2X) technology of 5G NR, the non-terrestrial network (NTN) technology of 5G NR). For example, the telematics device may be configured to support the autonomous driving of the vehicle 1500.

[0236] For example, the gateway 1505 may be used to connect the software management cloud 1509 and the security cloud 1506, which are the networks inside and outside the vehicle 1500. For example, the software management cloud 1509 may be used to update or manage at least one software required for the driving and management of the vehicle 1500. For example, the software management cloud 1509 may be linked with the in-car security software 1510 installed in the vehicle. For example, the in-car security software 1510 may be used to provide security functions within the vehicle 1500. For example, the in-car security software 1510 may encrypt the data transmitted and received through the in-vehicle network using the encryption key obtained from an external authorized server to encrypt the in-vehicle network. In various embodiments, the encryption key used by the in-car security software 1510 may be generated corresponding to the identification information of the vehicle (license plate number, vehicle identification number (VIN)) or the information uniquely assigned to each user (such as, user identification information).

[0237] In various embodiments, the gateway 1505 may send data encrypted by the in-vehicle security software 1510 to the software management cloud 1509 and / or the security cloud 1506 based on an encryption key. The software management cloud 1509 and / or the security cloud 1506 decrypt the data using a decryption key capable of decrypting the data encrypted by the encryption key of the in-vehicle security software 1510, so as to identify from which vehicle or user the data is received. For example, since the decryption key is a unique key corresponding to the encryption key, the software management cloud 1509 and / or the security cloud 1506 can identify the sending entity of the data (e.g., the vehicle or the user) based on the data through the decryption key.

[0238] For example, the gateway 1505 is configured to support the in-vehicle security software 1510 and may be associated with the control device 2100. For example, the gateway 1505 may be associated with the control device 1400 to support the connection between the client device 1507 connected to the security cloud 1506 and the control device 1400. As another example, the gateway 1505 may be associated with the control device 1400 to support the connection between the third-party cloud 1508 connected to the security cloud 1506 and the control device 1400. However, it is not limited thereto.

[0239] In various embodiments, the gateway 1505 may be used to connect the software management cloud 1509 for managing the operating software of the vehicle 1500 to the vehicle 1500. For example, the software management cloud 1509 may monitor whether the operating software of the vehicle 1500 needs to be updated, and based on the monitoring that the operating software of the vehicle 1500 needs to be updated, provide data for updating the operating software of the vehicle 1500 through the gateway 1505. As another example, the software management cloud 1509 may receive a user request to update the operating software of the vehicle 1500 from the vehicle 1500 through the gateway 1505, and provide data for updating the operating software of the vehicle 1500 based on the received request. However, it is not limited thereto.

[0240] Figure 16 A block diagram of an electronic device according to an embodiment is shown. Figure 16 The electronic device 101 may include Figures 1 to 11 the electronic device 101.

[0241] Refer to Figure 16, the processor 1610 of the electronic device 101 may perform computations related to the neural network 1630 stored in the memory 1620. The processor 1610 may include at least one of a CPU (central processing unit), a GPU (graphics processing unit), or a neural processing unit (NPU). The NPU may be implemented as a chip separate from the CPU, or may be integrated in the form of a system on a chip (SoC) into a chip such as the CPU. The NPU integrated into the CPU may be referred to as a neural core and / or an artificial intelligence (AI) accelerator.

[0242] Referring to Figure 16 , the processor 1610 may identify the neural network 1630 stored in the memory 1620. The neural network 1630 may include an input layer 1632, one or more hidden layers 1634 (or a combination of intermediate layers and output layers 1636). The layers described above (e.g., the input layer 1632, one or more hidden layers 1634, and the output layer 1636) may include a plurality of nodes. The number of hidden layers 1634 may vary according to embodiments, and the neural network 1630 including multiple hidden layers 1634 may be referred to as a deep neural network. The action of training the deep neural network may be referred to as deep learning.

[0243] In one embodiment, when the neural network 1630 has a feed forward neural network structure, the first nodes included in a specific layer are all the second nodes included in another layer before the specific layer. Within the memory 1620, the parameters stored for the neural network 1630 may include weights assigned to the connections between the multiple second nodes and the first nodes. In the neural network 1630 having a feed forward neural network structure, the value of the first node corresponds to the weighted sum of the values assigned to the multiple second nodes based on the weights assigned to the connections between the multiple second nodes and the first node.

[0244] In one embodiment, when the neural network 1630 has a structure of a convolutional neural network, a first node in a specific layer may correspond to a weighted sum of a part of a plurality of second nodes in another layer before the specific layer. A part of the plurality of second nodes corresponding to the first node may be identified by a filter corresponding to the specific layer. In the memory 1620, parameters stored for the neural network 1630 may include weights representing the filter. The filter may include one or more nodes among the plurality of second nodes to be used for calculating the weighted sum of the first node and weights corresponding to each of the one or more nodes.

[0245] The processor 1610 of the electronic device 101 according to an embodiment may train the neural network 1630 by using the learning data set 1640 stored in the memory 1620. Based on the learning data set 1640, the processor 1610 may adjust one or more parameters stored in the memory 1620 for the neural network 1630.

[0246] The processor 1610 of the electronic device 101 according to an embodiment may perform object detection, object recognition, and / or object classification by using the neural network 1630 trained based on the learning data set 1640. The processor 1610 may input an image (or video) acquired through the camera 1650 to the input layer 1632 of the neural network 1630. Based on the input layer 1632 into which the image is input, the processor 1610 may sequentially obtain values of a plurality of nodes in a plurality of layers in the neural network 1630 and obtain a set of values of the plurality of nodes in the output layer 1636 (for example, output data). The output data may be used for a result of inferring information in the image by using the neural network 1630. The embodiment is not limited thereto, and the processor 1610 may input an image (or video) acquired from an external electronic device connected to the electronic device 101 through the communication circuit 1660 to the neural network 1630.

[0247] In one embodiment, the neural network 1630 trained to process an image may be used to identify (object detection) a region corresponding to a subject in the image and / or identify (object recognition and / or object classification) a category of the subject represented in the image. For example, the electronic device 101 may use the neural network 1630 to segment a region corresponding to the subject in the image based on a rectangular shape such as a bounding box. For example, the electronic device 101 may use the neural network 1630 to identify at least one category that matches the subject among a plurality of specified categories.

[0248] According to an embodiment, an electronic device may include: at least one camera; a gyro sensor; a memory; and at least one processor operably connected to the at least one camera, the gyro sensor, and the memory. The at least one processor may be configured to: obtain image information using the at least one camera; based on the obtained image information, obtain first rotation data related to the electronic device using the gyro sensor, and obtain second rotation data using a specified estimation model; obtain a rotation value related to the image information based on the first rotation data and the second rotation data; and identify at least one object related to the image information based on the obtained rotation value.

[0249] According to an embodiment, the at least one processor may further be configured to: obtain the rotation value related to the image information using a complementary filter based on the first rotation data and the second rotation data.

[0250] According to an embodiment, the at least one processor may further be configured to: set the image information as input data of the specified estimation model, and obtain the second rotation data based on output data of the specified estimation model.

[0251] According to an embodiment, the at least one processor may further be configured to: cause the specified estimation model to learn based on a first learning image and a second learning image obtained by rotating the first learning image according to a specified rotation value.

[0252] According to an embodiment, the at least one processor may further be configured to: obtain a second image obtained by rotating a first image based on the obtained rotation value; in the second image, identify the at least one object using an object detection model; obtain a third image in which at least one bounding box related to the at least one object is displayed overlapping the second image; and obtain a fourth image in which at least one of the bounding boxes is displayed overlapping the first image based on the third image and the obtained rotation value.

[0253] According to an embodiment, the at least one processor may further be configured to: identify a first image based on the image information; identify the at least one object in the first image using an object detection model; obtain a second image in which at least one bounding box related to the at least one object is displayed overlapping the first image; and obtain a third image obtained by correcting and displaying at least one of the bounding boxes in the second image based on the obtained rotation value.

[0254] According to one embodiment, the first rotation data may include a rotation value identified based on an axis corresponding to a direction in which at least one camera is oriented.

[0255] According to one embodiment, the electronic device may further include: a display. The at least one processor may further be configured to: while a first image based on the image information is being displayed through the display, display at least one bounding box for identifying at least one object overlapping with the first image; and based on at least one of the bounding boxes, display information related to the at least one object overlapping with the first image.

[0256] According to one embodiment, the electronic device is configured in a vehicle and may acquire the image information by using at least one camera disposed toward the front of the vehicle.

[0257] According to one embodiment, the electronic device is configured to be wearable on a part of the user's body and may acquire the image information by using at least one camera disposed toward the front of the user.

[0258] According to one embodiment, a method of an electronic device may include: an operation of acquiring image information by using at least one camera of the electronic device; an operation of acquiring first rotation data related to the electronic device by using a gyro sensor of the electronic device and acquiring second rotation data by using a specified estimation model on the basis of the acquired image information; an operation of acquiring a rotation value related to the image information based on the first rotation data and the second rotation data; and an operation of identifying at least one object related to the image information based on the acquired rotation value.

[0259] According to one embodiment, the method may further include: an operation of acquiring the rotation value related to the image information by using a complementary filter based on the first rotation data and the second rotation data shown in the method.

[0260] According to one embodiment, the method may further include: an operation of setting the image information as input data of a specified estimation model; and an operation of acquiring the second rotation data based on output data of the specified estimation model.

[0261] According to one embodiment, the method may further include: an operation of causing a specified estimation model to learn based on a first learning image and a second learning image obtained by rotating the first learning image by a specified rotation value.

[0262] According to an embodiment, the method may further include: an operation of obtaining a second image obtained by rotating a first image based on the image information based on the obtained rotation value; an operation of identifying the at least one object in the second image by using an object detection model; an operation of obtaining a third image in which at least one bounding box associated with the at least one object is displayed overlapping the second image; and an operation of obtaining a fourth image in which at least one of the bounding boxes is displayed overlapping the first image based on the third image and the obtained rotation value.

[0263] According to an embodiment, the method may further include: an operation of identifying a first image based on the image information; an operation of identifying the at least one object in the first image by using an object detection model; an operation of obtaining a second image in which at least one bounding box associated with the at least one object is displayed overlapping the first image; and an operation of obtaining a third image by correcting and displaying at least one of the bounding boxes in the second image based on the obtained rotation value.

[0264] According to an embodiment, the first rotation data may include a rotation value identified based on an axis corresponding to a direction in which the at least one camera is oriented.

[0265] According to an embodiment, the method may further include: an operation of displaying at least one bounding box for identifying the at least one object overlapping the first image while the first image based on the image information is being displayed on a display of the electronic device; and an operation of displaying information associated with the at least one object overlapping the first image based on at least one of the bounding boxes.

[0266] According to an embodiment, the electronic device is disposed in a vehicle. The image information may be obtained by using at least one camera disposed facing the front of the vehicle.

[0267] According to an embodiment, the electronic device may be configured to be wearable on a part of the user's body; the image information may be obtained by using at least one camera disposed facing the front of the user.

[0268] According to an embodiment, even without a high-performance inertial measurement unit (IMU), an electronic device (e.g., electronic device 101) can use at least one of a gyro sensor and a specified estimation model to identify the rotation value of the electronic device (or vehicle). The electronic device can rotate an image based on the rotation value. The electronic device can identify at least one object in the rotated image. The electronic device can set the bounding box of at least one object to match the size of at least one object in the rotated image. Additionally, the electronic device can be configured for a vehicle and means of transportation including bicycles and floating boards.

[0269] It should be understood that the various embodiments herein and the terms used therein are not intended to limit the technical features described herein to a specific embodiment, and include various modifications, equivalents, or alternatives of the embodiment. In the description with reference to the accompanying drawings, similar reference numerals may be used for similar or related components. Unless the relevant context clearly indicates otherwise, the singular form of a noun corresponding to an item may include one or more of the above items. Herein, phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase or any possible combination thereof. Terms such as "first", "second", or "first or second" may simply be used to distinguish one component from another, and these components are not restricted in other aspects (such as importance or order). If a certain (e.g., first) component is referred to as "coupled" or "connected" to another (e.g., second) component, with or without the terms "functionally" or "communicatively", it means that the certain component can be connected to the other component in a direct manner (e.g., wired), wirelessly, or through a third component.

[0270] In the specific embodiments of the present disclosure above, according to the specific embodiments presented, the components included in the present disclosure are expressed in singular or plural. However, for convenience of description, the singular or plural expression is selected to suit the presented situation, and the present disclosure is not limited to singular or plural components. Even components expressed in plural can be configured in a singular manner, or even components expressed in singular can be composed of plural.

[0271] According to an embodiment, one or more of the corresponding components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., modules or programs) may be integrated into one component. In this case, the integrated component may perform one or more functions of each of the plurality of components in the same or similar manner as the corresponding component among the plurality of components performed before integration. According to an embodiment, the operations performed by a module, program, or other component may be performed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be performed in a different order, omitted, or one or more other operations may be added.

[0272] In the detailed description of the present disclosure, specific embodiments have been described, but of course, various modifications can be made without departing from the scope of the present disclosure.

Claims

1. An electronic device, wherein: include: At least one camera, Gyroscope sensor, Memory, and at least one processor operably connected to the at least one camera, the gyro sensor, and the memory; The at least one processor is configured to: acquiring image information using the at least one camera, On the basis of acquiring the image information, using the gyro sensor to acquire first rotation data related to the electronic device, and using a specified estimation model to acquire second rotation data, Based on the first rotation data and the second rotation data, obtaining a rotation value related to the image information, and At least one object related to the image information is identified based on the acquired rotation value.

2. The electronic device according to claim 1, wherein: The at least one processor is further configured to: Based on the first rotation data and the second rotation data, a complementary filter is used to obtain the rotation value related to the image information.

3. The electronic device according to claim 1, wherein: The at least one processor is further configured to: Setting the image information as input data of the specified estimation model, The second rotation data is acquired based on the output data of the specified estimation model.

4. The electronic device according to claim 1, wherein: The at least one processor is further configured to: The designated estimation model is learned based on a first learning image and a second learning image obtained by rotating the first learning image according to a designated rotation value.

5. The electronic device according to claim 1, wherein: The at least one processor is further configured to: Based on the obtained rotation value, a second image is obtained by rotating the first image based on the image information. identifying the at least one object in the second image using an object detection model, acquiring a third image in which at least one bounding box associated with the at least one object is displayed overlapping the second image, Based on the third image and the obtained rotation value, a fourth image is obtained in which at least one of the bounding boxes is displayed overlapping the first image.

6. The electronic device according to claim 1, wherein: The at least one processor is further configured to: identifying a first image based on the image information, identifying the at least one object in the first image using an object detection model, acquiring a second image in which at least one bounding box associated with the at least one object is displayed overlapping the first image, Based on the acquired rotation value, a third image is acquired by correcting at least one of the bounding boxes in the second image and then displayed.

7. The electronic device according to claim 1, wherein: The first rotation data includes a rotation value identified based on an axis corresponding to a direction in which the at least one camera is facing.

8. The electronic device according to claim 1, wherein: The electronic device also includes a display; The at least one processor is configured to: During the period when a first image based on the image information is displayed on the display, at least one bounding box for identifying the at least one object is displayed overlapping with the first image, and Based on at least one of the bounding boxes, information related to the at least one object is displayed overlapping the first image.

9. The electronic device according to claim 1, wherein: The electronic device is arranged in the vehicle, The image information is acquired using the at least one camera disposed toward the front of the vehicle.

10. The electronic device according to claim 1, wherein: The electronic device is configured to be wearable on a part of the user's body; The image information is acquired by using the at least one camera arranged toward the front of the user.

11. A method for object recognition using an electronic device, wherein: include: an action of acquiring image information using at least one camera of the electronic device, Based on the image information, the first rotation data related to the electronic device is obtained by using the gyro sensor of the electronic device, and the second rotation data is obtained by using a specified estimation model. an action of acquiring a rotation value related to the image information based on the first rotation data and the second rotation data, and Based on the acquired rotation value, a motion of at least one object related to the image information is identified.

12. The object recognition method using an electronic device according to claim 11, wherein: Also includes: The action of acquiring the rotation value related to the image information by using a complementary filter based on the first rotation data and the second rotation data.

13. The object recognition method using an electronic device according to claim 11, wherein: Also includes: The action of setting the image information as input data of the specified estimation model, and The step of acquiring the second rotation data based on the output data of the specified estimation model.

14. The object recognition method using an electronic device according to claim 11, wherein: Also includes: The operation of learning the designated estimation model based on a first learning image and a second learning image obtained by rotating the first learning image according to a designated rotation value.

15. The object recognition method using an electronic device according to claim 11, wherein: Also includes: An action of obtaining a second image obtained by rotating a first image based on the image information based on the obtained rotation value, recognizing, in the second image, a motion of the at least one object using an object detection model, an act of acquiring a third image having at least one bounding box associated with the at least one object displayed overlapping the second image, and An action of acquiring, based on the third image and the acquired rotation value, a fourth image in which at least one of the bounding boxes is displayed overlapping the first image.

16. The object recognition method using an electronic device according to claim 11, wherein: Also includes: identifying an action of a first image based on the image information, identifying a motion of the at least one object in the first image using an object detection model, an act of acquiring a second image having at least one bounding box associated with the at least one object displayed overlapping the first image, and An action of acquiring a third image by correcting at least one of the bounding boxes in the second image and displaying the third image based on the acquired rotation value.

17. The object recognition method using an electronic device according to claim 11, wherein: The first rotation data includes a rotation value identified based on an axis corresponding to a direction in which the at least one camera is facing.

18. The object recognition method using an electronic device according to claim 11, wherein: Also includes: An action of displaying at least one bounding box for identifying at least one object overlapping with a first image while a first image based on the image information is displayed on a display of the electronic device; as well as An action of displaying information related to the at least one object overlapping the first image based on at least one of the bounding boxes.

19. The object recognition method using an electronic device according to claim 11, wherein: The electronic device is arranged in the vehicle, The image information is acquired using the at least one camera disposed toward the front of the vehicle.

20. The object recognition method using an electronic device according to claim 11, wherein: The electronic device is configured to be wearable on a part of the user's body; The image information is acquired by using the at least one camera arranged toward the front of the user.

Images