Electronic device, method, and non-transitory computer-readable storage medium for protecting privacy

Abstract

The electronic device includes a camera, a memory, a first processor configured to perform calculations related to an artificial intelligence model, and a second processor configured to compress first video data acquired from the camera. The first processor receives the first video data from the camera, performs object recognition using the first video data to identify a subject area within the first video data, and transmits information related to the subject area to the second processor. The second processor receives the first video data from the camera and receives the information from the first processor, combines a mask object corresponding to the subject area indicated by the information with the first video data, compresses the first video data combined with the mask object to generate second video data, and stores the second video data in the memory.

Description

Technology Field

[0001] This relates to an electronic device, a method, and a non-transient computer-readable storage medium for protecting privacy. Background Technology

[0003] Recently, technologies for capturing, storing, and / or transmitting video data through electronic devices, including cameras, are being utilized in various fields. However, captured images may contain individuals' faces or identifiable information. In particular, technical countermeasures are required to effectively protect privacy information during the process of transmitting or storing images externally.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art in relation to the present disclosure. means of solving the problem

[0006] An electronic device is described. The electronic device includes a camera, a memory, a first processor configured to perform calculations related to an artificial intelligence model, and a second processor configured to compress first video data acquired from the camera. The first processor may receive the first video data from the camera, perform object recognition using the first video data to identify a subject area within the first video data, and transmit information related to the subject area to the second processor. The second processor may receive the first video data from the camera, receive the information from the first processor, combine a mask object corresponding to the subject area indicated by the information with the first video data, compress the first video data combined with the mask object to generate second video data, and store the second video data in the memory.

[0007] A method is described. The method may be executed by an electronic device comprising a camera, a first processor configured to perform calculations related to an artificial intelligence model, and a second processor configured to compress first video data acquired from the camera. The method may include controlling the first processor to receive the first video data from the camera, performing object recognition using the first video data to identify a subject area within the first video data, and transmitting information related to the subject area to the second processor. The method may include controlling the second processor to receive the first video data from the camera, receiving the information from the first processor, combining a mask object corresponding to the subject area indicated by the information with the first video data, compressing the first video data combined with the mask object to generate second video data, and storing the second video data in the memory.

[0008] A non-transient computer-readable storage medium is described. The non-transient computer-readable storage medium may store one or more programs. The one or more programs may be executed by an electronic device comprising a camera, a first processor configured to perform calculations related to an artificial intelligence model, and a second processor configured to compress first video data acquired from the camera. The above one or more programs may include instructions such as causing the electronic device to control the first processor to receive the first video data from the camera, perform object recognition using the first video data to identify a subject area within the first video data, and transmit information related to the subject area to the second processor, and to control the second processor to receive the first video data from the camera, receive the information from the first processor, combine a mask object corresponding to the subject area indicated by the information with the first video data, compress the first video data combined with the mask object to generate a second video data, and store the second video data in the memory. Brief explanation of the drawing

[0010] FIG. 1 is a simplified block diagram of the electronic device of the present invention. Figure 2 illustrates an image corresponding to the first video data acquired through the camera module of Figure 1. FIG. 3 illustrates an embodiment of performing object recognition of the image of FIG. 2. FIG. 4a illustrates an image corresponding to second video data generated by combining a mask object with the subject area of ​​the image of FIG. 2. FIG. 4b illustrates an image corresponding to the fourth video data created by deleting image data corresponding to the subject area of ​​the image in FIG. 2. Figure 5 is a flowchart showing the operation of the electronic device of Figure 1. Figure 6 is a flowchart showing the operation of an electronic device when a playback input is acquired. Figure 7 is a flowchart showing the operation of an electronic device when a transmission input is acquired. FIG. 8 is a flowchart illustrating an embodiment of generating fourth video data by deleting image data of the subject area. FIG. 9 illustrates an example of a block diagram illustrating an autonomous driving system of a vehicle according to one embodiment. FIGS. 10 and FIGS. 11 illustrate an example of a block diagram representing an autonomous driving vehicle according to one embodiment. FIG. 12 illustrates an example of a gateway associated with a user device according to various embodiments. FIG. 13 is a diagram illustrating the operation of an electronic device for training a neural network based on a set of training data according to one embodiment. FIG. 14 is a block diagram of an electronic device according to one embodiment. Specific details for implementing the invention

[0011] In the drawings below, identical, similar, or corresponding reference numerals may be assigned to identical, similar, or corresponding components, and descriptions thereof may not be repeated. In descriptions referring to specific drawings below, reference numerals of other drawings may be referenced.

[0012] In this specification, the expression “A, B, or C” is used in an inclusive sense, including “A,” “B,” “C,” or “any combination thereof,” unless the context clearly indicates otherwise. Furthermore, the expression “at least one of A, B, and C” should be interpreted to mean that it includes “A alone,” “B alone,” “C alone,” or “any combination of two or more of A, B, and C,” and selectively includes each component, even if the grammatical conjunction “and” is used. Moreover, this definition applies in the same manner even when there are three or more described elements.

[0013] In the present disclosure, the descriptions "A, B, and / or C" are merely abbreviated expressions for the sake of sentence brevity and should be interpreted as if each of the cases of "A alone," "B alone," "C alone," "A and B," "A and C," "B and C," and "A, B, and C as a whole" were actually described in detail. For example, the description that a component includes "A, B, and / or C" should be interpreted as if the component may alternatively include "A," include "B," include "C," include "A and B," include "A and C," include "B and C," or include "A, B, and C."

[0014] In the present disclosure, expressions in the singular form include plural objects unless the context clearly indicates otherwise, and expressions in the plural form are also intended to include singular objects unless the context clearly indicates otherwise. For example, a reference to "an element" includes "one or more elements," and a reference to "elements" may include "one element."

[0016] FIG. 1 is a simplified block diagram of the electronic device of the present invention.

[0017] Referring to FIG. 1, the electronic device (100) may include at least one processor (110), memory (120), camera module (130), and display (140). Embodiments of the present disclosure are not limited thereto. The electronic device (100) may include other components in addition to the components described above.

[0018] At least one processor (110) may be an application processor (AP) implemented as a system-on-chip (SoC) in the electronic device (100), but is not limited thereto. At least one processor (110) may perform operations according to embodiments of the present disclosure by executing instructions stored in memory (120).

[0019] In one embodiment, at least one processor (110) may include a first processor (110a), a second processor (110b), and a third processor (110c).

[0020] The first processor (110a) may be a processor configured to perform calculations related to an artificial intelligence model. For example, the first processor (110a) may be a neural processing unit (NPU). The first processor (110a) may perform object recognition using video data and / or image data captured and acquired through the camera module (130) as input. Through this, the first processor (110a) may identify a subject area (e.g., the first subject area (310) of FIG. 3) within the video data.

[0021] The second processor (110b) may be a processor configured to perform compression and / or decompression of video data. For example, the second processor (110b) may be a video codec. The second processor (110b) may receive video data obtained from a camera module (130) and may combine a mask object (e.g., the mask object (410) of FIG. 4a) corresponding to the subject area (310) within the video data based on information related to the subject area (310) transmitted from the first processor (110a). Subsequently, the second processor (110b) may compress the video data combined with the mask object (410) to create privacy-protected video data and store it in memory (120).

[0022] The third processor (110c) can perform general computation and control functions of the electronic device (100). The third processor (110c) may be a central processing unit (CPU). For example, the third processor (110c) can control operations related to the playback, storage, or transmission of video data to an external electronic device according to user input. The third processor (110c) can control the compression or decompression operations of the second processor (110b).

[0023] Memory (120) includes one or more storage media and can store instructions executed by the processor (110). Memory (120) can store various programs and data executed by at least one processor (110). For example, memory (120) may include volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). Volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). Non-volatile memory may include, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, and embedded multimedia card (EMMC).

[0024] The camera module (130) can capture images of the external environment and generate video data or image data that can be processed within the electronic device (100). The camera module (130) can output the captured images in the form of electrical signals and transmit them to at least one processor (110) or memory (120).

[0025] In one embodiment, the camera module (130) may include a camera (130a) and an image signal processor (ISP) (130b).

[0026] The camera (130a) can generate raw image data by receiving external light and converting it into an electrical signal on an image sensor. For example, the camera (130a) can capture an external image containing a person, an object, or the surrounding environment and output image data in video frame units.

[0027] The image signal processor (130b) can process the raw image data output from the camera (130a) and convert it into video data or image data capable of subsequent processing. For example, the image signal processor (130b) can perform image signal processing such as mosaicking, noise removal, color correction, gamma correction, and / or resolution conversion. The video data processed through the image signal processor (130b) can be transmitted to the first processor (110a) and / or the second processor (110b).

[0028] The display (140) can provide a user interface (UI) of the electronic device (100). The display (140) can provide visual information to the user of the electronic device (100). In one embodiment, the display (140) can display an image, video, or graphic object under the control of the third processor (110c). For example, the display (140) can display a video corresponding to video data stored in memory (120) or display user interface elements related to video playback.

[0030] FIG. 2 illustrates an image corresponding to the first video data acquired through the camera module of FIG. 1. Specifically, the image (200) illustrated in FIG. 2 represents an image corresponding to a video frame at a specific point in time of the first video data for the convenience of explanation. For example, FIG. 2 illustrates an image of a passenger riding inside a vehicle.

[0031] Referring to FIG. 2, the camera module (130) may be positioned facing the interior of the vehicle and configured to photograph the interior of the vehicle. The camera module (130) may be referred to as an interior camera or an interior black box.

[0032] The image (200) obtained by capturing through the camera module (130) may include a plurality of visual objects. The plurality of visual objects may include first to sixth visual objects (210, 220, 230, 240, 250, 260).

[0033] For example, the first visual object (210) may correspond to a passenger seated inside the vehicle. For example, the first visual object (210) may include a first-1 visual object (210a) corresponding to a passenger seated in the driver's seat and a first-2 visual object (210b) corresponding to a passenger seated in the passenger seat. However, the first visual object (210) may include one or three or more visual objects. The first visual object (210) may include privacy data including the passenger's face. The first visual object (210) may be referred to as a privacy object.

[0034] The second visual object (220) may correspond to components inside the vehicle. For example, the second visual object (220) may be an object corresponding to a seat, steering wheel, etc. inside the vehicle. The third visual object (230) may correspond to grass. The fourth visual object (240) may correspond to a road. The fifth visual object (250) may correspond to a tree. The sixth visual object (260) may correspond to the sky. Each of the second to sixth visual objects (220, 230, 240, 250, 260) may be referenced as non-privacy objects.

[0035] After acquiring the first video data through the camera module (130), if the first video data including the first visual object (210) is transmitted as is to an external electronic device, there is a possibility that privacy data may be leaked externally. For example, the external electronic device may be a device belonging to a third party unrelated to the user, or a server accessible via a network. In such cases, privacy data, such as the passenger's face, may be unintentionally exposed externally.

[0036] In addition, if no separate protective measures are taken for the privacy object (e.g., the first visual object (210)) during the process of copying or storing the first video data to an external electronic device, there is a possibility that the privacy data may be played, analyzed, or retransmitted. In the case of video data that includes an individual's private space, such as video footage taken inside a vehicle, there is a potential for violation of personal information protection laws, etc., and thus the need for technology to prevent the privacy object (e.g., the first visual object (210)) from being exposed to the outside arises.

[0037] Accordingly, the present disclosure may disclose a technique for identifying a first visual object (210) corresponding to privacy data among a first video data comprising a plurality of visual objects (210, 220, 230, 240, 250, 260) as illustrated in FIG. 2, and for selectively performing a masking process (e.g., see FIG. 4a) or a deletion process (e.g., see FIG. 4b) on the first visual object (210). Accordingly, when the first video data is transmitted to an external electronic device, leakage of privacy data can be prevented.

[0039] FIG. 3 illustrates an embodiment of performing object recognition of the image of FIG. 2. The description with reference to FIG. 3 may partially overlap with the description with reference to FIG. 2. Therefore, overlapping content may be omitted or simplified.

[0040] The image (300) illustrated in FIG. 3 represents the result of distinguishing and expressing a plurality of visual objects included in the image (200) of FIG. 2. More specifically, the image (300) of FIG. 3 illustrates an embodiment of performing object recognition that identifies the boundaries of each visual object included in the first video data and distinguishes each visual object (210, 220, 230, 240, 250, 260) according to the boundaries. More specifically, the image (300) of FIG. 3 illustrates an embodiment of performing object recognition, such as image segmentation, that identifies the boundaries of each visual object included in the first video data and divides each visual object (210, 220, 230, 240, 250, 260) according to the boundaries.

[0041] Referring to FIG. 3, a first processor (e.g., the first processor (110a) of FIG. 1) can distinguish regions corresponding to a human figure, a vehicle interior structure, and / or a vehicle exterior background included in the first video data into different visual objects. The first processor (110a) can perform object recognition operations on the visual objects (210, 220, 230, 240, 250, 260).

[0042] Each visual object (210, 220, 230, 240, 250, 260) may have a boundary identified so that it can be distinguished from adjacent areas. The boundary may indicate the separation of areas between adjacent visual objects. For example, as illustrated in FIG. 3, a first visual object (210a, 210b) corresponding to a human figure may have a boundary that distinguishes it from the second to fifth visual objects (220, 230, 240, 250, 260) corresponding to a vehicle interior structure or background. That is, the first processor (110a) can perform the object recognition by identifying a boundary that distinguishes the subject area (310) and the background area surrounding at least a part of the subject area (310).

[0043] In one embodiment, the first processor (110a) may define a subject area (310) corresponding to the first visual object (210). For example, the subject area (310) may include a first-1 subject area (310a) corresponding to the first-1 visual object (210a). The subject area (310) may include a first-2 subject area (310b) corresponding to the first-2 visual object (210b). The subject area (310) may be an area corresponding to a privacy object. The subject area (310) may be referred to as a privacy area. For example, the subject area (310) may be an area containing a passenger's face, a part of the body, or a shape capable of identifying an individual. The subject area (310) may be a part of the first video data determined to have captured a person.

[0044] The first processor (110a) can distinguish between areas requiring privacy protection and areas that do not, within the first video data by defining the subject area (310). Accordingly, the first processor (110a) can generate boundary information that distinguishes between the subject area (310) and the non-privacy area. The boundary information may be information having the form of a boundary that distinguishes between the subject area (310) and the background area, and the information transmitted to the second processor (110b) may be configured to represent a mask object having the form of the boundary. The boundary information may be used as information for creating a mask object (e.g., the mask object (410) of FIG. 4a) on the first video data or deleting image data (e.g., the embodiment of FIG. 5).

[0045] According to an embodiment of the present disclosure, by distinguishing visual objects as described above, a visual object corresponding to privacy information and a non-privacy object can be clearly distinguished. For example, a first visual object corresponding to a human figure can be identified as a subject area requiring privacy protection, and visual objects corresponding to an interior vehicle structure or an exterior background can be classified as a non-privacy area.

[0046] The object recognition illustrated in FIG. 3 can be performed by a first processor (110a). The first processor (110a) can transmit information related to the subject area (310) to a second processor (e.g., the second processor (110b) of FIG. 1). The first processor (110a) can transmit information regarding the subject area (310) obtained by performing object recognition to the second processor (110b). Upon receiving the information regarding the subject area (310), the second processor (110b) can perform an operation for the privacy of the first visual object (210). Hereinafter, with reference to FIG. 4a and FIG. 4b, the operation of the second processor (110b) upon receiving the information regarding the subject area (310) will be described.

[0048] FIG. 4a illustrates an image corresponding to second video data generated by combining a mask object with the subject area of ​​the image of FIG. 2. The description with reference to FIG. 4a may partially overlap with the description with reference to FIG. 3. Therefore, overlapping content may be omitted or simplified.

[0049] Referring to FIG. 4a, the second processor (110b) can generate a mask object (410) corresponding to a subject area (310). For example, the second processor (110b) can generate a first-1 mask object (410a) corresponding to a first-1 subject area (310a). The second processor (110b) can generate a first-2 mask object (410b) corresponding to a first-2 subject area (310b).

[0050] In one embodiment, the mask object (410) may be an object intended to make it difficult to visually identify the subject area (310). For example, the mask object (410) may be configured to protect privacy by being placed at a location corresponding to the subject area (310) and by modifying or obscuring the image information of the area. For example, the mask object (410) may be a translucent object for mosaic processing. For example, the mask object (410) may be configured to rearrange a plurality of pixels corresponding to the subject area (310) into a certain block unit or to reduce the resolution so that the detailed shape of the subject is not identifiable. However, the embodiments of the present disclosure are not limited thereto. The mask object (410) may be an object for blurring processing, or a visual object having a solid color or pattern that covers the subject area (310).

[0051] In one embodiment, the second processor (110b) can generate a second video data with privacy protection applied by combining a generated mask object (410) with a subject area (310) on the first video data. Specifically, the second processor (110b) can generate the second video data by combining the generated mask object (410) with the subject area (310) on the first video data and then compressing it. Because the mask object (410) is combined with the subject area (310), the second video data can maintain a protected state so that the privacy information contained in the subject area (310) is not identified from the outside. The second video data may include an image (410) in a specific frame. The image (410) of FIG. 4a may be an image corresponding to a video frame at a specific point in time of the second video data. The image (410) may represent a state in which the mask object (410) is combined with the first visual object (210). By combining a mask object (410) on the first visual object (210), the privacy information of the first visual object (210) can be kept from being exposed.

[0052] In one embodiment, the mask object (410) may be placed on a layer located on a first visual object (210) corresponding to a subject area (310). The mask object (410) may be placed on a layer closer to the user of the electronic device (100) than on the layer of the first video data. Accordingly, the depth value of the mask object (410) may be smaller than the depth value of the first visual object (210). As the mask object (410) is placed on a layer located on a first visual object (210) corresponding to a subject area (310), the mask object (410) may block the first visual object (210), which contains privacy information, from being visually exposed to the outside.

[0053] In one embodiment, the second processor (110b) may store the second video data generated by compressing the video data combined with the mask object (410) in memory (e.g., memory (120) of FIG. 1). The second video data may be video data with privacy protection applied. Meanwhile, the second processor (110b) may generate the third video data by compressing the first video data as a single input without combining the mask object (410). The second processor (110b) may store the third video data in memory (120).

[0054] That is, the second processor (110b) may be configured to selectively generate and store second video data with privacy protection applied and third video data without privacy protection processing applied.

[0056] FIG. 4b illustrates an image corresponding to a fourth video data generated by deleting image data corresponding to the subject area of ​​the image in FIG. 2. The description with reference to FIG. 4b may partially overlap with the description with reference to FIG. 4a. Therefore, overlapping content may be omitted or simplified.

[0057] Referring to FIG. 4b, the second processor (110b) can identify image data corresponding to the subject area (310) within the first video data by using information related to the subject area (310) received from the first processor (110a). For example, the second processor (110b) can identify image data corresponding to the first-1 subject area (310a) and the first-2 subject area (310b), respectively.

[0058] In one embodiment, the second processor (110b) can delete image data corresponding to the identified subject area (310) from the first video data. For example, the second processor (110b) may be configured to delete privacy information from the first video data by removing pixel data corresponding to the first visual object (210). Accordingly, privacy information contained in the first visual object (210) can be completely removed from the video data.

[0059] The second processor (110b) can generate fourth video data by compressing video data from which image data corresponding to the first visual object (210) has been deleted. The fourth video data may represent a state in which privacy information has been removed compared to the first video data. The fourth video data may be stored in memory (e.g., memory (120) of FIG. 1) in a state in which privacy information has been removed. Privacy protection may be maintained even when the fourth video data is transmitted to an external electronic device or stored inside the device.

[0060] The image (420) illustrated in FIG. 4b may be an image corresponding to a video frame at a specific point in time of the fourth video data. The image (420) may represent a state in which image data of an area corresponding to the first visual object (210) has been deleted, and accordingly, the area may not contain visual information that can identify a person's shape or an individual.

[0061] According to an embodiment of the present disclosure, unlike the mask object combination method illustrated in FIG. 4a, the image data deletion method illustrated in FIG. 4b can provide stronger privacy protection by fundamentally removing image data containing privacy information. This deletion-based method may be selectively applied depending on the purpose of image utilization or the level of security requirements.

[0063] Figure 5 is a flowchart showing the operation of the electronic device of Figure 1.

[0064] Referring to FIG. 5, each operation illustrated in FIG. 5 can be controlled by a third processor (110c) included in the electronic device (100). The third processor (110c) is a processor corresponding to a central processing unit (CPU) and can control the operations of the first processor (110a) and the second processor (110b). The third processor (110c) can manage the video data processing flow for privacy protection in its entirety.

[0065] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0066] In operation 510, the third processor (110c) can control the first processor (110a) to receive first video data from the camera module (130). The first video data may be video data captured and acquired through the camera module (130), and may include image data containing a plurality of visual objects (e.g., the first to sixth visual objects (210, 220) of FIG. 2), such as the interior of a vehicle. For example, the first video data may be a set of video frames including a person, a structure inside the vehicle, and a background outside the vehicle.

[0067] In operation 520, the third processor (110c) can control the first processor (110a) to perform object recognition using the first video data. Specifically, the first processor (110a) can identify a subject area (e.g., the subject area (310) of FIG. 3) within the first video data by distinguishing from one another areas corresponding to the shape of a person, the internal structure of a vehicle, and / or the background included in the first video data. The object recognition can be performed using image segmentation or an object recognition method based on an artificial intelligence model. The subject area (310) may be an area corresponding to privacy information and may include, for example, a passenger's face or a part of the body capable of identifying an individual.

[0068] In operation 530, the third processor (110c) can control the first processor (110a) to transmit information related to the subject area (310) from the first processor (110a) to the second processor (110b). The information related to the subject area (310) may include at least one of the location, size, shape, or boundary information of the subject area (310). For example, the information may be data including pixel coordinates occupied by the subject area (310) within the first video data.

[0069] In operation 540, the third processor (110c) can control the second processor (110b) to receive the first video data from the camera module (130). At this time, the second processor (110b) can receive the first video data in parallel and / or independently of the first processor (110a). For example, while the first processor (110a) performs object recognition, the second processor (110b) can perform data processing on the same first video data.

[0070] In operation 550, the third processor (110c) can control the second processor (110b) to receive information related to the subject area (310) transmitted from the first processor (110a). Accordingly, the second processor (110b) can receive the subject area (310) requiring privacy protection within the first video data. The second processor (110b) can be configured to process the subject area (310) and the non-privacy area separately.

[0071] In operation 560, the third processor (110c) can control the second processor (110b) to generate a mask object (410) corresponding to a subject area (310) included in the received information, and to combine the mask object (410) with the first video data. The mask object (410) may be an object configured to make it difficult to visually identify the subject area (310). For example, the mask object (410) may be a visual object including a mosaic effect, a blur effect, or a translucent pattern. The mask object (410) may be positioned to be aligned with a location corresponding to the subject area (310).

[0072] In operation 570, the third processor (110c) may control the second processor (110b) to generate second video data by compressing the first video data combined with the mask object (410). The second video data may be video data with privacy protection applied, and may be configured so that privacy information is not identified even when transmitted to or stored in an external electronic device. The compression may be performed based on a video codec.

[0073] In operation 580, the third processor (110c) can control the second processor (110b) to store the second video data in memory (120). Accordingly, the electronic device (100) can store video data with privacy information protected. The second video data can maintain its privacy protection state even when subsequently played, copied, or transmitted externally.

[0075] Figure 6 is a flowchart showing the operation of an electronic device when a playback input is acquired.

[0076] Referring to FIG. 6, in operation 610, the third processor (110c) can generate second video data by controlling the second processor (110b) to compress first video data combined with a mask object (410). The third processor (110c) can generate third video data by controlling the second processor (110b) to compress first video data in a state where the mask object (410) is not combined alone. The third processor (110c) can control the second processor (110b) to selectively generate and manage video data with privacy protection applied and data corresponding to the original video data. The second video data and the third video data can be stored in memory (120).

[0077] In operation 620, the third processor (110c) may obtain a playback input for displaying the first video data. The playback input may be an input obtained from a user. For example, the playback input may be a touch input, a button input, or a playback request signal from an external electronic device. The playback input may be an input requesting playback of the original image (e.g., the first video data).

[0078] In operation 630, the third processor (110c) can control the second processor (110b) to decompress the third video data based on the acquisition of the playback input. Since the third video data is data corresponding to the original image without the mask object (410) combined, the decompressed video data may include image data in a state where privacy protection processing has not been applied. The second processor (110b) can decompress the third video data using a video codec.

[0079] In operation 640, the third processor (110c) can control the second processor (110b) to display an image corresponding to the decompressed third video data through the display (140). Accordingly, the electronic device (100) can output video frames corresponding to the original image to the user. The image displayed through the display (140) may include a plurality of visual objects (210, 220, 230, 240, 250, 260), such as an image of the interior of a vehicle.

[0080] According to an embodiment of the present disclosure, when a playback input is obtained as shown in FIG. 6, the third processor (110c) may selectively decompress and display a third video data corresponding to the original video, rather than the second video data to which privacy protection is applied. Accordingly, the electronic device (100) can provide a video management structure that allows the original video to be verified during playback while maintaining privacy protection during the storage stage.

[0082] Figure 7 is a flowchart showing the operation of an electronic device when a transmission input is acquired.

[0083] Referring to FIG. 7, in operation 710, the third processor (110c) may control the second processor (110b) to generate second video data by compressing the first video data to which the mask object (410) is combined. Additionally, the third processor (110c) may control the second processor (110b) to generate third video data by compressing the first video data to which the mask object (410) is not combined alone. Operation 710 may be substantially the same as operation 610 of FIG. 6.

[0084] In operation 720, the third processor (110c) may receive a transmission input for transmitting an image corresponding to the first video data to an external electronic device. The transmission input may be an input received from a user. For example, the transmission input may include a request to share a specific video file, an upload request, or a request to transmit to an external electronic device. The external electronic device may be an electronic device of a user or another user, or a server accessible via a network.

[0085] In operation 730, the third processor (110c) can control the transmission of the second video data with privacy protection applied to the external electronic device based on the acquisition of the transmission input. Specifically, the third processor (110c) can control the second processor (110b) or the communication circuit to transmit the second video data to the external electronic device. Accordingly, the privacy information corresponding to the subject area (310) may not be exposed in the video data transmitted to the external electronic device.

[0086] According to an embodiment of the present disclosure, when a transmission input is obtained as shown in FIG. 7, the third processor (110c) can be controlled to selectively transmit a second video data with privacy protection applied, rather than a third video data corresponding to the original image. Accordingly, the electronic device (100) can effectively prevent the leakage of privacy information during the external transmission of video data and can block the transmission of video data capable of personal identification to the outside without user consent.

[0088] FIG. 8 is a flowchart illustrating an embodiment of generating fourth video data by deleting image data of the subject area.

[0089] Referring to FIG. 8, in operation 810, the third processor (110c) can control the second processor (110b) to receive the first video data from the camera (140a). The first video data may be video data captured and acquired through the camera module (130), and may include a plurality of visual objects (210, 220, 230, 240, 250, 260) including a human figure and a background.

[0090] In operation 820, the third processor (110c) can control the second processor (110b) to receive information related to the subject area (310) from the first processor (110a). The information related to the subject area (310) may be information about an area where privacy protection is required, and may include, for example, at least one of location information, size information, shape information, or boundary information of the subject area (310).

[0091] In operation 830, the third processor (110c) can control the second processor (110b) to delete image data of the first video data corresponding to the subject area (310) included in the received information. For example, the second processor (110b) can remove privacy information that can identify an individual from the first video data by removing pixel data corresponding to the first visual object (210).

[0092] In operation 840, the third processor (110c) can control the second processor (110b) to generate a fourth video data by compressing the first video data from which image data has been deleted. The fourth video data may be video data from which image data corresponding to the subject area (310) has been removed, and may correspond to the fourth video data illustrated in FIG. 4b.

[0093] In operation 850, the third processor (110c) can control the second processor (110b) to store the generated fourth video data in memory (120). Accordingly, the electronic device (100) can store video data in a state where privacy information is completely removed from the video data.

[0094] According to an embodiment of the present disclosure, a method of deleting image data corresponding to a subject area (310) as illustrated in FIG. 8 can provide stronger privacy protection by fundamentally removing image data containing privacy information, unlike a method of combining mask objects (410). This deletion-based method can be selectively applied in environments with high security requirements.

[0096] FIG. 9 illustrates an example of a block diagram illustrating an autonomous driving system of a vehicle according to one embodiment.

[0097] Referring to FIG. 9, the autonomous driving system (900) of a vehicle according to FIG. 9 may be a deep learning network comprising sensors (903), an image preprocessor (905), a deep learning network (907), an artificial intelligence (AI) processor (909), a vehicle control module (911), a network interface (913), and a communication unit (915). In various embodiments, each element may be connected through various interfaces. For example, sensor data sensed and output by the sensors (903) may be fed to the image preprocessor (905). Sensor data processed by the image preprocessor (905) may be fed to the deep learning network (907) running in the AI ​​processor (909). The output of the deep learning network (907) running in the AI ​​processor (909) may be fed to the vehicle control module (911). Intermediate results of the deep learning network (907) running in the AI ​​processor (907) can be fed to the AI ​​processor (909). In various embodiments, the network interface (913) communicates with an in-vehicle electronic device (e.g., the electronic device (110) of FIG. 2) to transmit autonomous driving path information and / or autonomous driving control commands for the autonomous driving of the vehicle to internal block components. In one embodiment, the network interface (913) may be used to transmit sensor data acquired through the sensor(s) (903) to an external server. In some embodiments, the autonomous driving control system (900) may include additional or fewer components as appropriate. For example, in some embodiments, the image preprocessor (905) may be an optional component. As another example, a post-processing component (not shown) may be included within the autonomous driving control system (900) to perform post-processing on the output of the deep learning network (907) before the output is provided to the vehicle control module (911).

[0098] In some embodiments, the sensors (903) may include one or more sensors. In various embodiments, the sensors (903) may be attached at different locations on the vehicle. The sensors (903) may face one or more different directions. For example, the sensors (903) may be attached to the front, sides, rear, and / or roof of the vehicle to face directions such as forward-facing, rear-facing, and side-facing. In some embodiments, the sensors (903) may be image sensors, such as high dynamic range cameras. In some embodiments, the sensors (903) include non-visual sensors. In some embodiments, in addition to image sensors, the sensors (903) include radar, light detection and running radar (LiDAR), and / or ultrasonic sensors. In some embodiments, the sensors (903) are not mounted on a vehicle having a vehicle control module (911). For example, the sensors (903) may be included as part of a deep learning system for capturing sensor data and may be attached to an environment or road and / or mounted on surrounding vehicles.

[0099] In some embodiments, an image pre-processor (905) may be used to preprocess sensor data of the sensors (903). For example, the image pre-processor (905) may be used to preprocess sensor data, to split sensor data into one or more components, and / or to post-process one or more components. In some embodiments, the image pre-processor (905) may be a graphics processing unit (GPU), a central processing unit (CPU), an image signal processor, or a specialized image processor. In various embodiments, the image pre-processor (905) may be a tone-mapper processor for processing high dynamic range data. In some embodiments, the image pre-processor (905) may be a component of an AI processor (909).

[0100] In some embodiments, the deep learning network (907) may be a deep learning network for implementing control commands for controlling an autonomous vehicle. For example, the deep learning network (907) may be an artificial neural network, such as a convolutional neural network (CNN), trained using sensor data, and the output of the deep learning network (907) is provided to a vehicle control module (911).

[0101] In some embodiments, the artificial intelligence (AI) processor (909) may be a hardware processor for running a deep learning network (907). In some embodiments, the AI ​​processor (909) is a specialized AI processor for performing inference on sensor data through a convolutional neural network (CNN). In some embodiments, the AI ​​processor (909) may be optimized for the bit depth of the sensor data. In some embodiments, the AI ​​processor (909) may be optimized for deep learning operations, such as operations of the neural network including convolution, dot product, vector and / or matrix operations. In some embodiments, the AI ​​processor (909) may be implemented through a plurality of graphics processing units (GPUs) capable of effectively performing parallel processing.

[0102] In various embodiments, the AI ​​processor (909) may be coupled via an input / output interface to a memory configured to provide an AI processor having instructions that induce deep learning analysis on sensor data received from the sensor(s) (903) while the AI ​​processor (909) is running and to determine the machine learning results used to operate the vehicle at least partially autonomously. In some embodiments, a Vehicle Control Module (911) may be used to process instructions for vehicle control output from the artificial intelligence (AI) processor (909) and to translate the output of the AI ​​processor (909) into instructions for controlling each vehicle module to control various modules of the vehicle. In some embodiments, the Vehicle Control Module (911) is used to control the vehicle for autonomous driving. In some embodiments, the Vehicle Control Module (911) may adjust the steering and / or speed of the vehicle. For example, the vehicle control module (911) may be used to control the driving of the vehicle, such as deceleration, acceleration, steering, lane change, and lane keeping. In some embodiments, the vehicle control module (911) may generate control signals for controlling vehicle lighting, such as brake lights, turn signals, and headlights. In some embodiments, the vehicle control module (911) may be used to control vehicle audio-related systems, such as the vehicle's sound system, vehicle's audio warnings, vehicle's microphone system, and vehicle's horn system.

[0103] In some embodiments, the vehicle control module (911) may be used to control notification systems, including warning systems, to notify passengers and / or the driver of driving events such as approach to an intended destination or a potential collision. In some embodiments, the vehicle control module (911) may be used to control sensors such as the vehicle's sensors (903). For example, the vehicle control module (911) may modify the orientation of the sensors (903), change the output resolution and / or format type of the sensors (903), increase or decrease the capture rate, adjust the dynamic range, and adjust the focus of the camera. Additionally, the vehicle control module (911) may turn the operation of the sensors on or off individually or collectively.

[0104] In some embodiments, the vehicle control module (911) may be used to change the parameters of the image preprocessor (905) in a manner such as modifying the frequency range of the filters, adjusting edge detection parameters for features and / or object detection, or adjusting channels and bit depth. In various embodiments, the vehicle control module (911) may be used to control the autonomous driving and / or driver assistance functions of the vehicle.

[0105] In some embodiments, the network interface (913) may serve as an internal interface between the block components of the autonomous driving control system (900) and the communication unit (915). Specifically, the network interface (913) may be a communication interface for receiving and / or sending data including voice data. In various embodiments, the network interface (913) may be connected to external servers to connect voice calls or receive and / or send text messages through the communication unit (915), transmit sensor data, update the software of the vehicle to the autonomous driving system, or update the software of the vehicle's autonomous driving system.

[0106] In various embodiments, the communication unit (915) may include various wireless interfaces such as cellular or WiFi. For example, a network interface (913) may be used to receive updates regarding operating parameters and / or commands for sensors (903), an image preprocessor (905), a deep learning network (907), an AI processor (909), and a vehicle control module (911) from an external server connected via the communication unit (915). For example, a machine learning model of the deep learning network (907) may be updated using the communication unit (915). According to another example, the communication unit (915) may be used to update operating parameters of the image preprocessor (905), such as image processing parameters, and / or firmware of the sensors (903).

[0107] In another embodiment, the communication unit (915) may be used to enable communication for emergency services and emergency contact in the event of an accident or near-accident. For example, in a collision event, the communication unit (915) may be used to call emergency services for assistance and to notify emergency services of the collision details and the location of the vehicle externally. In various embodiments, the communication unit (915) may update or obtain the estimated time of arrival and / or destination location.

[0108] According to one embodiment, the autonomous driving system (900) illustrated in FIG. 15 may be composed of an electronic device (110) of a vehicle. According to one embodiment, the AI ​​processor (909) of the autonomous driving system (900) can control the vehicle's autonomous driving software to learn by controlling the input of information related to the autonomous driving release event as training set data of a deep learning network when an autonomous driving release event occurs from a user during the vehicle's autonomous driving.

[0110] FIGS. 10 and 11 illustrate an example of a block diagram representing an autonomous driving vehicle according to one embodiment. FIG. 12 illustrates an example of a gateway associated with a user device according to various embodiments.

[0111] Referring to FIG. 10, the autonomous driving vehicle (1000) according to the present embodiment may include a control device (1100), a sensing module (1004a, 1004b, 1004c, 1004d), an engine (1006), and a user interface (1008).

[0112] The autonomous driving vehicle (1000) may be equipped with an autonomous driving mode or a manual mode. For example, it may switch from a manual mode to an autonomous driving mode or switch from an autonomous driving mode to a manual mode depending on user input received through a user interface (1008).

[0113] When the vehicle (1000) is operated in autonomous driving mode, the autonomous vehicle (1000) can be operated under the control of the control device (1100).

[0114] In this embodiment, the control device (1100) may include a controller (1120) including a memory (1122) and a processor (1124), a sensor (1110), a communication device (1130), and an object detection device (1140).

[0115] Here, the object detection device (1140) can perform all or part of the functions of the distance measuring device.

[0116] That is, in this embodiment, the object detection device (1140) is a device for detecting an object located outside the moving body (1000), and the object detection device (1140) can detect an object located outside the moving body (1000) and generate object information based on the detection result.

[0117] Object information may include information on the existence or non-existence of an object, location information of the object, distance information between the moving body and the object, and relative velocity information between the moving body and the object.

[0118] The object may include various objects located outside the moving body (1000), such as lanes, other vehicles, pedestrians, traffic signals, light, roads, structures, speed bumps, terrain features, animals, etc. Here, the traffic signal may be a concept including traffic lights, traffic signs, patterns or text drawn on the road surface. And, the light may be light generated from a lamp equipped on another vehicle, light generated from a street light, or sunlight.

[0119] In addition, structures may be objects located near roads and fixed to the ground. For example, structures may include streetlights, street trees, buildings, utility poles, traffic lights, and bridges. Landmarks may include mountains, hills, etc.

[0120] Such an object detection device (1140) may include a camera module. The controller (1120) may extract object information from an external image captured by the camera module and allow the controller (1120) to process the information.

[0121] Additionally, the object detection device (1140) may further include imaging devices for recognizing the external environment. In addition to LIDAR, RADAR, GPS devices, odometers, and other computer vision devices, ultrasonic sensors, and infrared sensors may be used, and these devices may be selected or operated simultaneously as needed to enable more precise detection.

[0122] Meanwhile, a distance measuring device according to one embodiment of the present invention can calculate the distance between an autonomous driving vehicle (1000) and an object, and control the operation of the vehicle based on the calculated distance in conjunction with a control device (1100) of the autonomous driving vehicle (1000).

[0123] For example, if there is a possibility of collision depending on the distance between the autonomous vehicle (1000) and the object, the autonomous vehicle (1000) may control the brakes to reduce speed or stop. As another example, if the object is a moving object, the autonomous vehicle (1000) may control the driving speed of the autonomous vehicle (1000) to maintain a predetermined distance from the object.

[0124] A distance measuring device according to one embodiment of the present invention may be configured as a module within a control device (1100) of an autonomous vehicle (1000). That is, the memory (1122) and processor (1124) of the control device (1100) may be configured to implement the collision prevention method according to the present invention in software.

[0125] Additionally, the sensor (1110) can acquire various sensing information by connecting the internal / external environment of the moving body to the sensing module (1004a, 1004b, 1004c, 1004d). Here, the sensor (1110) may include an attitude sensor (e.g., yaw sensor, roll sensor, pitch sensor, collision sensor, wheel sensor, speed sensor, inclination sensor, weight sensing sensor, heading sensor, gyro sensor, position module, moving body forward / backward sensor, battery sensor, fuel sensor, tire sensor, steering sensor based on handle rotation, internal temperature sensor of the moving body, internal humidity sensor of the moving body, ultrasonic sensor, illuminance sensor, accelerator pedal position sensor, brake pedal position sensor, etc.

[0126] Accordingly, the sensor (1110) can acquire sensing signals regarding the attitude information of the moving body, collision information of the moving body, direction information of the moving body, position information of the moving body (GPS information), angle information of the moving body, speed information of the moving body, acceleration information of the moving body, tilt information of the moving body, forward / backward information of the moving body, battery information, fuel information, tire information, lamp information of the moving body, internal temperature information of the moving body, internal humidity information of the moving body, steering wheel rotation angle, external illumination of the moving body, pressure applied to the accelerator pedal, pressure applied to the brake pedal, etc.

[0127] Additionally, the sensor (1110) may further include an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake air temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), etc.

[0128] In this way, the sensor (1110) can generate moving body state information based on the sensing data.

[0129] A wireless communication device (1130) is configured to implement wireless communication between autonomous vehicles (1000). For example, the autonomous vehicle (1000) is enabled to communicate with a user's mobile phone, or other wireless communication device (1130), other vehicles, a central device (traffic control device), a server, etc. The wireless communication device (1130) can transmit and receive wireless signals according to a wireless protocol. The wireless communication protocol may be Wi-Fi, Bluetooth, Long-Term Evolution (LTE), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), or Global Systems for Mobile Communications (GSM), but the communication protocol is not limited thereto.

[0130] In addition, in this embodiment, the autonomous vehicle (1000) can also implement communication between vehicles through a wireless communication device (1130). That is, the wireless communication device (1130) can perform vehicle-to-vehicle communication with other vehicles on the road and other vehicles. The autonomous vehicle (1000) can transmit and receive information such as driving warnings and traffic information through vehicle-to-vehicle communication, and can also request information from or receive requests from other vehicles. For example, the wireless communication device (1130) can perform V2V communication using a dedicated short-range communication (DSRC) device or a cellular V2V (C-V2V) device. In addition, in addition to communication between vehicles, communication between a vehicle and other objects (e.g., electronic devices carried by pedestrians) (V2X, Vehicle to Everything communication) can also be implemented through the wireless communication device (1130).

[0131] Additionally, the wireless communication device (1130) can obtain information generated from various mobilities, including infrastructure located on the road (traffic lights, CCTV, RSU, eNode B, etc.) or other autonomous driving / non-autonomous driving vehicles, through a non-terrestrial network rather than a terrestrial network, as information for autonomous driving of the autonomous driving vehicle (1000).

[0132] For example, the wireless communication device (1130) can perform wireless communication through a non-terrestrial network dedicated antenna mounted on an autonomous vehicle (1000) with a low Earth orbit (LEO) satellite system, a medium Earth orbit (MEO) satellite system, a geostationary orbit (GEO) satellite system, a high altitude platform (HAP) system, etc., which constitute non-terrestrial networks.

[0133] For example, the wireless communication device (1130) can perform wireless communication with various platforms constituting the NTN according to the wireless access specifications according to the 5G NR NTN (5TH Generation New Radio Non-Terrestrial Network) standard specifications currently being discussed by 3GPP, etc., but is not limited thereto.

[0134] In this embodiment, the controller (1120) can select a platform capable of appropriately performing NTN communication by considering various information such as the location of the autonomous vehicle (1000), the current time, and available power, and can control the wireless communication device (1130) to perform wireless communication with the selected platform.

[0135] In this embodiment, the controller (1120) is a unit that controls the overall operation of each unit within the vehicle (1000), and may be configured by the manufacturer of the vehicle at the time of manufacturing or additionally configured after manufacturing to perform autonomous driving functions. Alternatively, configurations for performing continuous additional functions may be included through an upgrade of the controller (1120) configured at the time of manufacturing. Such a controller (1120) may also be named an ECU (Electronic Control Unit).

[0136] The controller (1120) can collect various data from connected sensors (1110), object detection devices (1140), communication devices (1130), etc., and based on the collected data, transmit control signals to other components within the moving body, such as sensors (1110), engines (1006), user interfaces (1008), communication devices (1130), and object detection devices (1140). Additionally, although not illustrated, control signals can also be transmitted to acceleration devices, braking systems, steering devices, or navigation devices related to the driving of the moving body.

[0137] In this embodiment, the controller (1120) can control the engine (1006), for example, by detecting the speed limit of the road on which the autonomous vehicle (1000) is driving and controlling the engine (1006) so that the driving speed does not exceed the speed limit, or by controlling the engine (1006) to accelerate the driving speed of the autonomous vehicle (1000) within a range that does not exceed the speed limit.

[0138] Additionally, the controller (1120) can determine whether the approach to or departure from the lane is due to normal driving conditions or other driving conditions when the autonomous vehicle (1000) approaches or departs from the lane while driving, and can control the engine (1006) to control the driving of the vehicle according to the result of the determination. Specifically, the autonomous vehicle (1000) can detect lanes formed on both sides of the lane in which the vehicle is driving. In this case, the controller (1120) determines whether the autonomous vehicle (1000) approaches or departs from the lane, and if it is determined that the autonomous vehicle (1000) approaches or departs from the lane, it can determine whether such driving is due to normal driving conditions or other driving conditions. Here, as an example of a normal driving condition, it may be a situation where the vehicle needs to change lanes. And, as an example of other driving conditions, it may be a situation where the vehicle does not need to change lanes. If the controller (1120) determines that the autonomous vehicle (1000) is approaching a lane or deviating from a lane in a situation where a lane change of the vehicle is not necessary, the controller can control the driving of the autonomous vehicle (1000) so that the autonomous vehicle (1000) drives normally in the vehicle without deviating from the lane.

[0139] If there is another vehicle or obstacle in front of the vehicle, the engine (1006) or braking system can be controlled to decelerate the vehicle, and in addition to speed, the trajectory, driving path, and steering angle can be controlled. Alternatively, the controller (1120) can control the driving of the vehicle by generating necessary control signals based on recognition information of the vehicle's driving lane, driving signal, and other external environment.

[0140] In addition to generating its own control signals, the controller (1120) can also control the movement of the mobile body by communicating with a surrounding mobile body or a central server and transmitting commands to control surrounding devices through the received information.

[0141] Additionally, since accurate recognition of a moving object or lane according to the present embodiment may be difficult if the position or angle of view of the camera module (1150) is changed, the controller (1120) may generate a control signal to control the camera module (1150) to perform calibration in order to prevent this. Accordingly, in the present embodiment, by generating a calibration control signal to the camera module (1150), the controller (1120) can continuously maintain the normal mounting position, direction, angle of view, etc. of the camera module (1150) even if the mounting position of the camera module (1150) is changed due to vibrations or shocks caused by the movement of the autonomous moving object (1000). The controller (1120) can generate a control signal to perform calibration of the camera module (1120) when the initial mounting position, direction, and angle of view information of the camera module (1120) measured during driving of the autonomous vehicle (1000) differs by more than a threshold value from the initial mounting position, direction, and angle of view information of the camera module (1120) that is stored in advance.

[0142] In this embodiment, the controller (1120) may include a memory (1122) and a processor (1124). The processor (1124) may execute software stored in the memory (1122) according to a control signal of the controller (1120). Specifically, the controller (1120) stores data and commands for performing the lane detection method according to the present invention in the memory (1122), and the commands may be executed by the processor (1124) to implement one or more methods disclosed herein.

[0143] At this time, the memory (1122) may be stored on a recording medium executable on a non-volatile processor (1124). The memory (1122) may store software and data through appropriate internal and external devices. The memory (1122) may consist of a RAM (random access memory), ROM (read only memory), a hard disk, and a memory (1122) device connected to a dongle.

[0144] Memory (1122) can store at least an operating system (OS), user applications, and executable commands. Memory (1122) can also store application data and array data structures.

[0145] The processor (1124) can be a controller, microcontroller, or state machine, or a microprocessor or a suitable electronic processor.

[0146] The processor (1124) may be implemented as a combination of computing devices, and the computing device may be a digital signal processor, a microprocessor, or a suitable combination thereof.

[0147] Meanwhile, the autonomous vehicle (1000) may further include a user interface (1008) for user input to the control device (1100) described above. The user interface (1008) may allow the user to input information through appropriate interaction. For example, it may be implemented as a touchscreen, keypad, operation buttons, etc. The user interface (1008) transmits input or commands to the controller (1120), and the controller (1120) may perform control operations of the vehicle in response to the input or commands.

[0148] Additionally, the user interface (1008) can communicate with the autonomous vehicle (1000) via a wireless communication device (1130) to a device outside the autonomous vehicle (1000). For example, the user interface (1008) can be linked with a mobile phone, tablet, or other computer device.

[0149] Furthermore, although the autonomous vehicle (1000) in this embodiment has been described as including an engine (1006), it is also possible to include other types of propulsion systems. For example, the vehicle may be operated by electric energy, hydrogen energy, or a hybrid system combining these. Accordingly, the controller (1120) may include a propulsion mechanism according to the propulsion system of the autonomous vehicle (1000) and provide control signals to the components of each propulsion mechanism.

[0150] Hereinafter, the detailed configuration of the control device (1100) according to the present invention according to the present embodiment will be described in more detail with reference to FIG. 17.

[0151] The control unit (1100) includes a processor (1124). The processor (1124) may be a general-purpose single or multi-chip microprocessor, a dedicated microprocessor, a microcontroller, a programmable gate array, etc. The processor may also be referred to as a central processing unit (CPU). In addition, in this embodiment, the processor (1124) may be used as a combination of multiple processors.

[0152] The control unit (1100) also includes a memory (1122). The memory (1122) may be any electronic component capable of storing electronic information. The memory (1122) may also include a combination of memories (1122) in addition to a single memory.

[0153] Data and instructions (1122a) for performing the distance measurement method of the distance measuring device according to the present invention may be stored in memory (1122). When the processor (1124) executes the instructions (1122a), all or part of the instructions (1122a) and the data (1122b) required for the execution of the instructions may be loaded (1124a, 1124b) onto the processor (1124).

[0154] The control device (1100) may include a transmitter (1130a), a receiver (1130b), or a transceiver (1130c) for allowing the transmission and reception of signals. One or more antennas (1132a, 1132b) may be electrically connected to the transmitter (1130a), the receiver (1130b), or each transceiver (1130c) and may additionally include antennas.

[0155] The control unit (1100) may include a digital signal processor (DSP) (1170). Through the DSP (1170), the mobile body can be enabled to process digital signals quickly.

[0156] The control device (1100) may include a communication interface (1180). The communication interface (1180) may include one or more ports and / or communication modules for connecting other devices to the control device (1100). The communication interface (1180) may enable the user and the control device (1100) to interact.

[0157] Various components of the control device (1100) may be connected together by one or more buses (1190), and the buses (1190) may include power buses, control signal buses, status signal buses, data buses, etc. Under the control of the processor (1124), the components may transmit information to each other through the buses (1190) and perform intended functions.

[0158] Meanwhile, in various embodiments, the control device (1100) may be associated with a gateway for communication with a security cloud. For example, referring to FIG. 12, the control device (1100) may be associated with a gateway (1205) for providing information obtained from at least one of the components (1001 to 1004) of the vehicle (1200) to the security cloud (1206). For example, the gateway (1205) may be included within the control device (1100). In another example, the gateway (1205) may be configured as a separate device within the vehicle (1200) distinct from the control device (1100). The gateway (1205) enables communication between a software management cloud (1209) having a different network, a security cloud (1206), and a network within the vehicle (1200) secured by in-vehicle security software (1210).

[0159] For example, the component (1201) may be a sensor. For example, the sensor may be used to obtain information about at least one of the state of the vehicle (1200) or the state around the vehicle (1200). For example, the component (1201) may include a sensor 910.

[0160] For example, the components (1202) may be electronic control units (ECUs). For example, the ECUs may be used for engine control, transmission control, airbag control, and tire pressure management.

[0161] For example, the component (1203) may be an instrument cluster. For example, the instrument cluster may refer to a panel located in front of the driver's seat on the dashboard. For example, the instrument cluster may be configured to display information necessary for driving to the driver (or passenger). For example, the instrument cluster may be used to display at least one of visual elements for indicating the engine's revolutions per minute (RPM, revolutions per minute or rotate per minute), visual elements for indicating the speed of the vehicle (1200), visual elements for indicating the remaining fuel amount, visual elements for indicating the gear status, or visual elements for indicating information obtained through the component (1201).

[0162] For example, the component (1204) may be a telematics device. For example, the telematics device may refer to a device that combines wireless communication technology and GPS (global positioning system) technology to provide various mobile communication services, such as location information and safe driving, within the vehicle (1200). For example, the telematics device may be used to connect the vehicle (1200) with the driver, a cloud (e.g., security cloud (1206)), and / or the surrounding environment. For example, the telematics device may be configured to support high bandwidth and low latency for 5G NR standard technologies (e.g., 5G NR V2X technology, 5G NR NTN (Non-Terrestrial Network) technology). For example, the telematics device may be configured to support autonomous driving of the vehicle (1200).

[0163] For example, a gateway (1205) may be used to connect the network within the vehicle (1200) with the external network, which is a software management cloud (1209) and a security cloud (1206). For example, the software management cloud (1209) may be used to update or manage at least one software required for driving and managing the vehicle (1200). For example, the software management cloud (1209) may be linked with in-car security software 1010 installed in the vehicle. For example, the in-car security software (1210) may be used to provide security functions within the vehicle (1200). For example, the in-car security software (1210) may encrypt data transmitted and received through the in-car network using an encryption key obtained from an external authorized server for the encryption of the in-car network. In various embodiments, the encryption key used by the in-vehicle security software (1210) may be generated in correspondence with vehicle identification information (vehicle license plate, vehicle VIN (vehicle identification number)) or information uniquely assigned to each user (e.g., user identification information).

[0164] In various embodiments, the gateway (1205) can transmit data encrypted by the in-vehicle security software (1210) based on the encryption key to the software management cloud (1209) and / or security cloud (1206). The software management cloud (1209) and / or security cloud (1206) can identify which vehicle or user the data was received from by decrypting the data encrypted by the encryption key of the in-vehicle security software (1210) using a decryption key capable of decrypting the data. For example, since the decryption key is a unique key corresponding to the encryption key, the software management cloud (1209) and / or security cloud (1206) can identify the sender of the data (e.g., the vehicle or the user) based on the data decrypted through the decryption key.

[0165] For example, the gateway (1205) may be configured to support in-vehicle security software (1210) and may be associated with the control unit (1100). For example, the gateway (1205) may be associated with the control unit (1100) to support a connection between the control unit (1100) and a client device 1007 connected to the security cloud (1206). As another example, the gateway (1205) may be associated with the control unit (1100) to support a connection between the control unit (1100) and a third-party cloud (1208) connected to the security cloud (1206). However, it is not limited thereto.

[0166] In various embodiments, the gateway (1205) may be used to connect the vehicle (1200) with a software management cloud (1209) for managing the operating software of the vehicle (1200). For example, the software management cloud (1209) may monitor whether an update to the operating software of the vehicle (1200) is required, and may provide data to update the operating software of the vehicle (1200) through the gateway (1205) based on monitoring that an update to the operating software of the vehicle (1200) is required. In another example, the software management cloud (1209) may receive a user request from the vehicle (1200) through the gateway (1205) for an update to the operating software of the vehicle (1200), and may provide data to update the operating software of the vehicle (1200) based on said reception. However, it is not limited thereto.

[0168] FIG. 13 is a diagram illustrating the operation of an electronic device for training a neural network based on a set of training data according to one embodiment.

[0169] The operation described with reference to FIG. 13 can be performed by the aforementioned electronic device (e.g., the electronic device (110) of FIG. 2).

[0170] Referring to FIG. 13, in operation 1302, according to one embodiment, an electronic device may acquire a set of training data. The electronic device may acquire a set of training data for supervised learning. The training data may include input data and a pair of ground truth data corresponding to said input data. The ground truth data may represent output data to be acquired from a neural network that receives input data which is the pair of ground truth data. The ground truth data may be acquired by the aforementioned electronic device.

[0171] For example, when training a neural network for image recognition, the training data may include information about an image and one or more subjects included in the image. The information may include a classification (category or class) of the subjects identifiable through the image. The information may include the position, width, height, and / or size of a visual object corresponding to the subject within the image. A set of training data identified through operation 1302 may include pairs of multiple training data. In the above example of training a neural network for image recognition, a set of training data identified by an electronic device may include multiple images and basis truth data corresponding to each of the multiple images.

[0172] Referring to FIG. 13, in operation 1304, an electronic device according to one embodiment can perform training on a neural network based on a set of training data. In one embodiment where the neural network is trained based on supervised learning, the electronic device can input input data included in the training data into an input layer of the neural network. An example of a neural network including the input layer is described with reference to FIG. 15. From the output layer of the neural network that receives the input data through the input layer, the electronic device can obtain output data of the neural network corresponding to the input data.

[0173] In one embodiment, training of operation 1304 may be performed based on the difference between the output data and the basis truth data included in the training data and corresponding to the input data. For example, the electronic device may adjust one or more parameters related to the neural network based on a gradient descent algorithm so as to reduce the difference. The operation of the electronic device to adjust the one or more parameters may be referred to as tuning of the neural network. The electronic device may perform tuning of the neural network based on the output data using a function defined to evaluate the performance of the neural network, such as a cost function. The difference between the output data and the basis truth data described above may be included as an example of the cost function.

[0174] For reference, in operation 1306, according to one embodiment, an electronic device can identify whether valid output data is output from a neural network trained by operation 1304. That the output data is valid may mean that the difference (or cost function) between the output data and the basis truth data satisfies the conditions set for the use of the neural network. For example, if the average value and / or maximum value of the difference between the output data and the basis truth data is less than or equal to a specified threshold value, the electronic device may determine that valid output data is output from the neural network.

[0175] If valid output data is not output from the neural network (1306-No), the electronic device may repeatedly perform training of the neural network based on operation 1304. The embodiments are not limited thereto, and the electronic device may repeatedly perform operations (1302, 1304).

[0176] With valid output data obtained from the neural network (1306-e), based on operation 1308, an electronic device according to one embodiment may use the trained neural network. For example, the electronic device may input other input data to the neural network that is distinct from the input data that was input to the neural network as training data. The output data obtained from the neural network that received the other input data may be used by the electronic device as the result of performing inference on the other input data based on the neural network.

[0178] FIG. 14 is a block diagram of an electronic device according to one embodiment.

[0179] The electronic device (110) of FIG. 14 may include the aforementioned electronic device.

[0180] For example, the operation described with reference to FIG. 13 can be performed by the electronic device (110) of FIG. 14 and / or the processor (1410) of FIG. 14.

[0181] Referring to FIG. 14, the processor (1410) of the electronic device (110) can perform computations related to the neural network (1430) stored in memory (1420). The processor (1410) may include at least one of a CPU (center processing unit), a GPU (graphic processing unit), or a NPU (neural processing unit). The NPU may be implemented as a chip separate from the CPU, or may be integrated into the same chip as the CPU in the form of a system on a chip (SoC). An NPU integrated into the CPU may be referred to as a neural core and / or an artificial intelligence (AI) accelerator.

[0182] Referring to FIG. 14, the processor (1410) can identify a neural network (1430) stored in memory (1420). The neural network (1430) may include an input layer (1432), one or more hidden layers (1434) (or a combination of intermediate layers and output layers (1436). The layers described above (e.g., input layer (1432), one or more hidden layers (1434), and output layers (1436)) may include a plurality of nodes. The number of hidden layers (1434) may vary depending on the embodiment, and a neural network (1430) including a plurality of hidden layers (1434) may be referred to as a deep neural network. The operation of training the deep neural network may be referred to as deep learning.

[0183] In one embodiment, when the neural network (1430) has the structure of a feed-forward neural network, a first node included in a specific layer may be connected to all of the second nodes included in another layer prior to the specific layer. In memory (1420), parameters stored for the neural network (1430) may include weights assigned to the connections between the second nodes and the first node. In the neural network (1430) having the structure of a feed-forward neural network, the value of the first node may correspond to a weighted sum of values ​​assigned to the second nodes, based on the weights assigned to the connections connecting the second nodes and the first node.

[0184] In one embodiment, when the neural network (1430) has the structure of a convolutional neural network, a first node included in a specific layer may correspond to a weighted sum of some of the second nodes included in another layer prior to the specific layer. Some of the second nodes corresponding to the first node may be identified by a filter corresponding to the specific layer. In memory (1420), parameters stored for the neural network (1430) may include weights representing the filter. The filter may include one or more nodes among the second nodes to be used to calculate the weighted sum of the first node, and weights corresponding to each of the one or more nodes.

[0185] According to one embodiment, a processor (1410) of an electronic device (110) can perform training on a neural network (1430) using a training data set (1440) stored in memory (1420). Based on the training data set (1440), the processor (1410) can perform the operation described with reference to FIG. 13 to adjust one or more parameters stored in memory (1420) for the neural network (1430).

[0186] According to one embodiment, a processor (1410) of an electronic device (110) can perform object detection, object recognition, and / or object classification using a neural network (1430) trained based on a learning data set (1440). The processor (1410) can input an image (or video) acquired through a camera (1450) into an input layer (1432) of the neural network (1430). Based on the input layer (1432) into which the image is input, the processor (1410) can sequentially acquire the values ​​of the nodes of the layers included in the neural network (1430) to acquire a set of values ​​of the nodes of the output layer (1436) (e.g., output data). The output data can be used as a result of inferring information contained in the image using the neural network (1430). The embodiments are not limited thereto, and the processor (1410) may input an image (or video) obtained from an external electronic device connected to the electronic device (110) through a communication circuit (1460) into the neural network (1430).

[0187] In one embodiment, a neural network (1430) trained to process an image may be used to identify a region corresponding to a subject within the image (object detection) and / or to identify a class of the subject represented within the image (object recognition and / or object classification). For example, an electronic device (110) may use the neural network (1430) to segment a region corresponding to the subject within the image based on the shape of a rectangle, such as a bounding box. For example, an electronic device (110) may use the neural network (1430) to identify at least one class that matches the subject among a plurality of designated classes.

[0188] For example, the electronic device may be configured to determine a reference position of the second vehicle spaced apart from the first vehicle by a reference distance, and to determine the direction in which the camera is facing using the reference position. For example, the electronic device may be configured to obtain a plurality of first values ​​representing the size of the second vehicle using the plurality of images obtained using the camera, and to identify whether a range including second values ​​that are part of the plurality of first values ​​and are continuously obtained while the plurality of first values ​​are obtained is maintained until a threshold time elapses, and based on the range maintained until the threshold time elapses: to determine a reference position of the second vehicle spaced apart from the first vehicle by a reference distance using position values ​​of the second vehicle each associated with the second values; and to determine the direction in which the camera is facing using the reference position.

[0189] The electronic device as described above may include a memory for storing instructions. The electronic device may include a camera. The electronic device may include at least one processor. The at least one processor may be configured to acquire a plurality of images using the camera while the relative distance between the first vehicle including the electronic device and the second vehicle other than the first vehicle changes, acquire a plurality of first values ​​representing the size of the second vehicle using the plurality of images acquired using the camera, identify whether a range including second values ​​which are part of the plurality of first values ​​and are continuously acquired while the plurality of first values ​​are acquired is maintained until a threshold time elapses, determine a reference position of the second vehicle spaced apart from the first vehicle by a reference distance using position values ​​of the second vehicle each associated with the second values ​​based on the range maintained until the threshold time elapses, and determine the direction in which the camera is facing using the reference position.

[0190] According to one embodiment, the electronic device may be configured to determine a reference position of the second vehicle spaced apart from the first vehicle by a reference distance, and to determine the direction in which the camera is facing using the reference position. For example, the electronic device may determine whether a range including second values, which are part of the first values, is maintained until a threshold time elapses, by using a plurality of images obtained using the camera, and while the plurality of first values ​​are obtained, a range is continuously obtained and includes second values, which are part of the plurality of first values, is maintained until a threshold time elapses; and based on the range maintained until the threshold time elapses: determine a reference position of the second vehicle spaced apart from the first vehicle by a reference distance using position values ​​of the second vehicle, each associated with the second values; and to determine the direction in which the camera is facing using the reference position.

[0192] An electronic device is described. The electronic device may include a camera. The electronic device may include a memory. The electronic device may include a first processor configured to perform calculations related to an artificial intelligence model. The electronic device may include a second processor configured to compress first video data acquired from the camera. The first processor may be configured to receive the first video data from the camera, perform object recognition using the first video data to identify a subject area within the first video data, and transmit information related to the subject area to the second processor. The second processor may be configured to receive the first video data from the camera, receive the information from the first processor, combine a mask object corresponding to the subject area indicated by the information with the first video data, compress the first video data combined with the mask object to generate second video data, and store the second video data in the memory.

[0193] For example, the second processor may be configured to generate third video data by compressing the first video data alone and to store the third video data in the memory.

[0194] For example, the electronic device may further include a display configured to display an image and a third processor, which is a CPU (central processing unit).

[0195] For example, the third processor may be configured to acquire a playback input for displaying video data, control the second processor to decompress the third video data based on acquiring the playback input, and display an image corresponding to the decompressed third video data through the display.

[0196] For example, the third processor may be configured to acquire a transmission input for transmitting an image corresponding to the first video data to an external electronic device, and based on acquiring the transmission input, to transmit the second video data to the external electronic device.

[0197] For example, the second processor may be configured to delete image data of the first video data corresponding to the subject area, compress the first video data from which the image data has been deleted to generate a fourth video data, and store the fourth video data in the memory.

[0198] For example, the mask object may be placed on a layer located on the first video data.

[0199] For example, the subject area may be a part of the first video data in which a person is determined to have been captured.

[0200] For example, the first processor performs object recognition by identifying a boundary that distinguishes the subject area and a background area surrounding at least a part of the subject area, and the information transmitted to the second processor may be configured to represent the mask object having the shape of the boundary.

[0201] A method is described. The method may be executed by an electronic device comprising a camera, a first processor configured to perform calculations related to an artificial intelligence model, and a second processor configured to compress first video data acquired from the camera. The method may include controlling the first processor to receive the first video data from the camera, performing object recognition using the first video data to identify a subject area within the first video data, and transmitting information related to the subject area to the second processor, and controlling the second processor to receive the first video data from the camera, receiving the information from the first processor, combining a mask object corresponding to the subject area indicated by the information with the first video data, compressing the first video data combined with the mask object to generate second video data, and storing the second video data in the memory.

[0202] For example, the above method may further include the operation of generating a third video data by controlling the second processor to compress the first video data alone, and the operation of storing the third video data in the memory.

[0203] For example, the electronic device may further include a display configured to display an image and a third processor, which is a CPU (central processing unit).

[0204] For example, the above method may further include the operation of controlling the third processor to obtain a playback input for displaying video data, the operation of controlling the second processor to decompress the third video data based on obtaining the playback input, and the operation of displaying an image corresponding to the decompressed third video data through the display.

[0205] For example, the above method may further include the operation of controlling the third processor to obtain a transmission input for transmitting an image corresponding to the first video data to an external electronic device, and the operation of transmitting the second video data to the external electronic device based on obtaining the transmission input.

[0206] For example, the above method may further include the operation of controlling the second processor to delete image data of the first video data corresponding to the subject area, the operation of compressing the first video data from which the image data has been deleted to generate a fourth video data, and the operation of storing the fourth video data in the memory.

[0207] For example, the mask object may be placed on a layer located on the first video data.

[0208] For example, the subject area may be a part of the first video data in which a person is determined to have been captured.

[0209] For example, the above method includes an operation of performing object recognition by controlling the first processor to identify a boundary that distinguishes the subject area and a background area surrounding at least a part of the subject area, and the information transmitted to the second processor may further include an operation of representing the mask object having the shape of the boundary.

[0210] A non-transient computer-readable storage medium is described. The non-transient computer-readable storage medium may store one or more programs. The one or more programs may be executed by an electronic device comprising a camera, a first processor configured to perform calculations related to an artificial intelligence model, and a second processor configured to compress first video data acquired from the camera. The above one or more programs may include instructions such as causing the electronic device to control the first processor to receive the first video data from the camera, perform object recognition using the first video data to identify a subject area within the first video data, and transmit information related to the subject area to the second processor, and to control the second processor to receive the first video data from the camera, receive the information from the first processor, combine a mask object corresponding to the subject area indicated by the information with the first video data, compress the first video data combined with the mask object to generate a second video data, and store the second video data in the memory.

[0211] For example, the above one or more programs may include instructions that cause the electronic device to control the second processor to generate third video data by compressing the first video data alone and to store the third video data in the memory.

[0213] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.

Claims

Claim 1 An electronic device comprising: a camera; a memory; a first processor configured to perform calculations related to an artificial intelligence model; and a second processor configured to compress first video data acquired from the camera, wherein the first processor: receives the first video data from the camera and performs object recognition using the first video data to identify a subject area within the first video data, and transmits information related to the subject area to the second processor, and the second processor: receives the first video data from the camera and receives the information from the first processor, combines a mask object corresponding to the subject area indicated by the information with the first video data, compresses the first video data combined with the mask object to generate second video data, and stores the second video data in the memory. Claim 2 An electronic device according to claim 1, wherein the second processor is configured to generate third video data by compressing the first video data alone and to store the third video data in the memory. Claim 3 The electronic device of claim 2, further comprising: a display configured to display an image; and a third processor which is a central processing unit (CPU). Claim 4 An electronic device according to claim 3, wherein the third processor acquires a playback input for displaying video data, controls the second processor to decompress the third video data based on acquiring the playback input, and is configured to display an image corresponding to the decompressed third video data through the display. Claim 5 An electronic device according to claim 3, wherein the third processor is configured to acquire a transmission input for transmitting an image corresponding to the first video data to an external electronic device, and, based on acquiring the transmission input, to transmit the second video data to the external electronic device. Claim 6 An electronic device according to claim 1, wherein the second processor is configured to delete image data of the first video data corresponding to the subject area, compress the first video data from which the image data has been deleted to generate a fourth video data, and store the fourth video data in the memory. Claim 7 An electronic device according to claim 1, wherein the mask object is disposed on a layer located on the first video data. Claim 8 An electronic device according to claim 1, wherein the subject area is a part of the first video data determined to have captured a person. Claim 9 An electronic device according to claim 1, wherein the first processor performs object recognition by identifying a boundary that distinguishes the subject area and a background area surrounding at least a portion of the subject area, and the information transmitted to the second processor is configured to represent the mask object having the shape of the boundary. Claim 10 A method executed by an electronic device comprising a camera, a first processor configured to perform calculations related to an artificial intelligence model, and a second processor configured to compress first video data acquired from the camera, wherein the first processor is controlled: The operation of receiving the first video data from the camera; An operation of performing object recognition using the first video data to identify a subject area within the first video data; and The operation of transmitting information related to the subject area to the second processor, and controlling the second processor: The operation of receiving the first video data from the camera; The operation of receiving the information from the first processor; The operation of combining a mask object corresponding to the subject area appearing by the above information with the first video data; The operation of compressing the first video data combined with the mask object to generate second video data; and A method comprising the operation of storing the above second video data in the memory. Claim 11 A method according to claim 10, further comprising: controlling the second processor to generate third video data by compressing the first video data alone; and storing the third video data in the memory. Claim 12 The method of claim 11, wherein the electronic device further comprises a display configured to display an image; and a third processor which is a central processing unit (CPU). Claim 13 A method according to claim 12, further comprising: controlling the third processor to obtain a playback input for displaying video data; controlling the second processor to decompress the third video data based on obtaining the playback input; and displaying an image corresponding to the decompressed third video data through the display. Claim 14 A method according to claim 12, further comprising: controlling the third processor to obtain a transmission input for transmitting an image corresponding to the first video data to an external electronic device, and, based on obtaining the transmission input, transmitting the second video data to the external electronic device. Claim 15 A method according to claim 10, further comprising: controlling the second processor to delete image data of the first video data corresponding to the subject area; compressing the first video data from which the image data has been deleted to generate a fourth video data; and storing the fourth video data in the memory. Claim 16 A method according to claim 10, wherein the mask object is placed on a layer located on the first video data. Claim 17 The method of claim 10, wherein the subject area is a part of the first video data determined to have captured a person. Claim 18 A method according to claim 10, comprising: controlling the first processor to perform object recognition by identifying a boundary that distinguishes the subject area and a background area surrounding at least a portion of the subject area, and further comprising the operation of the information transmitted to the second processor representing the mask object having the shape of the boundary. Claim 19 A non-transient computer-readable storage medium for storing one or more programs, wherein the one or more programs are executed by an electronic device comprising a camera, a first processor configured to perform calculations related to an artificial intelligence model, and a second processor configured to compress first video data acquired from the camera, wherein the electronic device controls the first processor to: receive the first video data from the camera, perform object recognition using the first video data to identify a subject area within the first video data, and transmit information related to the subject area to the second processor, and controls the second processor to: receive the first video data from the camera, receive the information from the first processor, combine a mask object corresponding to the subject area indicated by the information with the first video data, compress the first video data combined with the mask object to generate second video data, and store the second video data in the memory. Claim 20 A non-transient computer-readable storage medium according to claim 19, wherein the one or more programs comprise instructions that cause the electronic device to control the second processor to: generate a third video data by compressing the first video data alone, and store the third video data in the memory.

Images