Wearable device for de-identifying a region within an image using a plurality of processors and method for operating the same

Abstract

According to various embodiments, a method of operation of a wearable device may be provided, comprising: an operation of acquiring at least one image using a camera; an operation of setting a task type for each of a plurality of processors for processing the at least one image; an operation of processing the acquired at least one image using at least one first processor among the plurality of processors and storing it in a specific storage; and an operation of acquiring result information by processing the at least one processed image stored in the specific storage based on an AI model while storing the at least one processed image in the specific storage using at least one second processor among the plurality of processors; wherein the task type for each of the plurality of processors is set based on a resource scheduling module implemented in a hardware abstraction layer supported by an operating system of the wearable device, and the specific storage is controlled to enable access by the at least one first processor and the at least one second processor based on a shared storage management module implemented in a driver layer supported by the operating system.

Description

Technology Field

[0001] The various embodiments disclosed in this document relate to a wearable device for de-identifying regions within an image using a plurality of processors and a method of operating the same. Background Technology

[0003] Wearable devices such as body cams and 360-degree wearable cameras require continuous operation for more than 8 hours in police, security, and industrial safety settings, and there are limitations on increasing battery capacity and product size due to considerations regarding user wearability. In addition, since it is difficult to apply fan-based active cooling structures, designing to minimize power consumption and heat generation within a passive cooling structure becomes a key requirement.

[0004] Conventionally, techniques for partial buffer sharing or memory copy reduction have been proposed in specific sections of the pipeline that process data input from image sensors or other sensors. However, these conventional techniques have failed to consistently implement an End-to-End Zero-Copy pipeline in which multiple heterogeneous processors share the same physical memory region without CPU memory copying across the entire software stack, from the sensor input processing layer to the output layer.

[0005] Furthermore, conventional heterogeneous processor scheduling technology has focused primarily on optimizing computational performance and has not taken into account the constraints of long-term stable operation in passive cooling environments unique to wearable devices. In other words, no priority-based dynamic throughput control structure is known that integrates real-time monitoring of the complex state of the wearable device, such as processor temperature, battery level, and output queue status, to absolutely protect the continuity of sensor data acquisition and storage functions while selectively adjusting the AI ​​inference cycle and rendering post-processing steps.

[0006] Furthermore, conventional shared memory-based technologies have only handled sections specialized for a single purpose; no hybrid anonymization pipeline structure is known in which color-based rule processing and AI-based object detection simultaneously access the same sensor data buffer without CPU intervention to perform parallel execution, and the results are directly transmitted to the output encoder without memory copying. Therefore, a new technology is required to integrate and resolve this. The problem to be solved

[0008] According to one embodiment of the present application, one objective is to provide a method of operation for a wearable device and a wearable device capable of simultaneously performing real-time sensor data processing and on-device AI inference while minimizing power consumption and heat generation by implementing an End-to-End Zero-copy pipeline in which a first processor that acquires data from a sensor and a second processor that performs AI model inference on the data share reference a shared physical memory area allocated by a shared memory management driver without CPU copying.

[0009] Another task is to provide a priority-based throughput control structure that sets the task type of each processor based on processing task mapping information by processor type stored in a resource schedule database, and integrates real-time monitoring of the complex state of the wearable device to dynamically adjust the AI ​​inference execution cycle and rendering post-processing step while absolutely protecting sensor data acquisition and storage with the highest priority.

[0010] Another task is to provide a hybrid anonymization pipeline structure in which a first processor directly accesses a sensor data buffer in a shared physical memory area without CPU intervention to extract a first region using color-based rules, and a second processor simultaneously directly accesses the same buffer to detect a second region using an AI model, performing these operations in parallel, and directly transfers the anonymization processing result buffer based on the two regions to an output encoder without memory copying.

[0011] The problems that this application aims to solve are not limited to those described above, and problems not mentioned will be clearly understood by those skilled in the art from this specification and the attached drawings. means of solving the problem

[0013] According to various embodiments, a method of operating a wearable device comprises: an operation of acquiring at least one image using a camera; an operation of setting a work type for each of a plurality of processors for processing the at least one image; an operation of identifying at least one first region corresponding to a specific color for the acquired at least one image using at least one first processor among the plurality of processors; an operation of identifying at least one second region using an object identification AI model for the at least one image using at least one second processor among the plurality of processors; and an operation of identifying the specific region associated with both the first region and the second region using at least one third processor among the plurality of processors, storing information about the identified specific region in a specific storage, and performing de-identification of the specific region. A method of operation may be provided in which the task type of each of the plurality of processors is set based on a resource scheduling module implemented in a hardware abstraction layer supported by the operating system of the wearable device, and the specific storage is controlled to enable access by the at least one first processor, the at least one second processor, and the at least one third processor based on a shared storage management module implemented in a driver layer supported by the operating system.

[0014] According to various embodiments, as a wearable device, a camera; A wearable device may be provided comprising: a plurality of processors; wherein the device acquires at least one image using a camera, sets a task type for each of the plurality of processors for processing the at least one image, identifies at least one first region corresponding to a specific color for the acquired at least one image using at least one first processor among the plurality of processors, identifies at least one second region using an object identification AI model for the at least one image using at least one second processor among the plurality of processors, identifies the specific region associated with both the first region and the second region using at least one third processor among the plurality of processors, stores information regarding the identified specific region in a specific storage, and controls the de-identification of the specific region to be performed, wherein the task type of each of the plurality of processors is set based on a resource scheduling module implemented in a hardware abstraction layer supported by an operating system of the wearable device, and the specific storage is controlled to enable access by the at least one first processor, the at least one second processor, and the at least one third processor based on a shared storage management module implemented in a driver layer supported by the operating system. there is.

[0015] According to various embodiments, as a computer program, when the computer program is executed by a wearable device: an operation of acquiring at least one image using a camera; an operation of setting a task type for each of a plurality of processors for processing the at least one image; an operation of identifying at least one first region corresponding to a specific color for the acquired at least one image using at least one first processor among the plurality of processors; and an operation of identifying at least one second region using an object identification AI model for the at least one image using at least one second processor among the plurality of processors; and an operation of identifying the specific region associated with both the first region and the second region using at least one third processor among the plurality of processors, storing information about the identified specific region in a specific storage, and performing de-identification of the specific region. A computer program may be provided in which the task type of each of the plurality of processors is set based on a resource scheduling module implemented in a hardware abstraction layer supported by the operating system of the wearable device, and the specific storage is controlled to enable access by the at least one first processor, the at least one second processor, and the at least one third processor based on a shared storage management module implemented in a driver layer supported by the operating system.

[0016] The means for solving the problem are not limited to the means described above, and unmentioned means of solving will be clearly understood by those skilled in the art to which this application pertains from this specification and the attached drawings. Effects of the invention

[0018] According to various embodiments, by applying an End-to-End Zero-copy pipeline across the entire software stack from the sensor input processing layer to the output encoder layer, the amount of DDR access resulting from repetitive memory copying in the existing structure can be significantly reduced. Based on actual measurements, it was confirmed that current consumption was reduced by approximately 18.8%, CPU utilization by approximately 9.6%, end-to-end latency by approximately 34.4%, and frame drops were achieved.

[0019] In addition, according to various embodiments, through a priority-based dynamic throughput control structure, the continuity of sensor data acquisition and storage is absolutely protected even when the processor temperature exceeds a threshold in a passive cooling environment, and long-term continuous operation is possible even in an environment where AI analysis is performed simultaneously for 6.9 hours in FHD and 5.5 hours in 4K.

[0020] In addition, according to various embodiments, through a hybrid anonymization pipeline, object detection accuracy of Precision 96.2%, Recall 94.1%, and F1-score 95.1% can be achieved without frame drops in an FHD 30fps environment, while minimizing power consumption compared to an AI-only method.

[0021] According to various embodiments, the effects are not limited to those described above, and unmentioned effects will be clearly understood by those skilled in the art from this specification and the accompanying drawings. Brief explanation of the drawing

[0023] FIG. 1 is a block diagram showing an example of the overall configuration of a wearable device according to various embodiments. FIG. 2 is a drawing showing examples of the external appearance of a wearable device according to various embodiments. FIG. 3 is a block diagram showing examples of hardware configurations of a wearable device according to various embodiments. FIG. 4 is a block diagram showing examples of the internal configuration and data flow of an on-device operation module according to various embodiments. FIG. 5 is a diagram illustrating an example of a zero-copy based data processing operation of a sensor data processing module according to various embodiments. FIG. 6 is a diagram showing an example of a processor distributed processing structure by operation type of an operation data acquisition module according to various embodiments. FIG. 7 is a diagram showing an example of a software layer structure of an on-device operation module according to various embodiments. FIG. 8 is a block diagram showing an example of the configuration of a multimodal situation recognition module according to various embodiments. FIG. 9 is a flowchart illustrating an example of an on-device operation method of a wearable device according to various embodiments. FIG. 10 is an illustrative diagram for explaining the role distribution processing operation of FIG. 9 by processor according to various embodiments. FIG. 11 is a flowchart illustrating an example of a device state-based throughput dynamic control operation according to various embodiments. FIG. 12 is an illustrative diagram for explaining the temperature-based throughput control operation of FIG. 11 according to various embodiments. FIG. 13 is a flowchart illustrating an example of a method for de-identifying a wearable device according to various embodiments. FIG. 14 is an illustrative diagram for explaining the results of the de-identification processing of FIG. 13 according to various embodiments. Specific details for implementing the invention

[0024] The embodiments described in this specification are intended to clearly explain the concept according to various embodiments to those skilled in the art to which this application belongs; therefore, this application is not limited by the embodiments described in this specification, and the scope according to various embodiments should be interpreted to include modifications or variations.

[0025] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other respect. Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0026] As used in this document, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0027] Various embodiments of this document may be implemented as software comprising one or more instructions stored in a storage medium (e.g., internal memory or external memory) that can be read by a machine (e.g., wearable device (1)). The storage medium that can be read by the machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0028] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online, either through an application store or directly between two user devices.

[0029] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0030] According to various embodiments, a method of operating a wearable device comprises: an operation of acquiring at least one image using a camera; an operation of setting a work type for each of a plurality of processors for processing the at least one image; an operation of identifying at least one first region corresponding to a specific color for the acquired at least one image using at least one first processor among the plurality of processors; an operation of identifying at least one second region using an object identification AI model for the at least one image using at least one second processor among the plurality of processors; and an operation of identifying the specific region associated with both the first region and the second region using at least one third processor among the plurality of processors, storing information about the identified specific region in a specific storage, and performing de-identification of the specific region. A method of operation may be provided in which the task type of each of the plurality of processors is set based on a resource scheduling module implemented in a hardware abstraction layer supported by the operating system of the wearable device, and the specific storage is controlled to enable access by the at least one first processor, the at least one second processor, and the at least one third processor based on a shared storage management module implemented in a driver layer supported by the operating system.

[0031] According to various embodiments, the at least one first processor includes an ISP and a GPU, and the operation of processing the at least one acquired image using the at least one first processor may be provided, comprising: an operation of allocating a sensor data buffer output from the ISP to a specific storage accessible to another processor using the shared storage management module and creating a memory handle; an operation of converting the memory handle into an image handle object of the graphics processing layer using the image handle conversion module; and an operation of directly binding the image handle object to an external texture of the GPU using a texture binding interface so that the GPU directly accesses and processes the sensor data of the specific storage without CPU copying.

[0032] According to various embodiments, the at least one second processor includes an NPU and the GPU, and the operation of obtaining result information by processing at least one processed image stored in the specific storage based on an AI model may be provided, comprising: providing the sensor data buffer of the specific storage as an input tensor of the AI ​​model using an AI runtime; causing the NPU to perform AI model inference by directly accessing the sensor data buffer without CPU copying using an AI accelerator driver; obtaining result information by performing the operation of the GPU performing processing on the same buffer of the specific storage in parallel; and outputting the result information by directly transmitting it to an output encoder without CPU copying through a buffer transfer interface.

[0033] According to various embodiments, a method of operation may be provided in which the shared storage management module is implemented as a driver that supports cross-device buffer sharing between a plurality of heterogeneous processors, the image handle conversion module is implemented as a graphics extension interface that provides mutual image handle conversion between a graphics processing layer and a native memory layer, the texture binding interface is implemented as a graphics processing interface that directly connects an external memory region to a GPU texture, and the buffer transfer interface is implemented as a native window interface that directly connects a GPU processing result buffer to an input buffer of an output encoder.

[0034] According to various embodiments, a method of operation may be provided in which the shared storage management module is implemented as an ION Driver or a DMA-BUF (Direct Memory Access Buffer) driver, the graphics extension interface is implemented as an eglCreateImageKHR function, the graphics processing interface is implemented as a glEGLImageTargetTexture2DOES function and an OES External Texture extension, and the native window interface is implemented as a Surface or ANativeWindow.

[0035] According to various embodiments, the operation of setting the task type of each of the plurality of processors for processing the at least one image is performed based on a resource schedule database, and the resource schedule database includes mapping information such that sensor data acquisition and sensor synchronization are performed by the CPU and ISP, AI-based object detection and de-identification inference are performed by the NPU and GPU, image synthesis is performed by the CPU and GPU, OSD synthesis is performed by the CPU, and output encoding is performed by a hardware encoder, and a method of operation may be provided.

[0036] According to various embodiments, an operation method may be provided in which the operation of setting a task type for each of the plurality of processors for processing at least one image includes: an operation of identifying the state of the wearable device; wherein the state of the wearable device includes at least one of a processor temperature, a battery level, a processor usage rate, or an output encoder queue state in real time, and the operation of dynamically adjusting the amount of processing for each task type set for each of the plurality of processors based on the state of the wearable device.

[0037] According to various embodiments, an operation for dynamically adjusting the throughput for each task type may be provided, comprising: an operation of outputting a user warning and reducing the AI ​​inference execution cycle by a first ratio of a reference value when the temperature of a specific processor among the plurality of processors exceeds a first threshold; an operation of further reducing the AI ​​inference execution cycle by a second ratio lower than the first ratio and completely reducing the rendering post-processing step when the temperature of a specific processor among the plurality of processors exceeds a second threshold higher than the first threshold; and an operation of performing a forced system shutdown while maintaining sensor data acquisition and storage when the temperature of a specific processor among the plurality of processors exceeds a third threshold higher than the second threshold, and then recovering the throughput stepwise according to a hysteresis structure after the processor temperature has decreased below a recovery threshold.

[0038] According to various embodiments, as a wearable device, a camera; A wearable device may be provided comprising: a plurality of processors; wherein the device acquires at least one image using a camera, sets a task type for each of the plurality of processors for processing the at least one image, identifies at least one first region corresponding to a specific color for the acquired at least one image using at least one first processor among the plurality of processors, identifies at least one second region using an object identification AI model for the at least one image using at least one second processor among the plurality of processors, identifies the specific region associated with both the first region and the second region using at least one third processor among the plurality of processors, stores information regarding the identified specific region in a specific storage, and controls the de-identification of the specific region to be performed, wherein the task type of each of the plurality of processors is set based on a resource scheduling module implemented in a hardware abstraction layer supported by an operating system of the wearable device, and the specific storage is controlled to enable access by the at least one first processor, the at least one second processor, and the at least one third processor based on a shared storage management module implemented in a driver layer supported by the operating system. there is.

[0039] According to various embodiments, the at least one first processor includes an ISP and a PU, and the wearable device may be provided, configured to use the at least one first processor to control, as at least as part of the operation of processing the at least one acquired image: allocating a sensor data buffer output from the ISP to the specific storage accessible to another processor using the shared storage management module and creating a memory handle, converting the memory handle into an image handle object of the graphics processing layer using the image handle conversion module, and directly binding the image handle object to an external texture of the GPU using a texture binding interface, so that the GPU directly accesses and processes the sensor data of the specific storage without CPU copying.

[0040] According to various embodiments, the at least one second processor includes an NPU and the GPU, and the wearable device may be provided in that, as at least part of the operation of obtaining result information by processing at least one processed image stored in the specific storage using the at least one second processor based on an AI model, the wearable device is configured to: provide the sensor data buffer of the specific storage as an input tensor of the AI ​​model using an AI runtime, enable the NPU to perform AI model inference by directly accessing the sensor data buffer without CPU copying using an AI accelerator driver, obtain result information by performing the operation of the GPU performing processing on the same buffer of the specific storage in parallel, and output the result information by directly transmitting it to an output encoder without CPU copying through a buffer transfer interface.

[0041] According to various embodiments, a wearable device may be provided in which the shared storage management module is implemented as a driver that supports cross-device buffer sharing between a plurality of heterogeneous processors, the image handle conversion module is implemented as a graphics extension interface that provides mutual image handle conversion between a graphics processing layer and a native memory layer, the texture binding interface is implemented as a graphics processing interface that directly connects an external memory area to a GPU texture, and the buffer transfer interface is implemented as a native window interface that directly connects a GPU processing result buffer to an input buffer of an output encoder.

[0042] According to various embodiments, a wearable device may be provided in which the shared storage management module is implemented as an ION Driver or a DMA-BUF (Direct Memory Access Buffer) driver, the graphics extension interface is implemented as an eglCreateImageKHR function, the graphics processing interface is implemented as a glEGLImageTargetTexture2DOES function and an OES External Texture extension, and the native window interface is implemented as a Surface or ANativeWindow.

[0043] According to various embodiments, the operation of setting the task type of each of the plurality of processors for processing the at least one image is performed based on a resource schedule database, and the resource schedule database includes mapping information such that sensor data acquisition and sensor synchronization are performed by the CPU and ISP, AI-based object detection and de-identification inference are performed by the NPU and GPU, image synthesis is performed by the CPU and GPU, OSD synthesis is performed by the CPU, and output encoding is performed by a hardware encoder, and a wearable device may be provided.

[0044] According to various embodiments, as a computer program, when the computer program is executed by a wearable device: an operation of acquiring at least one image using a camera; an operation of setting a task type for each of a plurality of processors for processing the at least one image; an operation of identifying at least one first region corresponding to a specific color for the acquired at least one image using at least one first processor among the plurality of processors; and an operation of identifying at least one second region using an object identification AI model for the at least one image using at least one second processor among the plurality of processors; and an operation of identifying the specific region associated with both the first region and the second region using at least one third processor among the plurality of processors, storing information about the identified specific region in a specific storage, and performing de-identification of the specific region. A computer program may be provided in which the task type of each of the plurality of processors is set based on a resource scheduling module implemented in a hardware abstraction layer supported by the operating system of the wearable device, and the specific storage is controlled to enable access by the at least one first processor, the at least one second processor, and the at least one third processor based on a shared storage management module implemented in a driver layer supported by the operating system.

[0046] FIG. 1 is a block diagram showing an example of the overall configuration of a wearable device (1) according to various embodiments.

[0047] Referring to FIG. 1, the wearable device (1) may include hardware (10), an on-device operation module (20), and a multimodal situational awareness module (30). Meanwhile, without being limited to the described and / or illustrated examples, the wearable device (1) may be implemented to include more components and / or fewer components.

[0048] According to various embodiments, the wearable device (1) is a device that is worn on the user's body and operated, and may be, for example, a wearable shooting device such as a body cam, a 360-degree wearable camera, or a smart helmet camera, but is not limited thereto. The wearable device (1) is designed with a passive cooling method and has a structure that minimizes power consumption and heat generation without a separate cooling fan, and is equipped with on-device AI so that real-time AI inference can be performed within the device without transmitting data to an external server.

[0049] According to various embodiments, the on-device operation module (20) may be implemented to obtain result information by analyzing information obtained from at least some of the hardware (10) of the wearable device (1) (e.g., sensor device (11)) using a multimodal situation awareness module (30). The result information may be used to recognize the surrounding situation of a user wearing the wearable device (1).

[0050] According to various embodiments, the on-device operation module (20) can perform role distribution scheduling operations by task type based on the resource schedule database (440).

[0051] According to various embodiments, the on-device operation module (20) may provide an End-to-End Zero-copy pipeline that sequentially links a shared memory management driver, an image handle conversion module, a texture binding interface, and a buffer transfer interface across the entire software stack from the sensor input processing layer to the output encoder layer.

[0052] According to various embodiments, the on-device operation module (20) can perform a dynamic throughput control mechanism that integrates real-time monitoring of the complex state of the wearable device (1) and links a priority system with a multi-stage temperature threshold to absolutely protect the continuity of sensor data acquisition and storage.

[0053] According to various embodiments, the multimodal situation awareness module (30) may be implemented to analyze information (e.g., an image captured by a camera) obtained by at least some of the hardware (10) of the wearable device (1) (e.g., a sensor device (11)).

[0055] FIG. 2 is a drawing showing examples of the external appearance of a wearable device (1) according to various embodiments.

[0056] The wearable device (1) can be implemented in various forms, such as a body cam as in (a) of FIG. 2 or a 360-degree camera as in (b) of FIG. 2. Not limited to the described and / or illustrated examples, the wearable device (1) has low-power based sustainable operation capability as a priority design goal over AI computation performance itself under battery capacity and size constraints.

[0058] FIG. 3 is a block diagram illustrating an example of the configuration of hardware (10) of a wearable device (1) according to various embodiments. FIG. 3 will be further explained below with reference to FIG. 4 to 8.

[0059] Referring to FIG. 3, the hardware (10) may include a sensor device (11), an output device (13), a processor (15), and a battery (17). Meanwhile, the hardware (10) may include fewer or more configurations than those shown in FIG. 3, and is not limited to those shown in FIG. 3.

[0060] According to various embodiments, the sensor device (11) may include a camera (11a), a microphone (11b), and other sensors (11c). The camera (11a) is a device for acquiring images and / or at least one image of the external environment of the wearable device (1), and may be implemented, for example, as a camera module capable of capturing images of FHD (1920×1080) to 4K (3840×2160) resolution at 30fps or higher, but is not limited thereto. The microphone (11b) is a device configured to acquire acoustic signals from the outside, and may be implemented, for example, as a MEMS microphone. The above other sensor (11c) is a device configured to detect the position, posture, movement, or surrounding environment conditions of the wearable device (1), and may include, for example, at least one of a 6-axis IMU (Inertial Measurement Unit), a positioning sensor (e.g., GPS), a short-range wireless communication-based position recognition receiver (e.g., BLE Beacon receiver), a geomagnetic sensor (magnetometer), a barometric pressure sensor, a temperature sensor, or an illuminance sensor, but is not limited thereto.

[0061] According to various embodiments, the output device (13) may include a communication circuit (13a), a speaker (13b), and other output means. The communication circuit (13a) provides wireless communication functions and may support real-time streaming and event transmission, etc.

[0062] According to various embodiments, the processor (15) may be configured as a heterogeneous processor structure including a CPU (15a), a GPU (15b), an NPU (15c), and an ISP (15d). This heterogeneous processor structure may be integrated within a single System on Chip (SoC). The main roles of each processor and the types of tasks suitable for processing are as shown in Table 1 below.

[0063] processor Key roles Suitable task types and implementation examples CPU (15a) Low-latency sequential control tasks such as system control, event processing, sensor synchronization, and streaming session management Suitable for low-latency sequential operations; e.g., Kryo CPU core (Qualcomm Snapdragon) GPU (15b) Massive parallel pixel processing tasks such as image compositing, mosaic processing, and color-based region extraction Suitable for massively parallel computation; e.g., Adreno GPU, OpenGL ES / OpenCL runtimes NPU (15c) AI tensor inference operations such as object detection, face recognition, and context awareness Specialized for neural network inference; e.g., Hexagon DSP / NPU, SNPE / QNN Runtime ISP (15d) After processing image correction on the data input from the image sensor, output to the shared memory area. Sensor input processing layer hardware accelerator; e.g., Spectra ISP, CamX / CHI

[0064] According to various embodiments, as illustrated in Table 1, the CPU (15a), GPU (15b), NPU (15c), and ISP (15d) share tasks optimized for their respective characteristics, thereby preventing problems such as memory bottlenecks, reduced cache efficiency, and processing delays that occur when work is concentrated on specific computational resources. Not limited to the described and / or illustrated examples, the processor (15) may be implemented with an equivalent heterogeneous processor structure from other manufacturers.

[0065] FIG. 4 is a block diagram showing examples of the internal configuration and data flow of an on-device operation module (20) according to various embodiments.

[0066] Referring to FIG. 4, the on-device operation module (20) may include a sensor data processing module (410), a computation data acquisition module (420), a resource scheduling module (430), a resource scheduling database (440), a shared storage (450), and a shared storage management module (460). Meanwhile, not limited to what is shown in FIG. 4, the on-device operation module (20) may include fewer or more configurations (e.g., a delivery interface (470)) than those shown.

[0067] The operation of each module described below can be understood as being performed by the processor (15) performing an operation (or executing a function) corresponding to the on-device operation module (20) based on the execution of the on-device operation module (20) (e.g., sensor data processing module (410), computation data acquisition module (420), resource scheduling module (430), resource scheduling database (440), shared storage (450), and shared storage management module (460)).

[0068] According to various embodiments, the sensor data processing module (410) may be implemented to process sensor data obtained from the sensor device (11). Specifically, the wearable device (10) may process data received from the sensor device (11) using the sensor data processing module (410) by using at least one first processor among the processors (15). The processed data may be stored in a shared storage (450) in a zero-copy manner using the shared storage management module (460) described later. The zero-copy method may mean allocating a physical memory area that can be shared by a plurality of heterogeneous processors. For example, when an image captured by a camera (11a) is obtained, the wearable device (10) may include a structure that creates an image handle object for it without CPU copying using an image handle conversion module (not shown) of the sensor data processing module (410), and directly binds the image handle object to an external texture of the GPU using a texture binding interface. For example, the shared storage management module (460) may be implemented as an ION Driver or a DMA-BUF Driver, the image handle conversion module as an eglCreateImageKHR function of EGL, and the texture binding interface as a glEGLImageTargetTexture2DOES function of OpenGL ES, but are not limited thereto.

[0069] According to various embodiments, the computation data acquisition module (420) can acquire computation data for AI model computation. Specifically, the computation data acquisition module (420) can acquire result information by performing AI model inference by directly accessing the physical memory area of ​​the shared storage (450) that stores the aforementioned processed sensor data without CPU copying through an AI runtime (not shown) and an AI accelerator driver (not shown) using at least one second processor. For example, the AI ​​runtime may be implemented as a Snapdragon Neural Processing Engine (SNPE) or a Qualcomm Neural Network (QNN) Runtime, and the AI ​​accelerator driver may be implemented as a DSP Driver (Hexagon DSP / NPU control), but is not limited thereto. According to various embodiments, when the de-identification function is enabled, the sensor data processing module (410) directly accesses the buffer of the shared storage (450) using the GPU (15b) in the HAL layer (720) and extracts the first region by color-based rule processing, and the computation data acquisition module (420) simultaneously directly accesses the same buffer without CPU copying using the NPU (15c) and detects the second region with an AI model. The two operations are performed in parallel through the shared physical memory area of ​​the Driver layer (730) by being located in the same HAL layer (720).

[0070] According to various embodiments, the resource scheduling module (430) can set task types for multiple processors and schedule processing to be performed in parallel according to processing task mapping information by processor type stored in the resource scheduling database (440). Parallel processing means that operations can be performed by each of the multiple processors (15) during the same time interval. In addition, the resource scheduling module (430) monitors complex device status in real time and can redistribute resources according to the priority system of Table 2 below when the system load increases.

[0071] priority function Load Handling Policy Level 1 Sensor data acquisition and storage Cannot be interrupted under any temperature or load conditions; protected until just before forced shutdown Level 2 Encoding and streaming Maintaining output encoder-based processing and transmission continuity as the top priority Level 3 AI inference Gradually reduce the AI ​​inference execution cycle under load from 50% to 25% of the baseline value. Level 4 Rendering Post-processing and OSD Processing sequence: Rendering effect reduction upon load → Full reduction in post-processing stage. Level 5 Background tasks Remote updates, log uploads, etc. — Prioritize deferred processing during load

[0072] According to various embodiments, the resource scheduling database (440) may store mapping information between a task type and a processor type to process the task. For example, information may be stored in which sensor data acquisition and synchronization are assigned to the CPU (15a) and ISP (15d), AI-based object detection and de-identification inference to the NPU (15c) and GPU (15b), 360-degree image synthesis to the CPU (15a) and GPU (15b), OSD synthesis to the CPU (15a), and output encoding to the hardware encoder, respectively, but is not limited thereto. According to various embodiments, the shared storage (450) may be implemented as a shared memory structure in which a plurality of heterogeneous processors can directly reference the same physical memory area without copying CPU memory. For example, the shared memory structure may be implemented as ION Shared Memory, DMA-BUF (Direct Memory Access Buffer), or AHardwareBuffer, but is not limited thereto.

[0074] FIG. 5 is a diagram illustrating an example of a zero-copy based data processing operation of a sensor data processing module (410) according to various embodiments. Below, an example of an operation for processing an image acquired by a camera (11a) is described using each step illustrated in FIG. 5.

[0075] According to various embodiments, the sensor data processing module (410) can perform the sensor data acquisition (510), shared memory binding (520), and output encoding (530) processes in an End-to-End Zero-Copy manner.

[0076] According to various embodiments, in the sensor data acquisition (510) step, the ISP (15d) performs image corrections such as color correction, noise removal, white balance, and image stabilization on raw image data input from the camera (11a), and outputs the processed frame data to the physical memory area of ​​the shared storage (450) through the shared memory management driver. At this point, the sensor data is not copied to the CPU memory space, and the shared memory management driver creates a memory handle for the corresponding physical memory area and transmits it to the upper layer. For example, the shared storage management module (460) may be implemented as an ION Driver or a DMA-BUF Driver, and the memory handle may be implemented as a Native Buffer Handle or an AhardwareBuffer, but is not limited thereto. Meanwhile, data acquired from sensors other than the camera (11a) (e.g., microphone (11b), other sensors (11c)) may also be shared with other heterogeneous processors through the same shared memory structure.

[0077] According to various embodiments, in the shared memory binding (520) step, the image handle conversion module creates an image handle object of the graphics processing layer using the memory handle as an argument. The image handle conversion module belongs to the HAL layer (720) and performs the role of converting a physical memory area allocated by the shared storage management module (460) of the Driver layer (730) into a handle of the graphics processing layer without CPU memory copying. In this process, no CPU memory copying occurs. Subsequently, the texture binding interface directly binds the image handle object to an external texture of the GPU (15b), allowing the GPU Shader to directly access sensor data of the shared storage (450) without CPU copying. Through this, GPU-based parallel pixel processing, such as image compositing, OSD compositing, and mosaic processing, can be performed without separate memory copying. For example, the image handle conversion module may be implemented as the eglCreateImageKHR function of EGL, the image handle object as an EGLImageKHR object, and the texture binding interface as the glEGLImageTargetTexture2DOES function of OpenGL ES and the OES External Texture extension, but is not limited thereto.

[0078] According to various embodiments, in the output encoding (530) step, the result buffer processed by the GPU (15b) can be directly transferred to the output encoder through a buffer transfer interface without additional CPU copying and encoded. For example, the buffer transfer interface may be implemented as an Android Surface or ANativeWindow interface, and the output encoder may be implemented as an H.264 / H.265 hardware codec, but is not limited thereto.

[0079] According to various embodiments, the step-by-step flow of the End-to-End Zero-copy pipeline is as shown in Table 3 below.

[0080] step Processing details Processing entity Implementation example 1 Raw data input and correction processing from sensors ISP(15d) / Sensor Input Processing Layer CamX / CHI, Spectra ISP (for image sensors) 2 Allocate processed data to a shared physical memory area; create memory handle Shared storage management module (460) ION Driver, DMA-BUF 3 Convert memory handle to image handle object in the graphics processing layer (no CPU copy) Image handle conversion module EGL / eglCreateImageKHR 4 Directly bind image handle objects to external GPU textures Texture binding interface glEGLImageTargetTexture2DOES 5 Parallel pixel processing via direct access to the GPU (Compositing / OSD / Mosaic) GPU(15b) / Graphics Runtime OpenGL ES 6 Pass the processing result buffer directly to the output encoder (no additional CPU copy) Delivery interface (470) / framework layer (710) Surface / ANativeWindow 7 The output encoder directly reads the result buffer for encoding and output. Output Encoder / Hardware Layer H.264 / H.265 HW Codec

[0081] According to various embodiments, as illustrated in Table 3, the End-to-End Zero-copy pipeline of the present application implements a structure in which no CPU memory copy occurs along the entire path from the sensor input processing layer (steps 1-2) through the graphics processing layer (steps 3-5) to the output encoder layer (steps 6-7). Unlike prior art which dealt with partial buffer sharing in specific sections, the present application is differentiated in that it implements an End-to-End structure in which heterogeneous processors share and reference the same physical memory area throughout the entire path from the sensor input to the encoder output.

[0082] According to various embodiments, an End-to-End Zero-copy pipeline is maintained even when the de-identification function is enabled. Specifically, the AI ​​inference module of the second processor (e.g., NPU (15c)) performs detection inference by directly accessing the same sensor data buffer in the shared storage (450) without CPU copying via the AI ​​runtime and AI accelerator driver, writes the detection result (Detection Metadata) to the shared storage (450), and the first processor (e.g., GPU (15b)) performs mosaic processing by referencing this and then directly transmits the result buffer to the output encoder via the buffer transfer interface. Based on actual measurements, power consumption when the de-identification function is enabled is controlled to increase by approximately 0.28W compared to recording alone.

[0083] According to various embodiments, data acquired from sensors other than the camera (11a) (e.g., microphone (11b), IMU (other sensor (11c))) can also be directly accessed by the AI ​​runtime through the same shared memory structure and utilized for multimodal AI inference. This allows image-based processing and other sensor-based processing to be integrated and operated on the shared memory.

[0085] FIG. 6 is a diagram showing an example of a processor distributed processing structure by operation type of a computation data acquisition module (420) according to various embodiments. Below, an example of an operation for processing data acquired by the aforementioned sensor device (11) using an AI model (30) is described.

[0086] Referring to FIG. 6, the computation data acquisition module (420) can distribute and process various types of computations occurring during the AI ​​model computation process according to the characteristics of each processor. The CPU (15a) is responsible for sequential control flow and low-latency event processing, the GPU (15b) is responsible for massive parallel pixel computation and matrix computation, and the NPU (15c) is responsible for neural network inference computation through the AI ​​runtime, thereby distributing computations to the most efficient computation resources. For example, the AI ​​runtime may be implemented as SNPE or QNN Runtime, but is not limited thereto.

[0087] According to various embodiments, the distributed processing structure by operation type operates in conjunction with the priority system of the resource scheduling module (430). When the temperature rises or a streaming processing delay occurs, the resource scheduling module (430) adjusts the execution cycle of AI inference (Level 3), gradually reduces the rendering post-processing (Level 4), and protects sensor data acquisition and storage (Level 1) and encoding and streaming (Level 2) under any circumstances.

[0089] FIG. 7 is a diagram showing an example of a software layer structure in which an on-device operation module (20) occupies an operating system (700) according to various embodiments.

[0090] Referring to FIG. 7, the operating system (700) may be configured as a multi-layered structure including a framework layer (710), a HAL layer (720), and a Driver layer (730). For example, the operating system (700) may be an Android OS (operating system).

[0091] According to various embodiments, the on-device operation module (20) is implemented across the framework layer (710), HAL layer (720), and Driver layer (730), and may include a delivery interface (470) belonging to the framework layer (710), a sensor data processing module (410), a computation data acquisition module (420), and a resource scheduling module (430) belonging to the HAL layer (720), and a shared storage management module (460) belonging to the Driver layer (730). Meanwhile, the on-device operation module (20) may include more or fewer configurations than those shown in FIG. 7, without being limited to those shown in FIG. 7.

[0092] According to various embodiments, conventional AI camera products have adopted a structure that adds AI functions only at the application layer or framework layer level. In such conventional structures, CPU memory copying occurs repeatedly during the process of passing through upper layers to access the physical memory area of ​​the lower driver layer, and there were limitations such as frame drops, streaming delays, and increased heat generation when the AI ​​inference load increased. On the other hand, the on-device operation module (20) of the present application is implemented with a structure that penetrates vertically from the framework layer (710) to the driver layer (730), and is fundamentally differentiated from the conventional structure in that it achieves an End-to-End Zero-copy pipeline that references the same shared physical memory area without CPU copying throughout the entire section from physical memory allocation to final output.

[0093] According to various embodiments, the delivery interface (470) belongs to the framework layer (710) and performs the role of directly delivering the result buffer processed by the GPU (15b) to the output encoder without CPU copying. The delivery interface (470) interacts with the media framework and sensor services of the framework layer (710) to mediate between the streaming control request of the upper application layer and the processing result of the lower HAL layer (720). The delivery interface (470) delivers only the processing result buffer of the GPU (15b) to the output encoder, and the AI ​​inference of the NPU (15c) shares the processing result directly through the shared storage (450) without passing through the delivery interface (470). For example, the delivery interface (470) may be implemented as a Surface or ANativeWindow interface, but is not limited thereto.

[0094] According to various embodiments, the sensor data processing module (410) and the computation data acquisition module (420) belong to the HAL layer (720) and are responsible for sensor data processing and AI model computation data acquisition, respectively. Since the two modules are located in the HAL layer (720), they can directly access the shared physical memory area allocated by the shared storage management module (460) of the Driver layer (730) without CPU copying through the image handle conversion module and texture binding interface. In particular, since the sensor data processing module (410) and the computation data acquisition module (420) are located in the same HAL layer (720) and reference the shared physical memory area of ​​the Driver layer (730) together, the rendering processing of the GPU (15b) and the AI ​​inference of the NPU (15c) can be performed in parallel on the same buffer without memory copying. For example, the image handle conversion module may be implemented as the eglCreateImageKHR function, the texture binding interface as the glEGLImageTargetTexture2DOES function, the AI ​​runtime as SNPE or QNN Runtime, and the AI ​​accelerator driver as the DSP Driver, but are not limited thereto.

[0095] According to various embodiments, the resource scheduling module (430) belongs to the HAL layer (720) and can set task types for multiple processors and schedule parallel processing according to processing task mapping information by processor type stored in the resource scheduling database (440). Since the resource scheduling module (430) is located in the HAL layer (720), it can directly obtain memory allocation status information from the shared storage management module (460) of the Driver layer (730), and collect processor temperature, battery level, processor usage rate, and output encoder queue status in real time through the system status monitoring interface provided by the operating system, and dynamically update scheduling information according to the priority system of Table 2. Specifically, the resource scheduling module (430) absolutely protects sensor data acquisition and storage (Level 1) with the highest priority, and when the load increases, it adjusts throughput stepwise from Level 5 (background work) to Level 4 (rendering post-processing) and Level 3 (AI inference), and Level 1 (sensor data acquisition and storage) and Level 2 (encoding and streaming) are absolutely protected in any overheating situation. Since the resource scheduling module (430) is implemented in the HAL layer (720) rather than the application layer, scheduling decisions are made at the layer that directly reads the hardware state, thus ensuring real-time performance of dynamic throughput adjustment based on Priority Class.

[0096] According to various embodiments, the shared storage management module (460) belongs to the Driver layer (730) and performs the role of allocating a shared physical memory area that allows multiple heterogeneous processors to directly reference the same physical memory area without CPU memory copying, and managing cross-device buffer sharing. By having the shared storage management module (460) located in the Driver layer (730), the sensor data processing module (410) and the computation data acquisition module (420) of the HAL layer (720) can directly reference the same physical address without additional memory copying through the upper layer, and the delivery interface (470) of the framework layer (710) can also directly deliver the buffer of the same physical memory area to the output encoder. For example, the shared storage management module (460) may be implemented as an ION Driver or a DMA-BUF driver, but is not limited thereto.

[0097] According to various embodiments, through a structure in which the on-device operation module (20) is vertically implemented across the framework layer (710), HAL layer (720), and Driver layer (730), the delivery interface (470) is responsible for the final output path in the framework layer (710), the sensor data processing module (410), the computation data acquisition module (420), and the resource scheduling module (430) perform actual data processing and scheduling in the HAL layer (720), and the shared storage management module (460) manages physical memory sharing in the Driver layer (730), thereby structurally implementing an End-to-End Zero-copy pipeline in which the three layers share and reference the same shared physical memory area without CPU copying.

[0098] According to various embodiments, by applying a structure that separates the hardware abstraction layer and the AI ​​function area, a new AI model can be independently updated via a remote update method without changing the entire firmware.

[0099] FIG. 8 is a block diagram showing an example of the configuration of a multimodal situation recognition module (30) according to various embodiments.

[0100] Referring to FIG. 8, the multimodal situation awareness module (30) may include an RIA module (810), a TAD module (820), and other AI models (830). Meanwhile, the multimodal situation awareness module (30) may include fewer or more configurations than those shown in FIG. 8, without being limited to those shown in FIG. 8.

[0101] According to various embodiments, the RIA module (810) may include a skin recognition module (811), which is a color-based rule processing module, and an object recognition AI model (813), which is an AI-based object detection module. The skin recognition module (811) uses a first processor (e.g., GPU (15b)) to directly access a sensor data buffer in a shared storage (450) via a texture binding interface and extracts candidate regions (first regions) corresponding to a specific color through color space conversion and threshold-based filtering. The object recognition AI model (813) uses a second processor (e.g., NPU (15c)) to simultaneously and directly access the same buffer without CPU copying via an AI runtime and an AI accelerator driver to detect object regions (second regions) using a lightweight AI detection model. The first region extraction by the skin recognition module (811) and the second region detection by the object recognition AI model (813) are performed in parallel, and a third processor (e.g., GPU (15b)) performs de-identification (mosaic) processing on a specific region based on the two regions. For example, the lightweight AI detection model may be an SCRFD-based face detection model, and the de-identification processing may be implemented as OpenGL ES-based mosaic or blur processing, but is not limited thereto. According to various embodiments, the result information generated by the RIA module (810) is a frame buffer in which de-identification processing is completed, and the first region extraction by the first processor (GPU (15b)) and the second region detection by the second processor (NPU (15c)) are performed in parallel on the same buffer of the shared storage (450), and the entire process in which the de-identification processing result based on the two regions is transmitted to the output encoder through the buffer transfer interface (470) without CPU copying is completed as an End-to-End Zero-copy pipeline.

[0102] According to various embodiments, the TAD module (820) includes an acoustic-based event recognition AI model (821) and can generate an event trigger by detecting abnormal sounds, etc., from acoustic data input through the microphone (11b). Other AI models (830) may include a threat recognition module, a fall recognition module, a fire recognition module, etc., and can operate as a multimodal situational awareness structure that fuses information obtained from the camera (11a), the microphone (11b), and other sensors (11c) based on the time axis.

[0104] FIG. 9 is a flowchart illustrating an example of an on-device operation method of a wearable device (1) according to various embodiments. The operations illustrated in FIG. 9 may be performed in various orders, not limited to the order illustrated, and more operations may be performed than those illustrated in FIG. 9, or at least one operation may be performed. FIG. 9 will be described below with reference to FIG. 10.

[0105] According to various embodiments, the wearable device (1) can acquire at least one sensor data using a sensor device (11) in operation 901. The sensor device (11) may include a camera (11a), a microphone (11b), and other sensors (11c). For example, the camera (11a) may be implemented as a camera module capable of capturing images of FHD to 4K resolution at 30fps or higher, but is not limited thereto. The ISP (15d) performs image correction on raw image data input from the camera (11a) and outputs the processed frame through the shared storage management module (460) to the physical memory area of ​​the shared storage (450).

[0106] According to various embodiments, the wearable device (1) can set the task type of each of the plurality of processors for processing the at least one sensor data in operation 903. The resource scheduling module (430) performs scheduling based on processing task mapping information by processor type stored in the resource scheduling database (440), distributing sensor data acquisition and sensor synchronization tasks to the CPU (15a) and ISP (15d), massively parallel processing tasks to the GPU (15b), and AI tensor inference operations to the NPU (15c). At the same time, the resource scheduling module (430) can collect processor temperature, battery level, processor usage, and output encoder queue status in real time through a system status monitoring interface, and dynamically update scheduling information according to the priority system of Table 2.

[0107] According to various embodiments, the wearable device (1) can process the acquired at least one sensor data using at least one first processor (e.g., CPU (15a) and / or ISP (15d)) among a plurality of processors in operation 905 and store it in a shared storage (450).

[0108] According to various embodiments, when the de-identification function is enabled, in operation 905, a first processor (e.g., GPU (15b)) performs color-based rule processing (color space conversion and threshold-based filtering) on ​​the same buffer of the shared storage (450) to extract a first region, and in operation 907 described below, a second processor (e.g., NPU (15c)) simultaneously directly accesses the same physical memory region without CPU copying to detect a second region through AI model inference. The two operations are performed in parallel on the same buffer of the shared storage (450), which corresponds to the operation of the first processor performing processing on the same buffer of a specific storage and the operation of the NPU performing AI model inference being performed in parallel to obtain result information.

[0109] In one embodiment, the operation of storing in the shared storage (450) may include the following detailed operations.

[0110] In at least some of the operations of 905, the wearable device (1) can receive sensor data output from the sensor device (11) using a sensor input processing layer (e.g., ISP (15d), Camera HAL) and request the shared storage management module (460) to allocate a shared physical memory area to create a memory handle. Up to this point, the sensor data is not copied to the CPU memory space.

[0111] In at least some of the operations of 905, the wearable device (1) can convert the memory handle into an image handle object of the graphics processing layer using an image handle conversion module. In this process, no CPU memory copying occurs.

[0112] In at least some of the operations of 905, the wearable device (1) can directly bind the image handle object to an external texture of the first processor (e.g., GPU (15b)) using a texture binding interface. The first processor can then directly access the sensor data through the external texture without CPU copying to perform parallel processing such as image synthesis, OSD synthesis, and mosaic processing, and record the results in a shared storage (450).

[0113] According to various embodiments, the wearable device (1) may obtain result information by processing the processed sensor data stored in the shared storage (450) based on an AI model using at least one second processor (e.g., GPU (15b) and / or NPU (15c)) among a plurality of processors in operation 907. Operation 907 may be performed in parallel with operation 905 and includes the following detailed steps.

[0114] In at least some of the operations of 907, the wearable device (1) can use the AI ​​runtime to directly reference a physical memory region of the shared storage (450) as an input tensor of the AI ​​model, and through the AI ​​accelerator driver, allow a second processor (e.g., NPU (15c)) to directly access the physical memory region without CPU copying to perform AI model inference. Since the physical memory region accessed by the second processor is the same region that the first processor of the 905 operation has bound to an external texture, the processing of the first processor and the AI ​​inference of the second processor are performed simultaneously in parallel.

[0115] In at least some of the operations of 907, the wearable device (1) can store the AI ​​inference results of the second processor in a shared storage (450). Afterward, the wearable device (1) uses the first processor to refer to the stored AI inference results, performs post-processing on the corresponding area, and then writes the final result buffer to the shared storage (450).

[0116] In at least some of the operations of 907, the wearable device (1) can directly pass the final result buffer to the output encoder without additional CPU copy using a buffer passing interface, and perform encoding using the output encoder to output to a storage or streaming path. This completes an End-to-End Zero-copy pipeline across the entire software stack from the sensor input to the output encoder.

[0117] FIG. 10 is an illustrative diagram for explaining the role distribution processing operation of FIG. 9 by processor according to various embodiments.

[0118] Referring to FIG. 10, the CPU (15a) is responsible for sensor data acquisition requests, sensor synchronization, and streaming session management, the NPU (15c) performs AI inference through the AI ​​runtime and AI accelerator driver, and the GPU (15b) performs image rendering and mosaic processing through the graphics runtime. The three processors each directly access the same physical memory area of ​​the shared storage (450) managed by the shared storage management module (460) without CPU copying, thereby enabling parallel processing without an increase in memory access volume due to data transfer between processors.

[0120] FIG. 11 is a flowchart illustrating an example of a device state-based throughput dynamic control operation according to various embodiments. The operations illustrated in FIG. 11 may be performed in various orders, not limited to the order illustrated, and more operations may be performed than those illustrated in FIG. 11, or at least one operation may be performed. FIG. 11 will be described below with reference to FIG. 12. Meanwhile, the operation (1101) of FIG. 11 may be performed as at least part of the operation (903) of FIG. 9.

[0121] According to various embodiments, the wearable device (1) can identify the state of the wearable device in operation 1101. The state of the wearable device may include at least one of the processor temperature, battery level, CPU (15a), GPU (15b), NPU (15c) usage, output encoder queue status, and current processing cycle. The device state may be collected in real time through a system state monitoring interface provided by the operating system.

[0122] According to various embodiments, the wearable device (1) can dynamically adjust the throughput for each task type set for each of the multiple processors in conjunction with the priority system of Table 2, based on the device state identified in the 1103 operation. Specifically, the multi-stage temperature protection structure operates as shown in Table 4 below.

[0123] Protection level Temperature conditions Operation details (Priority system integration) 1st Processor temperature ≥ First threshold Output user alert; halt Level 5 (background tasks); reduce Level 3 (AI inference) execution cycle to 50% of the baseline value. 2nd Processor temperature ≥ Second threshold User alert and choice to continue operation; Level 4 (Post-rendering) full reduction; Level 3 (AI Inference) execution cycle further reduced to 25% of the baseline. 3rd Processor temperature ≥ 3rd threshold Maintain Level 1 (Sensor Data Acquisition / Storage) and Level 2 (Encoding / Streaming) protection; completely halt Levels 3–5 and force system shutdown restoration Processor temperature ≤ recovery threshold Application of hysteresis structure; stepwise recovery of throughput in the order of Level 4 → Level 3 → Level 5 after temperature stabilization

[0124] According to various embodiments, the temperature protection structure of Table 4 is linked with the priority system of Table 2, so that Level 1 (sensor data acquisition and storage) and Level 2 (encoding and streaming) are absolutely protected under any heat generation conditions, whereas Level 3 (AI inference), Level 4 (rendering post-processing), and Level 5 (background tasks) can be restricted stepwise depending on the processor temperature.

[0125] For example, the first threshold may be set to 42°C, the second threshold to 45°C, the third threshold to 49°C, and the recovery threshold to 38°C, but is not limited thereto and may be adjusted according to the operating environment and product specifications.

[0127] FIG. 12 is an illustrative diagram for explaining the temperature-based throughput control operation of FIG. 11 according to various embodiments.

[0128] Referring to FIG. 12, in a normal state, sensor data acquisition by the CPU (15a), AI inference by the NPU (15c), and rendering by the GPU (15b) are all performed in parallel at a normal cycle. However, when the processor temperature exceeds a first threshold, the AI ​​inference execution cycle is reduced to 50%, and when it exceeds a second threshold, the AI ​​inference cycle is further reduced to 25% and the rendering post-processing is reduced. In any case, a structure can be illustrated in which sensor data acquisition and output encoder-based encoding are maintained normally.

[0129] FIG. 13 is a flowchart illustrating an example of a method for de-identifying a wearable device (1) according to various embodiments. The operations illustrated in FIG. 13 may be performed in various orders, not limited to the order illustrated, and more operations may be performed than those illustrated in FIG. 13, or at least one operation may be performed. FIG. 13 will be described below with reference to FIG. 14.

[0130] According to various embodiments, the wearable device (1) acquires at least one sensor data using a sensor device (11) in operation 1301, and the shared storage management module (460) may allocate the processed sensor data buffer to a shared physical memory area. The sensor device (11) may include, for example, a camera (11a), but is not limited thereto, and may be used to determine a de-identification target area by fusing with data acquired from a microphone (11b) or other sensor (11c).

[0131] According to various embodiments, the wearable device (1) can set the task type of each of a plurality of processors for de-identifying a specific area within the sensor data in operation 1303. The resource scheduling module (430) performs the task type setting, assigning color-based rule processing to the GPU (15b), object detection by an AI model to the NPU (15c), and de-identification rendering to the GPU (15b).

[0132] According to various embodiments, the wearable device (1) can identify at least one first region using at least one first processor (e.g., GPU (15b)) among a plurality of processors in operation 1305. Specifically, the first processor directly accesses a sensor data buffer stored in a shared storage (450) through a texture binding interface, extracts a region (first region) corresponding to a specific color by performing color space conversion and threshold-based filtering, and records the extraction result in the shared storage (450).

[0133] According to various embodiments, the wearable device (1) may identify at least one second region using at least one second processor (e.g., NPU (15c)) among a plurality of processors in operation 1307. Specifically, the AI ​​runtime directly references the physical address of the same sensor data buffer stored in the shared storage (450) as an input tensor of the AI ​​model, and through the AI ​​accelerator driver, the second processor directly accesses the object detection AI model without CPU copying to detect the second region. The detection result is written to the shared storage (450). Without being limited to the described and / or illustrated examples, the operation 1305 and the operation 1307 are performed in parallel such that the first processor and the second processor simultaneously directly access the same physical memory region of the shared storage (450).

[0134] According to various embodiments, the wearable device (1) can de-identify a specific region associated with both a first region and a second region by using at least one third processor (e.g., GPU (15b)) among a plurality of processors in operation 1309. The third processor refers together to the first region result and the second region detection result of the shared storage (450) through the graphics runtime, performs mosaic or blur processing on the specific region based on the two regions, and writes the final result buffer to the shared storage (450). Subsequently, the de-identification processing is integrated and completed as part of an End-to-End Zero-copy pipeline by the buffer transfer interface directly transferring the final result buffer to an output encoder without CPU copying and encoding.

[0135] FIG. 14 is an illustrative diagram for explaining the results of the de-identification processing of FIG. 13 according to various embodiments.

[0136] Referring to FIG. 14, region (R1) represents an example of a first region identified by the first processor through color-based rule processing in operation 1305, region (R2) represents an example of a second region identified by the second processor through an AI model in operation 1307, and region (R3) represents an example in which de-identification processing by operation 1309 is performed on a specific region associated with both the first region and the second region. Not limited to the described and / or illustrated examples, de-identification processing methods may be applied in various ways, such as mosaic, blur, and masking.

Claims

Claim 1 A method of operating a wearable device comprises: an operation of acquiring at least one image using a camera; an operation of setting a work type for each of a plurality of processors for processing the at least one image; and an operation of identifying at least one first region corresponding to a specific color for the acquired at least one image using at least one first processor among the plurality of processors. The operation of identifying at least one second region using an object identification AI model for at least one image using at least one second processor among the plurality of processors; the operation of identifying a specific region associated with both the first region and the second region using at least one third processor among the plurality of processors, storing information about the identified specific region in a specific storage, and performing de-identification of the specific region; the operation method wherein the operation type of each of the plurality of processors is set based on a resource scheduling module implemented in a hardware abstraction layer supported by the operating system of the wearable device, and the specific storage is controlled to enable access by the at least one first processor, the at least one second processor, and the at least one third processor based on a shared storage management module implemented in a driver layer supported by the operating system. Claim 2 A method of operation according to claim 1, wherein the at least one first processor includes an ISP and a GPU, and the operation of processing the at least one acquired image using the at least one first processor comprises: an operation of allocating a sensor data buffer output from the ISP to a specific storage accessible to other processors using the shared storage management module and creating a memory handle; an operation of converting the memory handle into an image handle object of the graphics processing layer using the image handle conversion module; and an operation of directly binding the image handle object to an external texture of the GPU using a texture binding interface so that the GPU directly accesses and processes the sensor data of the specific storage without CPU copying. Claim 3 In claim 2, the at least one second processor includes an NPU and the GPU, and the operation of obtaining result information by processing at least one processed image stored in the specific storage based on an AI model comprises: an operation of providing the sensor data buffer of the specific storage as an input tensor of the AI ​​model using an AI runtime; an operation of causing the NPU to directly access the sensor data buffer without CPU copying using an AI accelerator driver to perform AI model inference; an operation of obtaining result information by performing the operation of the GPU performing processing on the same buffer of the specific storage in parallel; and an operation of directly transmitting the result information to an output encoder without CPU copying through a buffer transfer interface to output it. Claim 4 In claim 3, the shared storage management module is implemented as a driver that supports cross-device buffer sharing between a plurality of heterogeneous processors, the image handle conversion module is implemented as a graphics extension interface that provides mutual image handle conversion between a graphics processing layer and a native memory layer, the texture binding interface is implemented as a graphics processing interface that directly connects an external memory area to a GPU texture, and the buffer transfer interface is implemented as a native window interface that directly connects a GPU processing result buffer to an input buffer of an output encoder. Claim 5 A method of operation according to claim 4, wherein the shared storage management module is implemented as an ION Driver or a DMA-BUF (Direct Memory Access Buffer) driver, the graphics extension interface is implemented as an eglCreateImageKHR function, the graphics processing interface is implemented as a glEGLImageTargetTexture2DOES function and an OES External Texture extension, and the native window interface is implemented as a Surface or ANativeWindow. Claim 6 In claim 1, the operation of setting the task type of each of the plurality of processors for processing at least one image is performed based on a resource schedule database, and the resource schedule database includes mapping information such that sensor data acquisition and sensor synchronization are performed by a CPU and an ISP, AI-based object detection and de-identification inference are performed by an NPU and a GPU, image synthesis is performed by a CPU and a GPU, OSD synthesis is performed by a CPU, and output encoding is performed by a hardware encoder. Claim 7 In claim 6, the operation of setting the work type of each of the plurality of processors for processing at least one image comprises: an operation of identifying the state of the wearable device; wherein the state of the wearable device includes at least one of the processor temperature, battery level, processor usage rate, or output encoder queue state in real time; and a method of operation comprising dynamically adjusting the processing amount for each of the work types set for each of the plurality of processors based on the state of the wearable device. Claim 8 In claim 7, the operation of dynamically adjusting the throughput for each task type comprises: an operation of outputting a user warning and reducing the AI ​​inference execution cycle by a first ratio of a reference value when the temperature of a specific processor among the plurality of processors exceeds a first threshold; an operation of further reducing the AI ​​inference execution cycle by a second ratio lower than the first ratio and completely reducing the rendering post-processing step when the temperature of a specific processor among the plurality of processors exceeds a second threshold higher than the first threshold; and an operation of performing a forced system shutdown while maintaining sensor data acquisition and storage when the temperature of a specific processor among the plurality of processors exceeds a third threshold higher than the second threshold, and then recovering the throughput stepwise according to a hysteresis structure after the processor temperature has decreased below a recovery threshold. Claim 9 As a wearable device, a camera; A wearable device comprising: a plurality of processors; wherein at least one image is acquired using a camera, and a task type is set for each of the plurality of processors to process the at least one image; wherein at least one first processor among the plurality of processors is used to identify at least one first region corresponding to a specific color for the acquired at least one image; wherein at least one second processor among the plurality of processors is used to identify at least one second region using an object identification AI model for the at least one image; wherein at least one third processor among the plurality of processors is used to identify a specific region associated with both the first region and the second region, and information regarding the identified specific region is stored in a specific storage, and control is configured to perform de-identification of the specific region; wherein the task type of each of the plurality of processors is set based on a resource scheduling module implemented in a hardware abstraction layer supported by an operating system of the wearable device, and wherein the specific storage is controlled to enable access by the at least one first processor, the at least one second processor, and the at least one third processor based on a shared storage management module implemented in a driver layer supported by the operating system. Claim 10 In claim 9, the at least one first processor comprises an ISP and a GPU, and the wearable device is configured to use the at least one first processor to, as at least part of the operation of processing the at least one acquired image: allocating a sensor data buffer output from the ISP to the specific storage accessible to another processor using the shared storage management module and creating a memory handle, converting the memory handle to an image handle object of the graphics processing layer using the image handle conversion module, and directly binding the image handle object to an external texture of the GPU using a texture binding interface, thereby controlling the GPU to directly access and process the sensor data of the specific storage without CPU copying. Claim 11 In claim 10, the at least one second processor includes an NPU and the GPU, and the wearable device is configured to obtain result information by processing at least one processed image stored in the specific storage using the at least one second processor based on an AI model as at least part of the operation of: providing the sensor data buffer of the specific storage as an input tensor of the AI ​​model using an AI runtime, enabling the NPU to perform AI model inference by directly accessing the sensor data buffer without CPU copying using an AI accelerator driver, and obtaining result information by performing the operation in parallel in which the GPU performs processing on the same buffer of the specific storage, and to output the result information by directly transmitting it to an output encoder without CPU copying through a buffer transfer interface. Claim 12 A wearable device according to claim 11, wherein the shared storage management module is implemented as a driver that supports cross-device buffer sharing between a plurality of heterogeneous processors, the image handle conversion module is implemented as a graphics extension interface that provides mutual image handle conversion between a graphics processing layer and a native memory layer, the texture binding interface is implemented as a graphics processing interface that directly connects an external memory area to a GPU texture, and the buffer transfer interface is implemented as a native window interface that directly connects a GPU processing result buffer to an input buffer of an output encoder. Claim 13 A wearable device according to claim 12, wherein the shared storage management module is implemented as an ION Driver or a DMA-BUF (Direct Memory Access Buffer) driver, the graphics extension interface is implemented as an eglCreateImageKHR function, the graphics processing interface is implemented as a glEGLImageTargetTexture2DOES function and an OES External Texture extension, and the native window interface is implemented as a Surface or ANativeWindow. Claim 14 In claim 9, the operation of setting the task type of each of the plurality of processors for processing at least one image is performed based on a resource schedule database, and the resource schedule database includes mapping information such that sensor data acquisition and sensor synchronization are performed by a CPU and an ISP, AI-based object detection and de-identification inference are performed by an NPU and a GPU, image synthesis is performed by a CPU and a GPU, OSD synthesis is performed by a CPU, and output encoding is performed by a hardware encoder, and the wearable device. Claim 15 delete

Images