Method and apparatus for processing an image

Abstract

A method and apparatus for processing an image are disclosed. The method includes extracting a feature of a current local processing region of an input image frame by inputting pixel data of the current local processing region into a convolutional neural network (CNN), updating a hidden state of a recurrent neural network (RNN) regarding a context between the current local processing region and at least one previous local processing region by inputting the extracted feature into the RNN, and generating an image processing result of the input image frame based on the updated hidden state.

Description

[0001] This application claims priority to Korean Patent Application No. 10-2020-0110577, filed with the Korean Intellectual Property Office on August 31, 2020, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to methods and apparatus for image processing based on local images. Background Technology

[0003] The automation of recognition processing has been achieved through processor-implemented neural network models that, after extensive training, provide a computationally intuitive mapping between input and output patterns. Therefore, for example, such specially trained neural networks can generalize to generate relatively accurate outputs for input patterns that the neural network may not have been trained on. Summary of the Invention

[0004] This summary is provided to introduce, in a simplified form, the selection of concepts that will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0005] In one general aspect, a method for processing an image includes: extracting features of a current local processing region by feeding pixel data of a current local processing region of an input image frame into a convolutional neural network (CNN); updating the hidden state of the RNN with respect to the context between the current local processing region and at least one previous local processing region by feeding the extracted features into a recurrent neural network (RNN); and generating an image processing result of the input image frame based on the updated hidden state.

[0006] The CNN can be a multi-head CNN (MCNN). The extraction steps may include: extracting multiple intermediate features of the current local processing region by inputting pixel data of the current local processing region into each head of the MCNN, and generating encoded features of the current local processing region by fusing the extracted multiple intermediate features.

[0007] The update steps may include: converting the extracted features into a one-dimensional vector and inputting the one-dimensional vector into the RNN. The one-dimensional vector may include at least one of a channel direction vector and a width direction vector.

[0008] The generation steps may include: generating final context data based on the updated hidden state in response to the current local processing region corresponding to the final local processing region of the input image frame; and generating an image processing result based on the final context data. The image processing result may indicate whether a target object exists in the input image frame. The target object may be a person or a face.

[0009] The generation steps may include: generating intermediate context data based on the updated hidden state in response to the current local processing region not corresponding to the final local processing region of the input image frame, and generating intermediate detection results based on the intermediate context data. In response to the intermediate detection results indicating the presence of a target object in the input image frame, the image processing procedure for the input image frame may be terminated without performing additional image processing on at least one subsequent local processing region.

[0010] Each local processing region, including the current local processing region, can correspond to a group of pixel rows used for image signal processing (ISP) on an image sensor. Extraction, updating, and generation can be performed in low-power mode. The CNN can be pre-trained using object location information associated with each local processing region of the training image frame.

[0011] In another general aspect, an apparatus for processing an image includes a processor and a memory configured to store instructions executable by the processor, wherein, in response to the instructions being executed by the processor, the processor is configured to: extract features of the current local processing region by inputting pixel data of the current local processing region of an input image frame into a CNN; update the hidden state of the RNN with respect to the context between the current local processing region and at least one previous local processing region by inputting the extracted features into an RNN; and generate an image processing result of the input image frame based on the updated hidden state.

[0012] In another general aspect, an electronic device includes: a camera configured to generate an input image frame; a memory configured to store instructions; and a processor configured to execute the instructions stored in the memory to: extract features of the current local processing region by inputting pixel data of the current local processing region of the input image frame into a CNN; update the hidden state of the RNN with respect to the context between the current local processing region and at least one previous local processing region by inputting the extracted features into an RNN; and generate an image processing result of the input image frame based on the updated hidden state.

[0013] Extraction, updating, and generation can be performed in low-power mode, and the electronic device can be woken up in response to the detection of a target object in the input image frame to perform operations associated with image processing in normal mode.

[0014] Low-power modes may include an always-on mode. Image processing may include object detection, and operations associated with image processing may include any one or any combination of object tracking, object recognition, and unlocking.

[0015] In another general aspect, an apparatus includes a processor and a memory configured to store instructions executable by the processor, wherein, in response to the instructions being executed by the processor, the processor is configured to: extract features of a current local processing region of an input image frame using a convolutional neural network (CNN); determine whether the current local processing region of the input image frame is a final local processing region of the input image frame; if the current local processing region is not a final local processing region, repeatedly update the hidden state of the RNN by inputting the extracted features of the current local processing region into a recurrent neural network (RNN); generate intermediate context data based on the updated hidden state; and generate intermediate detection results based on the intermediate context data until the current local processing region is a final local processing region; and if the current local processing region is a final local processing region, update the hidden state of the RNN by inputting the extracted features of the current local processing region into the RNN; generate final context data based on the updated hidden state; and generate an image processing result of the input image frame based on the final context data.

[0016] After each iteration where the current local processing region is not the final local processing region, the processor can determine whether the intermediate detection result indicates the presence of a target object in the input image frame, and if the intermediate detection result indicates the presence of a target object in the input image frame, terminate any further processing of the input image frame.

[0017] In another general aspect, an apparatus includes a processor and a memory configured to store instructions executable by the processor, wherein, in response to the instructions being executed by the processor, the processor is configured to: extract features of a current local processing region of an input data frame using a convolutional neural network (CNN); update the hidden state of the RNN by inputting the extracted features into a recurrent neural network (RNN); generate context data based on the updated hidden state; generate a detection result based on the context data; determine whether the detection result indicates the presence of target data in the input data frame; terminate any further processing of the input data frame if the detection result indicates the presence of target data in the input data frame; and continue processing of the input data frame for a local processing region following the current local processing region if the detection result does not indicate the presence of target data in the input data frame.

[0018] If the detection result does not indicate the presence of target data in the input data frame and the current local processing region is the final local processing region of the input data frame, the processor can generate the image processing result of the input data frame based on the context data.

[0019] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0020] Figure 1 An example of the operation of an image processing device is shown.

[0021] Figure 2 An example of time-series image processing for each local image unit is shown.

[0022] Figure 3 An example of generating intermediate features using a multi-head convolutional neural network (MCNN) is shown.

[0023] Figure 4 This illustrates an example of generating encoded features by fusing intermediate features.

[0024] Figure 5 An example of rearranging coded features is shown.

[0025] Figure 6 An example of generating image processing results through object detection is shown.

[0026] Figure 7 This example illustrates how intermediate detection results are generated through object detection processing.

[0027] Figure 8 An example of training an object detection model is shown.

[0028] Figure 9 An example of training an object detection model is shown.

[0029] Figure 10 An example of training an object detection model is shown.

[0030] Figure 11 An example configuration of an image processing device is shown.

[0031] Figure 12 An example of the configuration of an electronic device is shown.

[0032] Figure 13 An example of an image processing method is shown.

[0033] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation

[0034] The examples will be described in detail below with reference to the accompanying drawings. However, various changes and modifications can be made to the examples. Here, the examples are not to be construed as limited to the disclosure. The examples should be understood to include all changes, equivalents, and substitutions within the scope of the disclosed concepts and techniques.

[0035] The terminology used herein is for the purpose of describing specific examples only and is not intended to limit the examples. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. It will also be understood that when the terms “comprising” and / or “including” are used herein, it indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups thereof.

[0036] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by a person skilled in the art to which the examples pertain. It will also be understood that, unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formalized manner.

[0037] When describing the examples with reference to the accompanying drawings, the same reference numerals denote the same constituent elements, and repetitive descriptions associated with them will be omitted. In the description of the examples, descriptions of well-known related structures or functions will be omitted where such detailed descriptions would lead to a vague interpretation of this disclosure.

[0038] Furthermore, in the description of components, when describing components of this disclosure, terms such as first, second, A, B, (A), (B), etc., may be used herein. These terms are used only to distinguish one constituent element from another, and the nature, sequence, or order of the constituent elements is not limited by these terms. When a constituent element is described as being "connected," "bonded," or "attached" to another constituent element, it should be understood that the one constituent element may be directly connected to or attached to the other constituent element, and intermediate constituent elements may also be "connected," "bonded," or "attached" to the constituent element.

[0039] The same names can be used to describe components included in the examples above and components with common functions. Unless otherwise stated, the descriptions of the examples are applicable to the examples below, therefore, for the sake of brevity, repeated descriptions will be omitted.

[0040] Figure 1 An example of the operation of an image processing device is shown. (Refer to...) Figure 1 The image processing device 110 performs image processing on the input image frame 120 and outputs the image processing results accordingly. Image processing may include recognition processing of the image frame 120. For example, image processing may include various recognition processes of the image frame 120 that can be processed by an image sensor (e.g., object detection to determine whether an object exists in the image frame 120 or object classification to determine the category of an object present in the image frame 120). Hereinafter, an example of object detection will be described representatively with respect to image processing. However, this description can also be applied to other types of recognition processing.

[0041] The input image frame 120 may correspond to a frame of the input image. For example, the input image may include moving images (videos) and / or still images, which may be captured by a camera in real time, or captured by a camera in the past and stored in a storage device or received from another device via a communication function. The input image frame 120 may be divided into local processing regions 121 to 12K, where K is an integer greater than 1. Each of the local processing regions 121 to 12K may correspond to a local image unit. The image processing device 110 may process the local processing regions 121 to 12K of the input image frame 120 sequentially. For example, the image processing device 110 may process local processing region 121 at time T(1), local processing region 122 at time T(2), local processing region 123 at time T(3), and local processing region 12K at time T(K).

[0042] Image processing device 110 may include a convolutional neural network (CNN) 111 and a recurrent neural network (RNN) 112. Image processing device 110 can perform recognition processing on input data by mapping input data and output data that are in a non-linear relationship based on deep learning.

[0043] Deep learning is a machine learning technique used to solve problems such as image recognition or speech recognition from large datasets. Deep learning is explained as an optimization problem-solving process that finds the point where energy is minimized while training a neural network using prepared training data. Through supervised or unsupervised learning in deep learning, the structure of the neural network, or the weights corresponding to the model, can be obtained, and input and output data can be mapped to each other through these weights.

[0044] Neural networks can be trained on training data during training operations and perform inference operations (such as classification, recognition, or detection) related to the input data during inference operations. Neural networks can be pre-trained to perform operations for a specific purpose. Here, "pre" can mean before the neural network is "started." A "started" neural network indicates that the neural network is ready to perform inference. For example, a "started" neural network may include the neural network being loaded into memory, or the input data for inference being fed into the neural network after it has been loaded into memory.

[0045] Neural networks can correspond to deep neural networks (DNNs) that consist of multiple layers. These layers include an input layer, hidden layers, and an output layer. Various types of neural networks can exist (such as fully connected networks (FCNs), CNNs, RNNs, etc.). Neural network models including CNN 111 and RNN 112 can be referred to as image processing models.

[0046] The data input to each layer in a CNN 111 can be called an input feature map, and the data output from each layer can be called an output feature map. Input and output feature maps can also be referred to as activation data. In the input layer, the input feature map corresponds to the input data.

[0047] RNN 112 can be implemented using a recurrent deep neural network (RDNN) and / or a bidirectional recurrent deep neural network (BRDNN). Typically, RDNN and BRDNN are suitable for recognizing continuous data (e.g., speech) determined by the passage of time. For example, an RDNN can consider previous speech frames to recognize the current speech frame, while a BRDNN can consider previous and subsequent speech frames to recognize the current speech frame. Image processing device 110 can use this characteristic of RNN 112 to determine the relationships between pixel data in local processing regions 121 to 12K.

[0048] Recursive models (such as RNN 112) may have regression loops. When the output of a recursive model is fed back into the recursive model, a new output can be generated from the input. For example, a recursive model may include nodes (or units) and may have a structure where the output of a node is fed back into the node. Recursive models may include Long Short-Term Memory (LSTM) and / or Gated Recursive Units (GRU). However, LSTM and GRU are merely examples, and the structure of recursive models is not limited to these.

[0049] Image processing device 110 can sequentially process pixel data of local processing regions 121 to 12K using CNN 111 and RNN 112. For example, CNN 111 can be used to extract features from the pixel data of local processing regions 121 to 12K, and RNN 112 can be used to determine the context between local processing regions 121 to 12K. The context may include visual and geometric relationships between pixels in local processing regions 121 to 12K.

[0050] For example, image processing device 110 can extract features of local processing region 122 by inputting pixel data of local processing region 122 into CNN 111 at time T(2), and update the hidden state of RNN 112 by inputting the extracted features into RNN 112. The hidden state of RNN 112 can reflect the context between the currently processed local processing region (hereinafter referred to as the current local processing region) and the previously processed local processing region (hereinafter referred to as the previous local processing region). In the example above, local processing region 122 corresponds to the current local processing region. Therefore, the hidden state of RNN 112 can represent the context between local processing region 121 and local processing region 122.

[0051] Image processing device 110 can generate image processing results based on the hidden state of RNN 112. Target objects may include various objects that can be detected or classified through an image (e.g., a person, a person's body, face, eyes, iris, etc.). Image processing may be object detection used to determine whether a target object exists in an input image, and the object detection result may indicate the presence or absence of an object in the input image.

[0052] Image processing device 110 can generate an image processing result after processing all local processing regions 121 to 12K. Optionally, if a local region clearly indicates the presence of an object in the input image frame 120, image processing device 110 can generate an image processing result after processing a portion of the local processing regions 121 to 12K. For example, if the target object is a face, since local processing region 121 includes the face, image processing device 110 can process local processing region 121 and then generate an image processing result without processing the other local processing regions 122 to 12K.

[0053] When image processing operations are performed on each of the local processing regions 121 to 12K as described above, memory can be used for each local processing region instead of the entire frame, thus reducing memory usage. For example, if the entire frame is to be processed, memory is needed to store the input feature map corresponding to the entire frame, while if each local processing region is to be processed, only memory is needed to store the input feature map corresponding to each local processing region. Even when data other than the input feature map is to be processed for object recognition, memory usage can be reduced. In addition, since image processing operations can begin immediately when pixel data for each local processing region is collected, latency can be reduced compared to conventional methods that begin image processing operations when pixel data for the entire frame is collected.

[0054] Such image processing can be referred to as local image unit-based image processing and can be used in environments with limited memory usage (e.g., in low-power mode). For example, the resources available in low-power mode may be limited to the resources within the sensor, rather than all the resources of the device equipped with the sensor. Such an environment, where available resources are limited to sensor resources, can be referred to as an on-sensor environment.

[0055] Low-power modes may include always-on mode. In always-on mode, a portion of the device screen may always be on to provide predetermined information (e.g., date, time, simple device status information, etc.). For example, the power consumption for driving always-on mode may be limited to approximately 1 milliwatt (mW), and this power consumption may be equivalent to maintaining approximately 40 kilobytes (KB) of model and code in memory. Therefore, there may be a need to implement image processing (such as object detection) using only approximately 40 KB of memory while driving always-on mode. The local image unit-based image processing described in the example can be used to perform image processing in such a limited environment.

[0056] Each local processing region may include one or more rows of pixels. For example, when the size of the input image frame 120 is represented as W×H, the row of pixels may correspond to a vector with a size of W×1.

[0057] Image processing can be technically combined with image signal processing (ISP) algorithms of the image sensor. For example, pixel rows can be generated sequentially by an image signal processing front-end (ISP FE), and ISP can be performed on the generated pixel rows when a predetermined number of pixel rows (e.g., 5 pixel rows) are generated. For ease of description, the predetermined number of pixel rows can be referred to as a pixel row group. The ISP FE can perform analog-to-digital conversion (ADC), autofocus (AF) pixel indication, bad pixel (BP) indication, resolution reduction, signal characteristic correction, etc. ISP may include bit reduction, BP processing, noise removal, pixel binning, artifact removal, resolution reduction, etc.

[0058] As pixel row groups are formed through ISP FE and ISP processing is performed on the pixel row groups iteratively, ISP can be performed on all pixel row groups, thus generating a single image frame. Frame-based image processing can be performed after a single image frame is generated through the ISP algorithm described above. Local image unit-based image processing can be performed on at least one pixel row (e.g., each pixel row group).

[0059] Figure 2 An example of time-series image processing for each local image unit is shown. (See reference...) Figure 2 The image processing device processes the pixel data of the local processing region 211 at time T(1). Specifically, the image processing device can extract features of the local processing region 211 by inputting the pixel data of the local processing region 211 into a CNN, and update the hidden state of the RNN by inputting the extracted features into the RNN.

[0060] Similarly, the image processing device can process the pixel data of local processing region 212 at time T(2) and the pixel data of local processing region 21K at time T(K). Boxes 221, 222 and 223 illustrate the processing of pixel data of local processing regions 211 to 21K.

[0061] Local processing region 21K corresponds to the final local processing region of input image frame 210. Therefore, the image processing device can generate an image processing result based on the final hidden state updated via box 223. For example, the image processing device can generate the image processing result by inputting the final context data corresponding to the final hidden state into the FCN. Box 224 illustrates the process of generating the image processing result.

[0062] Figure 3 This illustrates an example of generating intermediate features using a multi-head convolutional neural network (MCNN). (See reference...) Figure 3MCNN 320 comprises sub-CNNs corresponding to multiple heads. The number of sub-CNNs and the number of heads can be defined as M. Each sub-CNN may include layers 1 through N, and CNN-related operations (such as convolution, pooling, etc.) may be performed in each layer. For example, the values ​​of M and N may be determined based on system specifications (such as memory availability).

[0063] Sub-CNNs can be trained with different network parameters and extract features from the input data based on these different parameters. Therefore, even with the same input data, sub-CNNs can extract features from different aspects, and there can be diversity among the features extracted by sub-CNNs.

[0064] The image processing device can extract multiple intermediate features 330 by inputting pixel data 310 of the current local processing region into each head of the MCNN 320. The image processing device can generate encoded features of the current local processing region by fusing the intermediate features 330. The process of extracting features from each local processing region as described above can be considered as an encoding process. Therefore, the features extracted from each local processing region can be called encoded features.

[0065] Figure 4 This illustrates an example of generating encoded features by fusing intermediate features. (See reference...) Figure 4 The image processing device can generate encoded features 430 by fusing intermediate features 410. Box 420 illustrates the feature fusion process, which may include a concatenation process 421 and a feature projection process 422. Feature projection may include convolution operations and may be performed based on the direction of concatenation. For example, if intermediate features 410 are concatenated in the channel direction, a feature survey may be performed, such that the size in the channel direction can be reduced. As a result of concatenation and feature projection, the size of encoded feature 430 may be the same as the size of each intermediate feature 410.

[0066] Figure 5 An example of rearranging encoded features is shown. As described above, an image processing device can extract features from pixel data in each local processing region and input the extracted features into an RNN. In this example, the extracted features can be converted into a one-dimensional (1D) vector and then input into the RNN. This transformation process can be called rearrangement or reshape. Memory can be saved by using a 1D vector-based RNN instead of a multi-dimensional (2D or 3D) vector-based RNN.

[0067] Reference Figure 5As can be seen, the training characteristics change depending on the rearrangement direction. For example, if the encoded feature 511 is converted into a 1D channel direction vector 512 and input into the RNN, the RNN can reflect the context in the channel vector direction. If the encoded feature 521 is converted into a 1D width direction vector 522 and input into the RNN, the RNN can reflect the context in the width vector direction. If the encoded feature 531 is converted into both a 1D channel direction vector 532 and a 1D width direction vector 533 and input into the RNN, the RNN can reflect the context in both directions (i.e., the channel direction and the width direction). If both directions are considered as described above, a larger memory capacity may be required compared to considering only one direction.

[0068] Figure 6 This illustrates an example of image processing results generated through object detection. (See reference...) Figure 6 At time T(1), the image processing device extracts features 612 of the first local processing region by inputting pixel data 611 of the first local processing region into the CNN, and updates the hidden state of the RNN by converting the features 612 into a 1D vector 613 and inputting the 1D vector 613 into the RNN. Features 612 may correspond to encoded features, and the hidden vector 614 may correspond to the hidden state of the RNN updated at time T(1). Figure 6 In the example, feature 612 is rearranged in the channel direction, but it can also be rearranged in other directions.

[0069] At time T(2), the image processing device can extract features 622 of the second local processing region by inputting pixel data 621 of the second local processing region into the CNN, and update the hidden vector 624 corresponding to the hidden state of the RNN by converting the features 622 into a 1D vector 623 and inputting the 1D vector 623 into the RNN. The hidden state of the RNN can be updated based on the input data and the hidden state at a previous time. For example, the hidden state at time T(2) can be updated based on the 1D vector 623 and the hidden state at time T(1). The hidden state at a previous time in time T(1) is "0". Therefore, the hidden state at time T(1) can be considered to be updated based on the 1D vector 613 and the zero state.

[0070] At time T(K), the image processing device can extract features 632 of the Kth local processing region by inputting the pixel data 631 of the Kth local processing region into the CNN, and update the hidden vector 634 corresponding to the hidden state of the RNN by converting the features 632 into a 1D vector 633 and inputting the 1D vector 633 into the RNN. If the current local processing region corresponds to the final local processing region of the input image frame, the image processing device can generate final context data based on the final hidden state. Figure 6 In the example, the Kth local processing region corresponds to the final local processing region. Therefore, the image processing device can generate a hidden vector 634 corresponding to the final hidden state updated at time T(K). The hidden vector 634 corresponds to the final context data.

[0071] The image processing device can generate an image processing result based on the final context data. For example, the image processing device can input the final context data into the FCN and generate an image processing result based on the output from the FCN. The image processing device can appropriately transform the final context data (e.g., flatten the final context data) and then input the transformed context data into the FCN.

[0072] If image processing includes object detection, category "1" can indicate the presence of an object in the input image, and category "0" can indicate the absence of an object in the input image. The FCN can include output values ​​for nodes indicating the probability of category "1" and output values ​​for nodes indicating the probability of category "0," and can output the category with the higher probability value as the image processing result. For example, if the probability value of category "1" is greater than the probability value of category "0," the image processing result indicating the presence of an object in the input image can be output.

[0073] As another example, if image processing includes object classification, category "1" might indicate the presence of a first object (e.g., a cat) in the input image, and category "0" might indicate the presence of a second object (e.g., a dog). For instance, if the probability value of category "1" is greater than the probability value of category "0," the image processing result indicating the presence of the first object (e.g., a cat) in the input image can be output. A greater number of categories can be defined for object classification compared to object detection.

[0074] Figure 7 This illustrates an example of generating intermediate detection results through object detection processing. Figure 6 The examples are different, in Figure 7 In the example, intermediate detection results can be generated before processing the final local processing region. Specifically, at time T(1), the image processing device extracts features 712 by inputting the pixel data 711 of the first local processing region into the CNN, and updates the hidden state of the RNN by converting the features 712 into a 1D vector 713 and inputting the 1D vector 713 into the RNN.

[0075] The image processing device can generate intermediate context data from the hidden vector 714 corresponding to the updated hidden state, and generate intermediate detection results by inputting the intermediate context data into the FCN. The intermediate detection results can indicate whether a target object exists in the input image, and can be determined based on the intermediate context data. For object classification, intermediate classification results indicating the category of the object present in the input image can be generated.

[0076] If intermediate detection results indicate the presence of a target object in the input image frame, object detection processing for the input image frame can be terminated without performing additional object detection processing on subsequent local processing regions. This is because if a portion of the local processing region indicates the presence of an object in the input image frame, it is unnecessary to analyze the remaining local processing region. If an object is detected through intermediate detection results as described above, the time spent on object detection can be reduced.

[0077] If the intermediate detection results do not indicate the presence of a target object in the input image frame, subsequent local processing regions can be processed. Specifically, at time T(2), the image processing device can extract features 722 by inputting the pixel data 721 of the second local processing region into the CNN, and update the hidden state of the RNN by converting the features 722 into a 1D vector 723 and inputting the 1D vector 723 into the RNN. The FCN can receive the hidden vector 724 corresponding to the intermediate context data and generate intermediate detection results. If the intermediate detection results indicate the presence of a target object in the input image frame, the object detection processing of the input image frame can be terminated. If the intermediate detection results do not indicate the presence of a target object in the input image frame, subsequent local processing regions can be processed.

[0078] This process can be repeated up to time T(K). If the intermediate detection results up to time T(K-1) do not indicate the presence of a target object in the input image frame, then the Kth local processing region at time T(K) can be processed. Figure 6 The same approach is used to process the Kth local processing region at time T(K). Specifically, the image processing device can extract features 732 by inputting the pixel data 731 of the Kth local processing region into the CNN, and update the hidden state of the RNN by converting the features 732 into a 1D vector 733 and inputting the 1D vector 733 into the RNN. The image processing device can generate final context data from the hidden vector 734 corresponding to the hidden state updated at time T(K), and generate the image processing result by inputting the final context data into the FCN.

[0079] Figure 8An example of training an object detection model is shown. A neural network model can be called an image processing model in the sense of performing image processing, and also an object detection model in the sense of performing object detection. In the following text, the object detection model will be described representatively with respect to image processing models. However, this description can also be applied to various other models (such as object classification models).

[0080] like Figure 8 As shown, object detection models can include CNNs, RNNs, and FCNs. For example, an object detection model can be trained in a supervised manner based on training data. Training data can include training images and labels. Training images can be various images that include or exclude the target object to be detected, and labels can each indicate whether the target object to be detected exists in the corresponding training image.

[0081] Reference Figure 8 When pixel data 811, 821, and 831 are processed by CNN, RNN, ..., T(K) at times T(1), T(2), ..., T(K), a final category "1" or "0" can be output. A final category "1" indicates the presence of an object in the training image, while a final category "0" indicates the absence of an object in the training image. The object detection model can be trained to output the final category indicated by the label in this way. As mentioned above, a greater number of categories can be defined for object classification.

[0082] The object detection model can be trained using a loss function Loss1 based on the final loss. For example, the loss function Loss1 can be represented by Equation 1.

[0083] [Equation 1]

[0084] Loss1 = Final_Loss

[0085] In Equation 1, Final_Loss represents the difference between the final category and the label. The output from the FCN can include the probability value for the final category "1" and the probability value for the final category "0," and Final_Loss can be determined based on the difference between each output value and the label. The object detection model can be trained to minimize this loss function, Loss1. For example, it can be based on... Figure 8 Example trained object detection models to perform Figure 6 Image processing.

[0086] Figure 9 An example of training an object detection model is shown. (See reference...) Figure 9The image processing model has a structure for outputting intermediate categories. When pixel data 911 and 921 are processed by CNN, RNN, and FCN at times T(1) and T(2), intermediate categories can be output. Additionally, when pixel data 931 is processed by CNN, RNN, and FCN at time T(K), the final category can be output. The image processing model can be trained based on a loss function Loss2 based on the intermediate categories. For example, the loss function Loss2 can be represented by Equation 2.

[0087] [Equation 2]

[0088] Loss2=Final_Loss+α∑Inter_Loss

[0089] In Equation 2, Final_Loss represents the difference between the final class and the final label, and Inter_Loss represents the difference between intermediate classes and intermediate labels. For example, the process for calculating Final_Loss described in Equation 1 can be applied to the calculation of Final_Loss and Inter_Loss in Equation 2. α represents the weight associated with Inter_Loss, and α can be set to a value less than "1". The object detection model can be trained to minimize this loss function, Loss2. For example, it can be based on... Figure 9 Example trained object detection models to perform Figure 7 Image processing.

[0090] according to Figure 9 For example, training data may include final labels and intermediate labels as labels for each training image. Intermediate labels can be used for comparison with their respective intermediate categories and can indicate whether the presence of an object can be determined using the current context. Final labels can indicate whether an object exists in the corresponding training image. For example, when a target object exists in a first local processing region of a training image, the first intermediate label can indicate intermediate category "1". The first intermediate label can be an intermediate label corresponding to pixel data 911 of the first local processing region, and FCN 912 can perform intermediate detection on pixel data 911. FCN 912 may include the output values ​​of nodes indicating the probability of intermediate category "1" and nodes indicating the probability of intermediate category "0". In this case, the first intermediate label indicating intermediate category "1" can have a label value "1" and a label value "0", where label value "1" corresponds to the output value of the node indicating the probability of intermediate category "1" in FCN 912, and label value "0" corresponds to the output value of the node indicating the probability of intermediate category "0" in FCN 912.

[0091] As another example, if a target object exists in both a first local processing region and a second local processing region of the training image, either the first intermediate label or the second intermediate label may indicate an intermediate category "1". If it is highly likely that a target object exists in the training image through the first local processing region, the first intermediate label may indicate "1". Conversely, if it is necessary to examine both the first and second local processing regions to determine the presence of a target object in the training image, the first intermediate label may indicate "0" and the second intermediate label may indicate "1".

[0092] Figure 10 An example of training an object detection model is shown. (See reference...) Figure 10 The object detection model includes training structures to improve the feature extraction performance of the CNN. For example, at time T(1), feature 1012 extracted from pixel data 1011 by the CNN is input into the RNN to update the hidden state of the RNN. In this process, feature 1012 is transformed into another feature 1013 through a convolutional block Conv. The obtained feature 1013 and the masked label 1014 may have the same size. Feature 1013 can be compared with the masked label 1014, and the difference between feature 1013 and masked label 1014 can be used to train the object detection model. Similarly, feature 1012 extracted from pixel data 1021 at time T(2) can also be transformed into feature 1023, and feature 1023 is compared with masked label 1024.

[0093] Each of mask labels 1014 and 1024 may include object location information. The object location information indicates the position of the object in each local processing region of the training image frame. For example, in mask label 1014, a white area may be a region where the object exists, and a black area may be a region where the object does not exist. Since mask label 1014 includes white areas while mask label 1024 does not, it can be seen that when targeting… Figure 10 The example uses an object located at the center of the first local processing region in the training image.

[0094] The object detection model can be trained to extract features from training images based on object location information in mask labels 1014 and 1024. Specifically, the object detection model can be trained to generate results indicating the presence of an object in a local processing region and / or frame determined to include the object, based on the object location information. For example, feature extraction of the center of the first local processing region can be enhanced when the object detection model is trained using mask labels 1014 and 1024.

[0095] An object detection model can be trained using a loss function, Loss3, based on object location information. For example, the loss function Loss3 can be represented by Equation 3.

[0096] [Equation 3]

[0097] Loss3=Final_Loss+·α∑Inter_Loss+β∑CNN_Loss

[0098] In Equation 3, Final_Loss represents the difference between the final class and the final label, Inter_Loss represents the difference between intermediate classes and intermediate labels, and CNN_Loss represents the difference between the deformed features and the mask label. α represents the weights associated with Inter_Loss, and β represents the weights associated with CNN_Loss. α can be set to a value less than "1", and β can be set to a value less than α.

[0099] Figure 11 An example configuration of an image processing device is shown. (See reference...) Figure 11 The image processing device 1100 includes a processor 1110 and a memory 1120. The memory 1120 is connected to the processor 1110 and stores instructions executable by the processor 1110, data to be computed by the processor 1110, or data processed by the processor 1110. The memory 1120 includes non-transitory computer-readable media (e.g., high-speed random access memory) and / or non-volatile computer-readable storage media (e.g., at least one disk storage device, flash memory device, or other non-volatile solid-state memory device).

[0100] Processor 1110 executes instructions to perform reference Figures 1 to 10 At least one operation described. For example, processor 1110 can extract features of the current local processing region by inputting pixel data of the current local processing region of the input image frame into a CNN, updating the hidden state of the RNN about the context between the current local processing region and the previous local processing region by inputting the extracted features into an RNN, and generating an image processing result of the input image frame based on the updated hidden state. Additionally, refer to... Figures 1 to 10 The provided description can be applied to image processing device 1100.

[0101] Figure 12 An example configuration of an electronic device is shown. (See reference...) Figure 12The electronic device 1200 can acquire an input image and detect target objects in the acquired input image. Furthermore, the electronic device 1200 can perform various operations associated with object detection (e.g., object tracking, object recognition, and unlocking). For example, the electronic device 1200 can perform object detection (e.g., face detection) in a low-power mode (such as an always-on mode) with power consumption limited to 1mW or less, and the electronic device 1200 can be woken up in response to object detection to perform operations associated with object detection in normal mode. The electronic device 1200 may structurally and / or functionally include... Figure 1 Image processing device 110.

[0102] Electronic device 1200 may include a processor 1210, a memory 1220, a camera 1230, a storage device 1240, an input device 1250, an output device 1260, and a network interface 1270 that can communicate with each other via a communication bus 1280. For example, electronic device 1200 may be implemented as part of a mobile device (such as a mobile phone, smartphone, PDA, netbook, tablet computer, or laptop computer), a wearable device (such as a smartwatch, smart bracelet, or smart glasses), a computing device (such as a desktop computer or server), a home appliance (such as a television, smart TV, or refrigerator), a security device (such as a door lock), or a vehicle (such as a smart vehicle).

[0103] Processor 1210 executes instructions or functions that will be performed in electronic device 1200. For example, processor 1210 may process instructions stored in memory 1220 or storage device 1240. Processor 1210 may execute instructions via... Figures 1 to 11 The described operation.

[0104] Memory 1220 stores data used for object detection. Memory 1220 may include a computer-readable storage medium or a computer-readable storage device. Memory 1220 may store instructions to be executed by processor 1210, and may store relevant information when electronic device 1200 executes software and / or applications.

[0105] Camera 1230 can capture photos and / or videos. For example, camera 1230 can capture images including the user's body and / or facial images including the user's face. Photos and / or videos captured by camera 1230 can be used as input images. Camera 1230 can provide 3D images including depth information related to objects.

[0106] Storage device 1240 includes a computer-readable storage medium or a computer-readable storage device. Storage device 1240 can store various types of data that will be used in object detection processes, such as object detectors and threshold determination engines. Compared to memory 1220, storage device 1240 can store a greater amount of information for a longer period of time. For example, storage device 1240 may include a magnetic hard disk, optical disk, flash memory, floppy disk, or other non-volatile memory known in the art.

[0107] Input device 1250 can receive input from a user via a keyboard and mouse in traditional input methods as well as in new input methods such as touch input, voice input, and image input. For example, input device 1250 may include a keyboard, mouse, touch screen, microphone, or any other device that detects input from the user and sends the detected input to electronic device 1200.

[0108] Output device 1260 provides the output of electronic device 1200 to a user through visual, auditory, or tactile channels. Output device 1260 may include, for example, a display, touch screen, speaker, vibration generator, or any other device that provides output to the user. Network interface 1270 can communicate with external devices via a wired or wireless network.

[0109] Figure 13 An example of an image processing method is shown. (See reference...) Figure 13 In operation 1310, the image processing device extracts features of the current local processing region by inputting pixel data of the current local processing region of the input image frame into a CNN. In operation 1320, the image processing device updates the hidden state of the RNN regarding the context between the current local processing region and previous local processing regions by inputting the extracted features into an RNN. In operation 1330, the image processing device generates the image processing result of the input image frame based on the updated hidden state. Additionally, refer to... Figures 1 to 12 The provided description can be applied to object detection methods.

[0110] The methods described in the examples above can be recorded in a non-transitory computer-readable medium comprising program instructions for implementing the various operations described in the examples. The medium may also comprise program instructions, data files, data structures, etc., individually, or a combination thereof. The program instructions recorded on the medium may be program instructions specifically designed and constructed for the purposes of the examples, or they may be of types known and available to those skilled in the art of computer software. Examples of non-transitory computer-readable media include: magnetic media (such as hard disks, floppy disks, and magnetic tapes); optical media (such as CD-ROMs, DVDs, and / or Blu-ray discs); magneto-optical media (such as optical discs); and hardware devices specifically configured to store and execute program instructions (such as read-only memory (ROM), random access memory (RAM), flash memory (e.g., USB flash drives, memory cards, memory sticks, etc.)). Examples of program instructions include both machine code generated by a compiler and files containing higher-level code that can be executed by a computer using an interpreter. The aforementioned devices may be configured as one or more software modules to perform the operations described in the examples, and vice versa.

[0111] Software may include computer programs, code segments, instructions, or one or more combinations thereof, to independently or uniformly instruct or configure a processing device to operate as needed. Software and data may be permanently or temporarily embodied in any type of machine, component, physical or virtual device, computer storage medium, or device, or in propagated signal waves capable of providing instructions or data to or being interpreted by the processing device. Software may also be distributed across network-coupled computer systems, enabling the software to be stored and executed in a distributed manner. Software and data may be stored on one or more non-transitory computer-readable recording media.

[0112] Many examples have been described above. However, it should be understood that various modifications can be made to these examples. For example, suitable results may be achieved if the described techniques are performed in a different order and / or if the components in the described system, architecture, apparatus, or circuit are combined in a different manner and / or replaced or supplemented by other components or their equivalents.

[0113] Therefore, other embodiments are within the scope of the claims.

[0114] The devices, units, modules, apparatuses, and other components described herein are implemented by hardware components. Examples of hardware components that can be used to perform the operations described herein include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described herein. In other examples, one or more of the hardware components performing the operations described herein are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by a processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described herein. Hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms "processor" or "computer" are used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may implement a single hardware component or two or more hardware components. Hardware components may have any one or more different processing configurations, examples of which include: a single processor, a standalone processor, a parallel processor, single-instruction single-data (SISD) multiple processing, single-instruction multiple-data (SIMD) multiple processing, multiple-instruction single-data (MISD) multiple processing, and multiple-instruction multiple-data (MIMD) multiple processing.

[0115] The methods for performing the operations described in this application are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations may be executed by a single processor or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation or two or more operations.

[0116] Instructions or software for controlling a processor or computer to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure the processor or computer, such as a machine or special-purpose computer, to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by the processor or computer. In another example, the instructions or software include high-level code that is executed by the processor or computer using an interpreter. Those skilled in the art can readily write instructions or software based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing the operations performed by the hardware components and methods described above.

[0117] Instructions or software used to control a processor or computer to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, are recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state disk, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide said instructions or software and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0118] While this disclosure includes specific examples, it will be apparent to those skilled in the art that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example will be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be construed as included in the disclosure.

Claims

1. A method for processing an image, the method comprising: Features of the current local processing region are extracted by feeding the pixel data of the current local processing region of the input image frame into a convolutional neural network. The hidden state of the recurrent neural network is updated by feeding the extracted features into the recurrent neural network to understand the context between the current local processing region and at least one previous local processing region. as well as The image processing result is generated based on the updated hidden state to produce the input image frame. Among them, convolutional neural networks are multi-head convolutional neural networks. The extraction steps include: Multiple intermediate features of the current local processing region are extracted by feeding pixel data of the current local processing region into each head of a multi-head convolutional neural network; and The encoded features of the current local processing region are generated by fusing multiple extracted intermediate features. The step of fusing the extracted intermediate features includes concatenation processing and feature projection processing. Feature projection processing includes convolution operations and is performed based on the direction of concatenation. As a result of concatenation processing and feature projection processing, the size of the encoded feature is the same as the size of each of the multiple intermediate features.

2. The method of claim 1, wherein, The update steps include: converting the extracted features into a one-dimensional vector and inputting the one-dimensional vector into a recurrent neural network.

3. The method of claim 2, wherein, A one-dimensional vector includes at least one of a channel direction vector and a width direction vector.

4. The method of any one of claims 1 to 3, wherein, The generation process includes: In response to the current local processing region corresponding to the final local processing region of the input image frame, final context data is generated based on the updated hidden state; and Image processing results are generated based on the final context data.

5. The method of any one of claims 1 to 3, wherein, The image processing results indicate whether the target object exists in the input image frame.

6. The method of claim 5, wherein, The target audience is people.

7. The method of claim 5, wherein, The target object is a human face.

8. The method of any one of claims 1 to 3, wherein, The generation process includes: In response to the current local processing region not corresponding to the final local processing region of the input image frame, intermediate context data is generated based on the updated hidden state; and Intermediate detection results are generated based on intermediate context data.

9. The method of claim 8, wherein, In response to an intermediate detection result indicating the presence of a target object in the input image frame, the image processing procedure on the input image frame is terminated without performing additional image processing on at least one subsequent local processing region.

10. The method of any one of claims 1 to 3, wherein, Each local processing region, including the current local processing region, corresponds to a pixel row group used for image signal processing of the image sensor.

11. The method of any one of claims 1 to 3, wherein, Perform extraction, update, and build in low-power mode.

12. The method of any one of claims 1 to 3, wherein, The convolutional neural network is pre-trained using object location information associated with each local processing region of the training image frame.

13. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 12.

14. An image processing apparatus, the apparatus comprising: processor; as well as The memory is configured to store instructions that can be executed by the processor. In response to the execution of the instruction by the processor, the processor is configured to: Features of the current local processing region are extracted by feeding the pixel data of the current local processing region of the input image frame into a convolutional neural network. The hidden state of the recurrent neural network is updated by feeding the extracted features into the recurrent neural network to understand the context between the current local processing region and at least one previous local processing region. The image processing result is generated based on the updated hidden state to produce the input image frame. The convolutional neural network is a multi-head convolutional neural network, and the processor is further configured to: extract multiple intermediate features of the current local processing region by inputting pixel data of the current local processing region into each head of the multi-head convolutional neural network, and generate encoded features of the current local processing region by fusing the extracted multiple intermediate features. The process of fusing multiple extracted intermediate features includes concatenation and feature projection. Feature projection includes convolution operations and is performed based on the direction of concatenation. As a result of concatenation and feature projection, the size of the encoded feature is the same as the size of each of the multiple intermediate features.

15. The apparatus of claim 14, wherein, The processor is also configured to: generate final context data based on the updated hidden state in response to the current local processing region corresponding to the final local processing region of the input image frame, and generate an image processing result based on the final context data.

16. The apparatus of claim 14, wherein, The processor is also configured to: generate intermediate context data based on the updated hidden state in response to the current local processing region not corresponding to the final local processing region of the input image frame, and generate intermediate detection results based on the intermediate context data.

17. The apparatus of claim 14, wherein, Each local processing region, including the current local processing region, corresponds to a pixel row group used for image signal processing of the image sensor.

18. An electronic device comprising: The camera is configured to generate input image frames; The memory is configured to store instructions; as well as The processor is configured to execute instructions stored in memory to: Features of the current local processing region are extracted by feeding the pixel data of the current local processing region of the input image frame into a convolutional neural network. The hidden state of the recurrent neural network is updated by feeding the extracted features into the recurrent neural network to understand the context between the current local processing region and at least one previous local processing region. The image processing result is generated based on the updated hidden state to produce the input image frame. The convolutional neural network is a multi-head convolutional neural network, and the processor is further configured to: extract multiple intermediate features of the current local processing region by inputting pixel data of the current local processing region into each head of the multi-head convolutional neural network, and generate encoded features of the current local processing region by fusing the extracted multiple intermediate features. The process of fusing multiple extracted intermediate features includes concatenation and feature projection. Feature projection includes convolution operations and is performed based on the direction of concatenation. As a result of concatenation and feature projection, the size of the encoded feature is the same as the size of each of the multiple intermediate features. 19.The electronic device of claim 18, wherein, Extraction, updating, and generation are performed in low-power mode, and The electronic device is configured to be woken up in response to the detection of a target object in an input image frame in order to perform operations associated with image processing in normal mode. 20.The electronic device of claim 19, wherein, Low power modes include always-on mode. 21.The electronic device of claim 19, wherein, Image processing includes object detection, and Operations associated with image processing include any one or any combination of object tracking, object recognition, and unlocking.

22. An apparatus comprising: processor; as well as The memory is configured to store instructions that can be executed by the processor. In response to the execution of the instruction by the processor, the processor is configured to: Use a convolutional neural network to extract features from the current local processing region of the input image frame. Determine whether the current local processing region of the input image frame is the final local processing region of the input image frame. If the current local processing region is not the final local processing region, the hidden state of the recurrent neural network is repeatedly updated by inputting the features extracted from the current local processing region into the recurrent neural network. Intermediate context data is generated based on the updated hidden state, and intermediate detection results are generated based on the intermediate context data, until the current local processing region is the final local processing region. When the current local processing region is the final local processing region, the hidden state of the recurrent neural network is updated by inputting the extracted features of the current local processing region into the recurrent neural network. The final context data is then generated based on the updated hidden state, and the image processing result of the input image frame is generated based on the final context data. The convolutional neural network is a multi-head convolutional neural network, and the processor is further configured to: extract multiple intermediate features of the current local processing region by inputting pixel data of the current local processing region into each head of the multi-head convolutional neural network, and generate encoded features of the current local processing region by fusing the extracted multiple intermediate features. The process of fusing multiple extracted intermediate features includes concatenation and feature projection. Feature projection includes convolution operations and is performed based on the direction of concatenation. As a result of concatenation and feature projection, the size of the encoded feature is the same as the size of each of the multiple intermediate features.

23. The apparatus of claim 22, wherein, The processor is configured to: after each iteration where the current local processing region is not the final local processing region, determine whether the intermediate detection result indicates the presence of a target object in the input image frame, and terminate any further processing of the input image frame if the intermediate detection result indicates the presence of a target object in the input image frame.

24. An apparatus comprising: processor; as well as The memory is configured to store instructions that can be executed by the processor. In response to the execution of the instruction by the processor, the processor is configured to: Use a convolutional neural network to extract features from the current local processing region of the input data frame. The hidden state of a recurrent neural network is updated by inputting the extracted features into the recurrent neural network. Context data is generated based on the updated hidden state. Detection results are generated based on contextual data. Determine whether the detection result indicates the presence of target data in the input data frame. If the detection result indicates the presence of target data in the input data frame, terminate any further processing of the input data frame, and If the detection result does not indicate the presence of target data in the input data frame, processing of the input data frame continues in the local processing region following the current local processing region. The convolutional neural network is a multi-head convolutional neural network, and the processor is further configured to: extract multiple intermediate features of the current local processing region by inputting pixel data of the current local processing region into each head of the multi-head convolutional neural network, and generate encoded features of the current local processing region by fusing the extracted multiple intermediate features. The process of fusing multiple extracted intermediate features includes concatenation and feature projection. Feature projection includes convolution operations and is performed based on the direction of concatenation. As a result of concatenation and feature projection, the size of the encoded feature is the same as the size of each of the multiple intermediate features.

25. The apparatus of claim 24, wherein, The processor is configured to generate an image processing result for the input data frame based on context data when the detection result does not indicate the presence of target data in the input data frame and the current local processing region is the final local processing region of the input data frame.

Images