Neural network system and operating method of the same

Abstract

A neural network system according to the technical concept of the present disclosure includes a processor that detects a plurality of object candidates included in a first image by analyzing a first image, generates metadata of the plurality of object candidates based on the first image, and sets the data processing order of the plurality of object candidates based on the metadata, and a resource that performs data processing of the plurality of object candidates. The processor sequentially provides information used for data processing of the plurality of object candidates to the resource according to the set order, and the resource can sequentially perform data processing of the plurality of object candidates according to the order in which it receives the information used for data processing of the plurality of object candidates.

Description

Technology Field

[0001] The technical concept of the present disclosure relates to a neural network system and a method of operation thereof, and specifically to a neural network system and a method of operation thereof that performs data processing of object candidates included in an image on an object-by-object basis. Background Technology

[0002] An artificial neural network (ANN) refers to a computational architecture that models the biological brain. Deep learning or machine learning can be implemented based on artificial neural networks. Recently, as the computations that need to be processed using artificial neural networks have increased dramatically, there is a need to efficiently perform computational processing using artificial neural networks. The problem to be solved

[0003] The technical concept of the present disclosure provides a neural network system and a method of operation thereof that can prevent bottlenecks caused by continuous object recognition operations and can prioritize data processing for object candidates of high importance by establishing an order based on metadata of a plurality of object candidates included in an image and performing object recognition for each of the plurality of object candidates and tasks corresponding to the object recognition results on an object-by-object basis according to the established order. means of solving the problem

[0004] To achieve the above objectives, a neural network system according to one aspect of the technical concept of the present disclosure comprises a processor that detects a plurality of object candidates included in a first image by analyzing a first image, generates metadata of the plurality of object candidates based on the first image, and sets a data processing order of the plurality of object candidates based on the metadata, and a resource that performs data processing of the plurality of object candidates. The processor sequentially provides information used for data processing of the plurality of object candidates to the resource according to the set order, and the resource can sequentially perform data processing of the plurality of object candidates according to the order in which it receives the information used for data processing of the plurality of object candidates.

[0005] A method of operating a neural network system according to one aspect of the technical concept of the present disclosure may include the steps of detecting a plurality of object candidates included in a first image, performing a first object recognition for a first object candidate among the plurality of object candidates, performing a first task corresponding to the result of the first object recognition, and, when the first task is completed, performing a second object recognition for a second object candidate among the plurality of object candidates and performing a second task corresponding to the result of the second object recognition.

[0006] An electronic device according to one aspect of the technical concept of the present disclosure comprises: a resource that performs a computation; a sensor that acquires data surrounding the electronic device and converts the acquired data into an image form to output a first image; an object candidate detector that detects a plurality of object candidates within the first image; a memory that stores program information including a task manager that generates metadata of a plurality of object candidates based on the first image and selects an object candidate among the plurality of object candidates to perform object recognition based on the metadata; and a processor that selects a first object candidate based on the program information and provides information about the first object candidate to the resource. The resource can perform a first object recognition of the first object candidate based on the received information about the first object candidate and perform a first task corresponding to the result of the first object recognition. Effects of the invention

[0007] According to the neural network system and the method of operation thereof according to an exemplary embodiment of the present disclosure, the order of a plurality of object candidates included in an image is set according to priority, and object recognition for each of the plurality of object candidates and tasks corresponding to the object recognition results can be performed on an object-by-object basis according to the set order, thereby preventing bottlenecks caused by continuous object recognition operations and allowing data processing for object candidates of high importance to be processed preferentially. Brief explanation of the drawing

[0008] FIG. 1 is a block diagram showing a neural network system according to one embodiment of the present disclosure. FIG. 2 is a drawing showing in detail a neural network system according to one embodiment of the present disclosure. FIG. 3 is a conceptual diagram illustrating an object recognition method according to one embodiment of the present disclosure. FIG. 4 is a diagram showing metadata of an object candidate according to one embodiment of the present disclosure. FIG. 5 is a drawing showing a plurality of object candidates included in an image according to one embodiment of the present disclosure. FIG. 6 is a conceptual diagram illustrating a method for generating a sequence table according to one embodiment of the present disclosure. FIG. 7 is a conceptual diagram illustrating a method for generating a sequence table according to one embodiment of the present disclosure. FIG. 8 is a flowchart illustrating an example of the operation of a model processor and resources according to one embodiment of the present disclosure. FIG. 9 is a flowchart illustrating an example of the operation of a model processor and resources according to one embodiment of the present disclosure. FIG. 10 is a conceptual diagram illustrating a method for generating a sequence table according to one embodiment of the present disclosure. FIG. 11 is a flowchart illustrating an example of the operation of a model processor and resources according to one embodiment of the present disclosure. FIG. 12 is a flowchart illustrating an example of the operation of a model processor and resources according to one embodiment of the present disclosure. FIG. 13 is a block diagram illustrating a neural network system according to one embodiment of the present disclosure. FIG. 14 is a block diagram showing a neural network system according to one embodiment of the present disclosure. FIG. 15 is a flowchart illustrating a method of operation of a neural network system according to one embodiment of the present disclosure. FIG. 16 is a block diagram showing an electronic device according to one embodiment of the present disclosure. FIG. 17 is a block diagram showing an autonomous driving device according to one embodiment of the present disclosure. Specific details for implementing the invention

[0009] FIG. 1 is a block diagram illustrating a neural network system according to one embodiment of the present disclosure. Referring to FIG. 1, the neural network system (10) may include a model processor (100) and a resource (200). Additionally, the model processor (100) may include an object candidate detector (110) and a task manager (120).

[0010] The neural network system (10) can perform neural network-based neural tasks based on various neural networks. The neural network may include, but is not limited to, various types of neural network models such as CNN (Convolutional Neural Network), R-CNN (Region with Convolutional Neural Network), fast R-CNN, RPN (Region Proposal Network), RNN (Recurrent Neural Network), S-DNN (Stacking-based Deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restricted Boltzmann Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, GAN (Generative Adversarial Network), IV3 (Inception V3), and Classification Network. In addition, a neural network performing a single task may include sub-neural networks, and the sub-neural networks may be implemented as homogeneous or heterogeneous neural network models.

[0011] The neural network system (10) can receive an image (IMG) from the outside and perform data processing on the received image (IMG). In one embodiment, the neural network system (10) can perform data processing on the received image (IMG) using a neural network model. In one embodiment, the neural network system (10) can use a neural network model such as RPN, R-CNN, or fast R-CNN to detect multiple object candidates included in the received image (IMG), perform object recognition on the multiple object candidates, and perform a task corresponding to the recognition result.

[0012] The model processor (100) may include an object candidate detector (110) and a task manager (120). Each component included in the model processor (100) may be implemented in software and may be implemented on a neural network platform such as EDEN (Evolutinary Deep Network). In another example, the components included in the model processor (100) may be implemented in hardware such as electronic circuits. In one example, the model processor (100) may be implemented as a neural network framework.

[0013] The object candidate detector (110) receives an image (IMG) from the outside and can detect multiple object candidates included in the image (IMG) by analyzing the received image (IMG). An object candidate may refer to an area within the image (IMG) where there is a high probability that an object of interest exists. An object candidate may be referred to as a Region of Interest (RoI). In one embodiment, the object candidate detector (110) can detect multiple object candidates included in the image (IMG) using a neural network model such as an RPN. The object candidate detector (110) can provide the detected multiple object candidates to the task manager (120).

[0014] A task manager (120) according to the technical concept of the present disclosure may receive a plurality of object candidates and set a data processing order for the plurality of object candidates. In one embodiment, the task manager (120) may set a data processing order for the plurality of object candidates according to the priority of the plurality of object candidates. The task manager (120) may provide information (Info_OC) used for data processing of the plurality of object candidates to a resource (200) according to the set order. For example, the task manager (120) may provide information used for data processing of a first object candidate, which is in an earlier order among the plurality of object candidates of an image (IMG), to the resource (200), and upon receiving a response signal from the resource (200) indicating the completion of the task for the first object candidate, information used for data processing of a second object candidate, which is in the next order, to the resource (200).

[0015] The resource (200) may include a computational resource capable of performing computations based on data received from the task manager (120). For example, the resource (200) may include various computational processing devices such as a CPU (Central Processing Unit), Graphic Processing Unit (GPU), AP (Application Processor), DSP (Digital Signal Processor), FPGA (Field-Programmable Gate Array), NPU (Neural Network Processing Unit), ECU (Electronic Control Unit), and ISP (Image Signal Processor). Meanwhile, the resource (200) may be composed of a plurality of computational resources, and may be composed of a plurality of homogeneous or heterogeneous computational resources according to the embodiment. For example, the resource (200) may include a first resource (not shown) which is an NPU and a second resource (not shown), or a first resource (not shown) which is an NPU and a second resource (not shown) which is a GPU.

[0016] A resource (200) according to the technical concept of the present disclosure can perform data processing for a plurality of object candidates in the order in which it receives information used for data processing of a plurality of object candidates from a task manager (120). In one embodiment, the resource (200) can perform object recognition for a plurality of object candidates and perform a task corresponding to the object recognition result in the order in which it receives information used for data processing of a plurality of object candidates. Meanwhile, the present disclosure is not limited thereto, and the data processing may perform separate operations in addition to object recognition, and may be composed of a single operation or three or more operations.

[0017] A resource (200) according to the technical concept of the present disclosure can perform data processing for a plurality of object candidates on an object candidate basis. Specifically, if there are a plurality of object candidates and there are a plurality of operation operations required for data processing for one object candidate, the resource (200) can perform operation operations for the next object candidate after performing all operation operations for one object candidate. For example, if information used for data processing of a first object candidate is received first from a task manager (120), and then information used for data processing of a second object candidate is received, a first object recognition for the first object candidate is performed, and a first task corresponding to the result of the first object recognition is performed, and then a second object recognition for the second object candidate is performed, and a second task corresponding to the second object recognition is performed.

[0018] A neural network system (10) according to the technical concept of the present disclosure sets an order according to the priority of a plurality of object candidates included in an image, and performs object recognition for the plurality of object candidates and tasks corresponding to the object recognition results on an object-by-object basis according to the set order, thereby preventing bottlenecks caused by continuous object recognition operations and prioritizing data processing for object candidates of high importance.

[0019] FIG. 2 is a drawing showing a neural network system according to one embodiment of the present disclosure in detail. In detail, FIG. 2 is a drawing showing the neural network system (10) of FIG. 1 in detail.

[0020] Referring to FIG. 2, the neural network system (10) may include a model processor (100) and a resource (200), the object candidate detector (110) may include a feature map generator (111) and a region of interest detector (112), and the task manager (120) may include a metadata generator (121) and a task scheduler (122).

[0021] The feature map generator (111) can generate a feature map (FM) using an image (IMG) received from the outside. In one embodiment, the feature map generator (111) can generate a feature map (FM) representing the features of the image (IMG) by performing a convolution operation using the image (IMG) received from the outside. Meanwhile, the present disclosure is not limited thereto, and the feature map generator (111) can generate a feature map (FM) through a separate operation. The feature map generator (111) can provide the generated feature map (FM) to the region of interest detector (112).

[0022] The region of interest detector (112) can detect a region of interest (i.e., object candidate) which is an area with a high probability of containing an object of interest by analyzing a feature map (FM) received from a feature map generator (111). In one embodiment, the region of interest detector (112) can detect multiple regions of interest (i.e., multiple object candidates (OC1~OCn)) using an RPN model. Meanwhile, the present disclosure is not limited thereto, and the region of interest detector (112) can detect multiple regions of interest using a separate model. The region of interest detector (112) can provide the detected multiple object candidates (OC1~OCn) to the task manager (120). Additionally, according to an embodiment, the region of interest detector (112) can provide an input image (IMG) to the task manager (120) along with the multiple object candidates (OC1~OCn).

[0023] The task manager (120) receives multiple object candidates (OC1~OCn) from the object candidate detector (110), generates metadata (MD1~MDn) of the multiple object candidates based on the input image (IMG), and can set the data processing order for the multiple object candidates based on the generated metadata (MD1~MDn).

[0024] Specifically, the metadata generator (121) receives a plurality of object candidates (OC1~OCn) from the object candidate detector (110) and can generate metadata (MD1~MDn) corresponding to each of the plurality of object candidates (OC1~OCn) using an image (IMG) and the received plurality of object candidates (OC1~OCn). The metadata may include information regarding the size of each of the plurality of object candidates, their position within the image (IMG), top-left coordinates, bottom-right coordinates, distance from a preset coordinate or area, depth, etc. Meanwhile, the present disclosure is not limited thereto, and the types of metadata may include additional information in addition to the information described above. The metadata generator (121) can provide the generated metadata (MD1~MDn) to the task scheduler (122).

[0025] The task scheduler (122) receives metadata (MD1~MDn) of multiple object candidates (OC1~OCn) from the metadata generator (121) and can set the data processing order for the multiple object candidates based on the metadata (MD1~MDn). In one embodiment, the task scheduler (122) can calculate a score indicating the importance of the multiple object candidates using the metadata (MD1~MDn) and set the data processing order for the multiple object candidates using the calculated score.

[0026] And the task scheduler (122) can provide information (Info_OC1~Info_OCn) used for data processing of multiple object candidates to the resource (200) according to the set order. In one embodiment, the information (Info_OC1~Info_OCn) used for data processing of multiple object candidates may include multiple object candidates (OC1~OCn) and metadata (MD1~MDn). That is, the task scheduler (122) can provide regions of interest detected by the region of interest detector (112) and metadata of the regions of interest to the resource (200).

[0027] The resource (200) can perform object recognition for each of the multiple object candidates (OC1~OCn) in the order in which it receives information (Info_OC1~Info_OCn) used for data processing of multiple object candidates, and can perform a task corresponding to the object recognition result. The task corresponding to the object recognition result may consist of one or more tasks and may be configured in various ways depending on the type or use of the device equipped with the neural network system, the user, the designer, or the manufacturer's settings. For example, if the neural network system (10) is equipped in an autonomous vehicle and the recognized object is a 'car', the resource (200) can perform tasks corresponding to the 'car', such as calculating the distance between the recognized 'car' and the autonomous vehicle, and generating a control command to control the speed of the autonomous vehicle using the calculated distance.

[0028] FIG. 3 is a conceptual diagram illustrating an object recognition method according to one embodiment of the present disclosure. More specifically, FIG. 3 is a conceptual diagram illustrating an object recognition method of a neural network system according to one embodiment of the present disclosure.

[0029] Referring to FIGS. 1 to 3, a model processor (100) of a neural network system according to one embodiment of the present disclosure can detect a plurality of object candidates by analyzing an input image (IMG). Since the method for detecting object candidates may be the same as the method described in FIGS. 1 and FIG. 2, a redundant description is omitted. The model processor (100) can generate metadata for the detected plurality of object candidates and set a data processing order for the object candidates based on the generated metadata.

[0030] And the model processor (100) can sequentially provide information (Info_OC1~Info_OCn) used for data processing of multiple object candidates to the resource (200) according to the set order. For example, referring to FIG. 3, the model processor (100) can provide information (Info_OC1) about the first object candidate, which is in the order of the multiple object candidates of the image (IMG), to the resource (200), and when it receives a response signal from the resource (200) indicating the completion of the task for the first object candidate, it can provide information (Info_OC2) about the second object candidate, which is in the order of the next, to the resource (200).

[0031] The resource (200) receives information (Info_OC1 to Info_OCn) used for data processing of multiple object candidates from the model processor (100), and can perform data processing for multiple object candidates in the order in which the information (Info_OC1 to Info_OCn) used for data processing of multiple object candidates is received. For example, referring to FIG. 3, when the resource (200) receives information (Info_OC1) about a first object candidate, it can perform a first object recognition (OR1) for the first object candidate (OC1) and perform a first task (T1) corresponding to the result of the first object recognition (OR1). And when the resource (200) receives information (Info_OC2) about a second object candidate, it can perform a second object recognition (OR2) for the second object candidate (OC2) and perform a second task (T2) corresponding to the result of the second object recognition (OR2). In this way, the resource (200) can perform object recognition (~ORn) in the same way for the remaining object candidates (~OCn) and then perform a task (~Tn) corresponding to the object recognition result.

[0032] Meanwhile, the model processor (100) of a neural network system according to a conventional method can detect multiple object candidates by analyzing an input image (IMG) and provide information used for data processing of the multiple object candidates to a resource (200). Then, the resource (200) can perform all object recognition for the multiple object candidates based on the information used for data processing of the multiple object candidates received from the model processor (100), and then perform tasks corresponding to the object recognition results. For example, referring to FIG. 3, the resource (200) can perform tasks (T1~Tn) corresponding to the object recognition results only after performing all object recognitions (OR1~ORn) for the multiple object candidates (OC1~OCn).

[0033] In other words, a neural network system based on the conventional method can perform object recognition on a frame-by-frame basis. Therefore, the time required for the neural network system to start a task corresponding to the object recognition result can increase in proportion to the number of objects (or object candidates) included in the image (IMG).

[0034] Recently, technologies utilizing object recognition in images have been increasing, and not only the number of images requiring recognition but also the types and number of objects of interest to be recognized within those images are growing. Consequently, when using conventional neural network systems, the time required to perform object recognition on images becomes prolonged (i.e., a bottleneck occurs due to object recognition), leading to a problem where the initiation of tasks corresponding to the recognition results is delayed. This issue has been particularly highlighted in technologies requiring fine control based on object recognition, such as autonomous driving technology, which controls a vehicle based on the results of recognizing surrounding objects.

[0035] Meanwhile, a neural network system according to one embodiment of the present disclosure can perform object recognition on an object-by-object basis. Accordingly, the time required for the neural network system to start a task corresponding to the object recognition result (t1, referring to FIG. 3) does not include the time required to perform object recognition (OR2~ORn) for the remaining object candidates. Therefore, the time required (t1) to start a task corresponding to the object recognition result may be less than the time required in a neural system according to a conventional method.

[0036] Accordingly, when using a neural network system according to one embodiment of the present disclosure, bottlenecks caused by frame-unit object recognition can be prevented, the timing for starting tasks corresponding to object recognition can be advanced, and data processing for high-importance object candidates can be performed preferentially. Accordingly, the neural network system according to one embodiment of the present disclosure can perform fine-grained control using object recognition.

[0037] FIG. 4 is a drawing showing metadata of an object candidate according to one embodiment of the present disclosure. Specifically, FIG. 4 is a drawing showing metadata of an object candidate generated by the task manager (120) of FIG. 2.

[0038] Referring to FIGS. 2 and FIGS. 4, a task manager (120) receives a plurality of object candidates (OC1 to OCn) from an object candidate detector (Fig. 2, 110) and can generate a metadata table (MT) for the plurality of object candidates (OC1 to OCn). The metadata table (MT) may be composed of metadata (MD1 to MDn) for the plurality of object candidates (OC1 to OCn). The metadata (MD1 to MDn) may include the names (or numbers, indices) of the plurality of object candidates (OC1 to OCn) and various information written in vector format. For example, the metadata (MD1 to MDn) may include information regarding the size, position within an image (IMG), top-left coordinates, bottom-right coordinates, distance from a preset coordinate or area, depth, etc., for each of the plurality of object candidates (OC1 to OCn). Meanwhile, the present disclosure is not limited thereto, and the types of metadata may include additional information in addition to the information described above. For example, although not illustrated in FIG. 4, metadata (MD1 to MDn) may additionally include metadata of an object corresponding to an object candidate in the current frame image among the objects recognized in the image of the previous frame.

[0039] The task manager (120) can create a metadata table (MT) and store the created metadata table (MT) in a memory area (not shown) included within the model processor (100) or in an external memory area (not shown) that the task manager (120) can access.

[0040] Hereinafter, a method for setting the data processing order for a plurality of object candidates using a metadata table (MT) generated by a model processor (100) is described with reference to FIGS. 5 to 7.

[0041] FIG. 5 is a drawing showing a plurality of object candidates included in an image according to one embodiment of the present disclosure. Referring to FIG. 5, the image (IMG) is a view of an autonomous vehicle captured by a camera attached to the autonomous vehicle.

[0042] Referring to FIGS. 1, 2 and 5, an object candidate detector (110) can detect multiple object candidates included in the image (IMG) by receiving an image (IMG) and analyzing the received image (IMG). For example, referring to FIG. 5, the object candidate detector (110) can detect six object candidates (OC1 to OC6) included in the image (IMG) using a neural network model such as an RPN. The object candidate detector (110) can provide the detected object candidates (OC1 to OC6) to a task manager (120).

[0043] FIG. 6 is a conceptual diagram illustrating a method for generating a sequence table according to one embodiment of the present disclosure. Referring to FIGS. 1, FIGS. 2, FIGS. 4 through 6, a metadata generator (121) of a task manager (120) receives a plurality of object candidates from an object candidate detector (110) and can generate a metadata table (MT) for the received plurality of object candidates. The metadata generator (121) can provide the generated metadata table (MT) to a task scheduler (122).

[0044] A task scheduler (122) according to the technical concept of the present disclosure can generate a score table (ST) of multiple object candidates using a metadata table (MT). The score table (ST) may include the name (or number, index) of each of the multiple object candidates and a score indicating the importance of each of the multiple object candidates. In one embodiment, the task scheduler (122) can calculate the scores of multiple object candidates by inputting information constituting metadata into a function implemented using multiple inputs and multiple weights. For example, the task scheduler (122) can calculate the score of each of the multiple object candidates using a function as follows.

[0045] Score = k1 * 1 / |distance| + k2 * size

[0046] (k1 is the weight for distance, k2 is the weight for size)

[0047] Meanwhile, the function for calculating the score using metadata is not limited to the example described above and can be implemented in various forms. For example, referring to FIG. 5, the object candidate with the highest score may be a large object candidate (OC6) located close to a preset coordinate (e.g., a coordinate corresponding to the viewpoint of an autonomous vehicle). Additionally, the object candidate with the lowest score may be a small object candidate (OC3) located farthest from the preset coordinate.

[0048] A task scheduler (122) according to the technical concept of the present disclosure can generate a sequence table (OT) using a generated score table (ST). In one embodiment, the task scheduler (122) can generate a sequence table (OT) indicating the data processing order for a plurality of object candidates by sorting the scores of a plurality of object candidates included in the score table (ST) in descending order. For example, referring to FIG. 6, the task scheduler (122) can generate a sequence table (OT) indicating the order from the object candidate (OC6) with the highest score to the object candidate (OC3) with the lowest score using the score table (ST).

[0049] And the task scheduler (122) can sequentially provide information about multiple object candidates to the resource (200) using the generated sequence table (OT). For example, referring to FIG. 6, the task scheduler (122) can use the sequence table (OT) to identify the object candidate (OC6) that is in the first order and provide information about the identified object candidate (OC6) to the resource (200). Then the task scheduler (122) can use the sequence table (OT) to identify the object candidate (OC1) that is in the next order and provide information about the identified object candidate (OC1) to the resource (200). Then the task scheduler (122) can repeat the above-described operation for the remaining object candidates.

[0050] Meanwhile, according to a modifiable embodiment, the task scheduler (122) may provide a sequence table (OT) to the resource (200) along with information about a plurality of object candidates. The resource (200) may use the received sequence table (OT) to perform object recognition for the object candidate in the earliest order and perform a task corresponding to the object recognition result. Then, the resource (200) may use the sequence table (OT) to perform object recognition for the object candidate in the next order and perform a task corresponding to the object recognition result.

[0051] For example, referring to FIG. 6, the resource (200) can perform object recognition for the object candidate (OC6) that is in the earliest order using the order table (OT) and perform a task corresponding to the object recognition result. Then, the resource (200) can perform object recognition for the object candidate (OC1) that is in the next order using the order table (OT) and perform a task corresponding to the object recognition result. Then, the resource (200) can repeat the above-described operations for the remaining object candidates.

[0052] Meanwhile, in illustrating and describing FIG. 6, the task scheduler (122) is illustrated and described as generating an order table (OT), but the task scheduler (122) may be implemented to generate a linked list, a search tree, or a FIFO (First In First Out) representing the data processing order of multiple object candidates.

[0053] FIG. 7 is a conceptual diagram illustrating a method for generating a sequence table according to one embodiment of the present disclosure. In detail, FIG. 7 is a conceptual diagram illustrating a modified embodiment of FIG. 6.

[0054] Referring to FIGS. 1, FIGS. 2, FIGS. 4 through 7, when the task scheduler (122) generates a sequence table (OT) using a score table (ST), it may set the order using only the remaining object candidates, excluding object candidates having a score less than a first threshold. The first threshold may mean the lowest value of a score that is determined to require object recognition.

[0055] For example, in the embodiment of FIG. 7, the first threshold value may be 50. The task scheduler (122) can generate an order table (OT) by identifying an object candidate (OC3) having a score less than the first threshold value (50) in the score table (ST) and sorting the scores of the remaining object candidates (OC1, OC2, OC4 to OC6), excluding the identified object candidate (OC3), in descending order. Then, the task scheduler (122) can provide information about a plurality of object candidates to a resource (200) using the generated order table (OT).

[0056] A neural network system according to one embodiment of the present disclosure can reduce the amount of computation and increase the computation speed by excluding object candidates for which object recognition is deemed unnecessary and performing object recognition only on the remaining object candidates.

[0057] FIG. 8 is a flowchart illustrating an example of the operation of a model processor and a resource according to an embodiment of the present disclosure. More specifically, it is a flowchart illustrating an example of the operation of a model processor (100) and a resource (200) of FIG. 1 or FIG. 2.

[0058] Referring to FIG. 8, the object candidate detector (110) can detect a plurality of object candidates (OC1 to OCn) by analyzing an input image (S110). The object candidate detector (110) can provide the detected plurality of object candidates (OC1 to OCn) to the task manager (120) (S115). According to an embodiment, the object candidate detector (110) can provide the input image to the task manager (120) along with the plurality of object candidates (OC1 to OCn).

[0059] The task manager (120) can generate metadata for the received multiple object candidates (OC1 to OCn) (S120). The task manager (120) can set the data processing order for the multiple object candidates (OC1 to OCn) using the generated metadata (S125). In one embodiment, the task manager (120) can calculate a score indicating the importance of each of the multiple object candidates (OC1 to OCn) using the metadata, as described above in FIG. 6, and generate an order table (OT) using the calculated score.

[0060] In one embodiment, the task manager (120) may generate a sequence table (OT) using only the remaining object candidates, excluding the object candidates having a score less than a first threshold among the calculated scores as described above in FIG. 7. Then, the task manager (120) may provide information (Info_OC1) used for data processing of the first object candidate to the resource (200) according to the set order (S130). In one embodiment, the task manager (120) may check the set order in the generated sequence table (OT) and provide information (Info_OC1) used for data processing of the first object candidate to the resource (200) according to the checked order. According to the embodiment, the task manager (120) may provide an input image to the resource (200) along with information (Info_OC1) used for data processing of the first object candidate.

[0061] The resource (200) can perform first object recognition using information (Info_OC1) used for processing data of the received first object candidate (S135). The resource (200) can perform a first task corresponding to the first object recognition (S140). The resource (200) can send a response signal indicating that the first task has been completed to the task manager (120) (S145).

[0062] When the task manager (120) receives a response signal from the resource (200), it can check the second object candidate in the order following the first object candidate according to the set order and provide the resource (200) with information (Info_OC2) used for data processing of the second object candidate (S150).

[0063] The resource (200) can perform second object recognition using information (Info_OC2) used for processing data of the received second object candidate (S155). The resource (200) can perform a second task corresponding to the second object recognition (S160). The resource (200) can send a response signal indicating that the second task has been completed to the task manager (120) (S165).

[0064] The task manager (120) and the resource (200) can repeatedly perform the operation of providing information used for data processing of the aforementioned object candidate, the operation of performing object recognition, the operation of performing a task corresponding to the object recognition result, and the operation of transmitting a response signal indicating task completion for the remaining object candidates as well (S170 to S185).

[0065] FIG. 9 is a flowchart illustrating an example of the operation of a model processor and a resource according to an embodiment of the present disclosure. In detail, FIG. 9 is a drawing of a modified embodiment of FIG. 8. Since steps S210 to S220 of FIG. 9 may be substantially the same as steps S110 to S120 of FIG. 8, a redundant description is omitted.

[0066] The task manager (120) can generate a sequence table (OT) indicating the data processing order for multiple object candidates (OC1~OCn) using metadata (S225). In one embodiment, the task manager (120) can generate a score indicating the importance of each of the multiple object candidates (OC1~OCn) using metadata as described above in FIG. 6, and generate a sequence table (OT) using the calculated score. In one embodiment, the task manager (120) can generate a sequence table (OT) using only the remaining object candidates, excluding the object candidate having a score less than a first threshold among the calculated scores, as described above in FIG. 7. The task manager (120) can provide the generated sequence table (OT) and information (Info_OC1~Info_OCn) used for data processing of the multiple object candidates to the resource (200) (S230).

[0067] The resource (200) can check the order using the received order table (OT) and perform object recognition and tasks corresponding to the object recognition results for each object candidate according to the checked order. That is, the resource (200) can perform object recognition for the first object candidate, which is the earliest in order (S235), and perform tasks corresponding to the object recognition results for the first object candidate (S240). Then, the resource (200) can perform object recognition for the second object candidate, which is the next in order (S245), and perform tasks corresponding to the object recognition results for the second object candidate (S250). The resource (200) can repeatedly perform the aforementioned object recognition operation and tasks corresponding to the object recognition results for the remaining object candidates as well (S255 and S260). Then, the resource (200) can transmit a response signal to the task manager (120) indicating that the tasks corresponding to multiple object candidates (OC1~OCn) have been completed (S265).

[0068] Meanwhile, in illustrating and describing FIG. 9, when the task manager (120) transmits information used for processing data of object candidates to the resource, it is illustrated and described as transmitting information for only one object candidate to the resource. However, the task manager (120) can also be implemented in a manner where it transmits information used for processing data of two or more preset numbers of object candidates to the resource (200), and the resource (200) performs object recognition operations for object candidates using the received information and performs tasks corresponding to the object recognition results.

[0069] FIG. 10 is a conceptual diagram illustrating a method for generating a sequence table according to one embodiment of the present disclosure. In detail, FIG. 10 is a conceptual diagram illustrating a modified embodiment of FIG. 6.

[0070] Referring to FIG. 10, a task scheduler (122) according to the technical concept of the present disclosure can generate a score table (ST) of a plurality of object candidates using a metadata table (MT) and identify object candidates having a score exceeding a second threshold using the score table (ST). The second threshold may refer to a score that is judged to have a very high necessity for performing object recognition. Object candidates having a score exceeding the second threshold are highly likely to be objects of interest and can serve as an important criterion for controlling a device to which a neural network system is applied.

[0071] If it is determined that there are no object candidates with scores exceeding the second threshold, the task scheduler (122) can generate an order table (OT) by sorting the scores in descending order as described above in FIG. 6.

[0072] On the other hand, if it is confirmed that there is an object candidate having a score exceeding the second threshold, the task scheduler (122) can immediately provide information used for data processing of the confirmed object candidate to the resource (200). Then, the resource (200) can use the information used for data processing of the received object candidate to perform object recognition for the confirmed object candidate and a task corresponding to the object recognition result.

[0073] For example, in the embodiment of FIG. 10, the second threshold may be 90. The task scheduler (122) can identify an object candidate (OC6) having a score exceeding the second threshold (90) in the score table (ST) and immediately provide information (Info_OC6) used for data processing of the identified object candidate (OC6) to the resource (200). Then, the resource (200) can use the received information (Info_OC6) to perform object recognition for the object candidate (OC6) and a task corresponding to the object recognition result.

[0074] The task scheduler (122) can generate a sequence table (OT) using scores for the remaining object candidates after providing information about object candidates having scores exceeding a second threshold to the resource (200). For example, referring to FIG. 10, the task scheduler (122) can generate a sequence table (OT) using scores for the remaining object candidates (OC1 to OC5).

[0075] And the task scheduler (122) can sequentially provide information (Info_OC1 to Info_OC6) used for data processing of the remaining object candidates (OC1 to OC5) to the resource (200) using the generated sequence table (OT). And the resource (200) can perform object recognition for the remaining object candidates (OC1 to OC5) and tasks corresponding to the object recognition results according to the order in which the information (Info_OC1 to Info_OC6) is received.

[0076] Meanwhile, according to a modifiable embodiment, the task scheduler (122) may provide a sequence table (OT) to the resource (200) along with information (Info_OC1 to Info_OC6) used for data processing of the remaining object candidates (OC1 to OC5). The resource (200) may use the received sequence table (OT) to perform object recognition for the remaining object candidates (OC1 to OC5) and a task corresponding to the object recognition result.

[0077] A neural network system according to one embodiment of the present disclosure can perform fine-grained control of a device to which the neural network system is applied by immediately performing object recognition for the identified object candidate and a task corresponding to the object recognition result when an object candidate is identified for which it is determined that there is a very high need to perform object recognition.

[0078] FIG. 11 is a flowchart illustrating an example of the operation of a model processor and a resource according to one embodiment of the present disclosure. In detail, FIG. 11 is a drawing of a modified embodiment of FIG. 8. Since steps S310 to S320 of FIG. 11 may be substantially the same as steps S110 to S120 of FIG. 8, a redundant description is omitted.

[0079] The task manager (120) can calculate scores indicating the importance of each of the multiple object candidates (OC1~OCn) using metadata (S325). Then, as described above in FIG. 10, the task manager (120) can identify an object candidate having a score exceeding a second threshold among the calculated scores. If a first object candidate having a score exceeding the second threshold is identified, information (Info_OC1) used for data processing of the first object candidate can be provided to the resource (200) (S330).

[0080] And the resource (200) can perform first object recognition using information (Info_OC1) used for processing data of the received first object candidate (S335). And the resource (200) can perform a first task corresponding to the first object recognition (S340). And the resource (200) can send a response signal indicating that the first task has been completed to the task manager (120) (S350).

[0081] The task manager (120) can provide the resource (200) with the information (Info_OC1) used for data processing of the first object candidate, and then set the order of data processing for the remaining object candidates using the scores of the remaining object candidates (S345). Then, when the task manager (120) receives a response signal from the resource (200), it can provide the resource (200) with the generated order table (OT) and the information (Info_OC2~Info_OCn) used for data processing of the remaining object candidates (S350).

[0082] The resource (200) can check the order using the received order table (OT) and perform object recognition and tasks corresponding to the object recognition results for each object candidate according to the checked order (S360 to S375). Then, the resource (200) can send a response signal indicating that the tasks have been completed to the task manager (120) (S380).

[0083] Meanwhile, in a modified embodiment, the task manager (120) may sequentially provide information (Info_OC2~Info_OCn) used for data processing of the remaining object candidates to the resource (200) according to a set order. The resource (200) may perform object recognition and tasks corresponding to the object recognition results according to the order in which it receives the information (Info_OC2~Info_OCn) used for data processing of the remaining object candidates.

[0084] FIG. 12 is a flowchart illustrating an example of the operation of a model processor and resources according to one embodiment of the present disclosure. In detail, FIG. 12 is a drawing of a modified embodiment of FIG. 8.

[0085] The object candidate detector (110) can detect object candidates by analyzing an input image. An object candidate detector (110) according to one embodiment of the present disclosure may be implemented to transmit an object candidate to a task manager (120) whenever an object candidate is detected by analyzing an input image. Specific operations for this are described below.

[0086] Referring to FIG. 12, the object candidate detector (110) can detect a first object candidate (OC1) by analyzing an input image (S410). Then, the object candidate detector (110) can transmit the detected first object candidate (OC1) to the task manager (120) (S415).

[0087] Then, the task manager (120) can generate metadata for the received first object candidate (OC1) (S420). Then, the task manager (120) can calculate a score indicating the importance of the first object candidate (OC1) based on the generated metadata (S425). Then, the task manager (120) can check whether the calculated score exceeds a first threshold. The first threshold may represent the lowest value of the score that is determined to require object recognition. If the score of the first object candidate (OC1) exceeds the first threshold, the task manager (120) can transmit information (Info_OC1) used for data processing of the first object candidate (OC1) to the resource (200) (S430).

[0088] The resource (200) can perform a first object recognition based on information used for processing data of the received first object candidate (OC1) (S435). The resource (200) can also perform a first task corresponding to the result of the first object recognition (S440).

[0089] The object candidate detector (110) can detect the next object candidate, the second object candidate (OC2), by continuously analyzing the image after transmitting the first object candidate (OC1) to the task manager (120) (S445). Then, the object candidate detector (110) can transmit the detected second object candidate (OC2) to the task manager (120) (S450).

[0090] Then, the task manager (120) can generate metadata for the received second object candidate (OC2) (S455). Then, the task manager (120) can calculate a score indicating the importance of the second object candidate (OC2) based on the generated metadata (S460). Then, the task manager (120) can check whether the calculated score exceeds a second threshold. If the score of the second object candidate (OC2) exceeds a first threshold, the task manager (120) can transmit information (Info_OC2) used for data processing of the second object candidate (OC2) to the resource (200) (S465).

[0091] The resource (200) can perform second object recognition based on information used for processing data of the received second object candidate (OC2) (S470). The resource (200) can also perform a second task corresponding to the result of the second object recognition (S475).

[0092] The object candidate detector (110), task manager (120), and resource (200) can repeatedly perform the aforementioned series of operations on the remaining object candidates as well (S480 to S510).

[0093] In a neural network system according to one embodiment of the present disclosure, the object candidate detector (110), task manager (120), and resource (200) perform operations in parallel, so the speed of data processing for an input image can be increased.

[0094] FIG. 13 is a block diagram showing a neural network system according to one embodiment of the present disclosure. In detail, FIG. 13 is a block diagram showing a modified embodiment of the neural network system (10) of FIG. 1.

[0095] Referring to FIG. 13, the neural network system (20) may include a model processor (300) and a resource (400), the model processor (300) may include an object candidate detector (310) and a task manager (320), and the resource (400) may include a first task module (410) and a second task module (420). The first task module (410) and the second task module (420) may each perform object recognition and perform a task corresponding to the object recognition result. For example, the first task module (410) and the second task module (420) can each use various types of neural network models such as CNN, R-CNN, fast R-CNN, RPN, RNN, S-DNN, S-SDNN, Deconvolution Network, DBN, RBM, Fully Convolutional Network, LSTM Network, GAN, IV3, Classification Network, etc., such as GoogLeNet, AlexNet, ResNet, VGG Network, etc.

[0096] A series of operations in which the model processor (300) detects multiple object candidates by analyzing an input image (IMG), generates metadata for the multiple object candidates, and sets the data processing order for the multiple object candidates based on the metadata may be identical or similar to the previously described embodiment, so a redundant description is omitted.

[0097] A task manager (320) of a model processor (300) according to one embodiment of the present disclosure can assign a plurality of object candidates to a first task module (410) or a second task module (420) using metadata.

[0098] In one embodiment, the task manager (320) can determine, using metadata, which module is more suitable among the first task module (410) and the second task module (420) for data processing for each of the plurality of object candidates (e.g., a module that can reduce the amount of computation during data processing, increase the computation speed, or reduce power consumption). The task manager (320) can then match the plurality of object candidates with the first task module (410) and the second task module (420) based on the determination result.

[0099] For example, the task manager (320) may assign a first object candidate with a small size to a first task module (410) using an IV3 model and assign a second object candidate with a large size to a second task module (420) using a Resnet model. Meanwhile, the present disclosure is not limited to the examples described above, and multiple object candidates may be assigned to the first task module (410) and the second task module (420) according to various methods.

[0100] And the model processor (300) can transmit information used for data processing of multiple object candidates to the assigned modules in a set order. For example, referring to FIG. 13, when a first object candidate (OC1) is assigned to a first task module (410) and a second object candidate (OC2) is assigned to a second task module (420), the model processor (300) can transmit information (Info_OC1) used for data processing of the first object candidate to the first task module (410) and transmit information (Info_OC2) used for data processing of the second object candidate to the second task module (420).

[0101] And each of the first task module (410) and the second task module (420) can perform object recognition and a task corresponding to the result of object recognition using the received information.

[0102] Meanwhile, although FIG. 13 illustrates and describes the resource (400) as including only the first task module (410) and the second task module (420), the resource (400) may be implemented to have three or more task modules.

[0103] In this way, a neural network system according to one embodiment of the present disclosure can perform efficient computation processing by matching a plurality of object candidates and a plurality of task modules based on metadata.

[0104] FIG. 14 is a block diagram showing a neural network system according to one embodiment of the present disclosure. In detail, FIG. 14 is a block diagram showing a modified embodiment of the neural network system (10) of FIG. 1.

[0105] Referring to FIG. 14, the neural network system (30) may include a model processor (500), a first resource (600_1), and a second resource (600_2), and the model processor (500) may include an object candidate detector (510) and a task manager (520). The first resource (600_1) and the second resource (600_2) may each include a computational resource capable of performing object recognition and performing a task corresponding to the object recognition result. For example, the first resource (600_1) and the second resource (600_2) may each include various computational processing devices such as a CPU, GPU, AP, DSP, FPGA, NPU, ECU, ISP, etc.

[0106] A series of operations in which the model processor (500) detects multiple object candidates by analyzing an input image (IMG), generates metadata for the multiple object candidates, and sets the data processing order for the multiple object candidates based on the metadata may be identical or similar to the previously described embodiment, so a redundant description is omitted.

[0107] A task manager (520) of a model processor (500) according to one embodiment of the present disclosure can assign a plurality of object candidates to a first resource (600_1) or a second resource (600_2) using metadata.

[0108] In one embodiment, the task manager (520) can determine, using metadata, the more suitable resource among the first resource (600_1) and the second resource (600_2) for data processing for each of the plurality of object candidates (e.g., a resource that can reduce the amount of computation, increase the computation speed, or reduce power consumption during task execution). The task manager (520) can then match the plurality of object candidates with the first resource (600_1) and the second resource (600_2) based on the determination result.

[0109] For example, the task manager (520) may allocate a first object candidate having a size of NxN multiple of 32 to a first resource (600_1) which is an NPU, and object candidates having other sizes to a second resource (600_2) which is a CPU or GPU. Meanwhile, the present disclosure is not limited to the examples described above, and multiple object candidates may be allocated to the first resource (600_1) and the second resource (600_2) according to various methods.

[0110] According to a modifiable embodiment, the task manager (520) may assign a plurality of object candidates to the first resource (600_1) and the second resource (600_2) by additionally considering the operational status of each of the first resource (600_1) and the second resource (600_2). Information regarding the operational status may include various information, such as whether data processing of the resource is in progress, information regarding the data processing target, and information regarding data processing in progress.

[0111] In one embodiment, the task manager (520) receives information regarding the operation status from the first resource (600_1) and the second resource (600_2), and, considering the information regarding the operation status and metadata of the plurality of object candidates, may assign the plurality of object candidates to the first resource (600_1) and the second resource (600_2). For example, if the task manager (520) confirms, based on the information regarding the operation status, that the first resource (600_1) is in a waiting state and the second resource (600_2) is in the process of data processing, it may assign the object candidate in the order prior to the first resource (600_1). Meanwhile, the method of assigning the plurality of object candidates to the plurality of resources based on information regarding the operation status of the neural network system according to the technical concept of the present disclosure is not limited to the example described above.

[0112] And the model processor (500) can transmit information used for data processing of multiple object candidates to the allocated resources in a set order. For example, referring to FIG. 14, when a first object candidate (OC1) is allocated to a first resource (600_1) and a second object candidate (OC2) is allocated to a second resource (600_2), the model processor (500) can transmit information (Info_OC1) used for data processing of the first object candidate to the first resource (600_1) and transmit information (Info_OC2) used for data processing of the second object candidate to the second resource (600_2).

[0113] And each of the first resource (600_1) and the second resource (600_2) can perform object recognition and a task corresponding to the result of object recognition using the received information.

[0114] Meanwhile, in FIG. 14, the neural network system (30) is illustrated and described as including only the first resource (600_1) and the second resource (600_2), but it may be implemented to have three or more resources.

[0115] Accordingly, a neural network system according to one embodiment of the present disclosure can match a plurality of object candidates with a plurality of resources based on metadata, or based on metadata and the operational state of a plurality of resources. Accordingly, a neural network system according to one embodiment of the present disclosure can perform efficient computation processing by reducing the amount of computation, increasing the computation speed, and reducing power consumption.

[0116] FIG. 15 is a flowchart illustrating a method of operation of a neural network system according to one embodiment of the present disclosure. In detail, the method of operation of a neural network system according to the present embodiment may include steps performed chronologically in the neural network systems (10, 20, 30) of FIG. 1, FIG. 2, FIG. 13 and FIG. 14.

[0117] Referring to FIG. 15, the neural network system can detect a plurality of object candidates included in the first image (S610). An object candidate may refer to an area within the first image where there is a high probability that an object of interest exists. An object candidate may be referred to as a Region of Interest (RoI). In one embodiment, the neural network system can detect a plurality of object candidates included in the first image using a neural network model such as an RPN.

[0118] The neural network system can perform first object recognition on a first object candidate among a plurality of object candidates (S620). In one embodiment, the neural network system can set a data processing order for a plurality of object candidates using metadata, and perform first object recognition on a first object candidate selected based on the set order.

[0119] Specifically, first, the neural network system can generate metadata for a plurality of object candidates based on a first image. The metadata may include information regarding the size, location within the image, top-left coordinate, bottom-right coordinate, distance from a preset coordinate or area, depth, etc., of each of the plurality of object candidates. Meanwhile, the present disclosure is not limited thereto, and the types of metadata may include additional information in addition to the information described above.

[0120] Furthermore, the neural network system can determine the data processing order for multiple object candidates by calculating scores representing the importance of each of the multiple object candidates based on the generated metadata and sorting the calculated scores in descending order. Then, the neural network system can identify the first object candidate, which is the earliest in the set order, and perform first object recognition on the identified first object candidate.

[0121] And the neural network system can perform a first task corresponding to the first object recognition result (S630). For example, if the neural network system is applied to an autonomous vehicle and the first object candidate is recognized as a car, the neural network system can perform an operation to calculate the distance between the recognized car and the autonomous vehicle as a task corresponding to the recognized car.

[0122] And when the first task is completed, the neural network system can perform second object recognition on the second object candidate (S640). In one embodiment, the neural network system can perform second object recognition on the second object candidate that is next in order to the first object candidate in a set order. And the neural network system can perform a second task corresponding to the result of the second object recognition (S650).

[0123] The neural network system can repeatedly perform the aforementioned object recognition operation and the execution of the task corresponding to the object recognition result for the remaining object candidates.

[0124] A method of operation of a neural network system according to the technical concept of the present disclosure can prevent bottlenecks caused by continuous object recognition operations and can prioritize data processing for object candidates of high importance by performing object recognition and tasks corresponding to object recognition results on an object-by-object basis for a plurality of object candidates included in an image.

[0125] FIG. 16 is a block diagram showing an electronic device according to one embodiment of the present disclosure.

[0126] Referring to FIG. 16, the electronic device (1000) may include a processor (1010), RAM (Random Access Memory) (1020), a model processor (1030), memory (1040), a sensor (1050), and a resource (1060), and the components of the electronic device (1000) may be connected to communicate with each other via a bus (1070). In this case, the model processor (1030) may correspond to the model processors (100, 300, 500) of the aforementioned embodiments, and the resource (1060) may correspond to the resources (200, 400, 600_1, 600_2) of the aforementioned embodiments. In some embodiments, the model processor (1030) and the resource (1060) may be implemented using the embodiments described above with reference to FIGS. 1 through 15.

[0127] The electronic device (1000) can analyze input data in real time based on a neural network to extract valid information, make a situational judgment based on the extracted information, or control the configurations of the electronic device on which the electronic device (1000) is mounted. For example, the electronic device (1000) can be applied to robot devices such as drones, Advanced Driver Assistance Systems (ADAS), smart TVs, smartphones, medical devices, mobile devices, video display devices, measurement devices, IoT devices, etc., and can also be mounted on at least one of various types of electronic devices.

[0128] The processor (1010) can control the overall operation of the electronic device (1000). For example, the processor (1010) can control the function of the model processor (1030) by executing programs stored in the RAM (1020). The RAM (1020) can temporarily store programs, data, applications, or instructions.

[0129] The sensor (1050) can sense or receive a signal (e.g., image signal, voice signal, magnetic signal, bio-signal, touch signal, etc.) from outside the electronic device (1000) and convert it into data. According to one embodiment of the present disclosure, the sensor (1050) may include an image sensor that receives an image signal from the surroundings of the electronic device (1000) and converts it into data in the form of an image. Additionally, the electronic device (1000) may include a plurality of sensors (1050).

[0130] The model processor (1030) can perform neural network operations by controlling the resource (1060) and generate information signals based on the results of the operation. In one example, the model processor (1030) can be implemented as software stored in system memory (e.g., ROM (Read-Only Memory)) and can operate based on the control of the processor (1010). The memory (1040) is a storage place for storing data and can store various data generated during the operation of the model processor (1030) and the resource (1060), for example.

[0131] According to one embodiment of the present disclosure, a model processor (1030) receives an image from a sensor (1050) and can detect a plurality of object candidates included in the received image. The model processor (1030) can generate metadata for the detected plurality of object candidates and set a data processing order for the plurality of object candidates based on the generated metadata. The model processor (1030) can sequentially provide information used for data processing of the plurality of object candidates to a resource (1060) according to the set order.

[0132] The resource (1060) may include a computation resource for performing a plurality of operations based on a neural network or a communication resource implemented with various wired or wireless interfaces capable of communicating with an external device. According to one embodiment of the present disclosure, the resource (1060) may sequentially perform data processing for a plurality of object candidates on an object-by-object basis according to the order in which information used for data processing of a plurality of object candidates is received from the model processor (1030). According to one embodiment of the present disclosure, the resource (1060) may include a plurality of resources, and the plurality of resources may be of the same type or different type.

[0133] Accordingly, the electronic device (1000) according to the technical concept of the present disclosure can perform fine-grained control using object recognition technology by prioritizing data processing for object candidates that are judged to be of high importance.

[0134] FIG. 17 is a block diagram showing an autonomous driving device according to one embodiment of the present disclosure.

[0135] Referring to FIG. 17, the autonomous driving device (2000) may include a processor (2010), RAM (2020), a model processor (2030), a memory (2040), a sensor (2050), a resource (2060), a driving unit (2070), and a communication interface (2080), and the components of the autonomous driving device (2000) may be connected to communicate with each other through a bus (2090). At this time, the model processor (2030) may correspond to the model processors (100, 300, 500, 1030) of the aforementioned embodiments, and the resource (2060) may correspond to the resources (200, 400, 600_1, 600_2, 1060) of the aforementioned embodiments. In some embodiments, the model processor (2030) and resource (2060) may be implemented using the embodiments described above with reference to FIGS. 1 through 16.

[0136] The autonomous driving device (2000) can perform situational judgment and vehicle operation control by analyzing data of the surrounding environment of the autonomous vehicle in real time based on a neural network.

[0137] The processor (2010) can control the overall operation of the autonomous driving device (2000). For example, the processor (2010) can control the function of the model processor (2030) by executing programs stored in the RAM (2020). The RAM (2020) can temporarily store programs, data, applications, or instructions.

[0138] The sensor (2050) may include a plurality of sensors that receive a video signal regarding the surrounding environment of the autonomous driving device (2000) and output it as an image. For example, the sensor (2050) may include an image sensor (2051) such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor), a LiDAR (Light Detection And Ranging) sensor (2052), a Radar (Radio Detecting And Ranging) sensor (2053), a depth camera (2054), etc. Meanwhile, the present disclosure is not limited thereto and may include an ultrasonic sensor (not shown), an infrared sensor (not shown), etc.

[0139] The model processor (2030) can perform operations of the neural network by controlling the resource (2060) and generate information signals based on the results of the operation. The memory (2040) is a storage place for storing data, and can store, for example, various data generated during the operation of the model processor (2030) and the resource (2060).

[0140] According to one embodiment of the present disclosure, a model processor (2030) receives an image of the surrounding environment of an autonomous vehicle device (2000) captured from a sensor (2050) and can detect a plurality of object candidates included in the received image. The model processor (2030) can generate metadata for the detected plurality of object candidates and set a data processing order for the plurality of object candidates based on the generated metadata. The model processor (2030) can sequentially provide information used for data processing of the plurality of object candidates to a resource (2060) according to the set order.

[0141] The resource (2060) may include a computation resource for performing a plurality of operations based on a neural network or a communication resource implemented with various wired or wireless interfaces capable of communicating with an external device. According to one embodiment of the present disclosure, the resource (2060) may sequentially perform data processing for a plurality of object candidates on an object-by-object basis according to the order in which information used for data processing of a plurality of object candidates is received from the model processor (2030). According to one embodiment of the present disclosure, the resource (2060) may include a plurality of resources, and the plurality of resources may be of the same type or different type.

[0142] The driving unit (2070) is configured to drive the autonomous driving device (2000) and may include an engine and motor (2071), a steering unit (2072), and a brake unit (2073). In one embodiment, the driving unit (2070) can adjust the propulsion, braking, speed, direction, etc. of the autonomous driving device (2000) using the engine and motor (2071), the steering unit (2072), and the brake unit (2073) according to the control of the processor (2010).

[0143] The communication interface (2080) can communicate with an external device using a wired or wireless communication method. For example, the communication interface (2080) can communicate using a wired communication method such as Ethernet, or communicate using a wireless communication method such as Wi-Fi or Bluetooth.

[0144] The processor (2010) can generate control commands to control the autonomous driving device (2000) using information generated as the resource (2060) performs data processing. For example, the resource (2060) can recognize fire as an object included in an image output from the sensor (2050) and generate information about the emergency reporting phone number 119 as a task corresponding to the fire. Then, the processor (2010) can control the communication interface (2080) to make a call to the emergency signal phone number 119. As another example, the resource (2060) can recognize fire and perform a task to change the driving route of the autonomous driving device (2000) as a task corresponding to the fire. Then, the processor (2010) can control the drive unit (2070) to drive according to the changed autonomous driving.

[0145] As described above, exemplary embodiments have been disclosed in the drawings and specification. Although specific terms have been used to describe the embodiments in this specification, they are used only for the purpose of explaining the technical concept of this disclosure and are not intended to limit the meaning or the scope of this disclosure as defined in the claims. Therefore, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of this disclosure should be determined by the technical concept of the appended claims.

Claims

Claim 1 A neural network system comprising: a processor that detects a plurality of object candidates included in a first image by analyzing a first image, generates metadata of the plurality of object candidates based on the first image, and sets a data processing order of the plurality of object candidates based on the metadata; and a resource that performs data processing of the plurality of object candidates; wherein the resource performs a first object recognition for a first object candidate in a first order and performs a first task corresponding to the result of the first object recognition in accordance with the order in which information used for data processing of the plurality of object candidates is received from the processor, and wherein the processor provides information used for data processing for a second object candidate in a second order, which is the next order after the first order among the plurality of object candidates, to the resource in accordance with the data processing order. Claim 2 A neural network system according to claim 1, wherein the resource receives information used for data processing of the second object candidate, performs second object recognition for the second object candidate, and performs a second task corresponding to the result of the second object recognition. Claim 3 A neural network system according to claim 1, wherein the processor calculates scores representing the importance of the plurality of object candidates based on the metadata, and sets the data processing order of the plurality of object candidates using the calculated scores. Claim 4 A neural network system according to claim 3, characterized in that, when the processor identifies an object candidate having a score exceeding a first threshold, it immediately provides information used for data processing of the identified object candidate to the resource. Claim 5 A neural network system according to claim 3, wherein the processor, upon identifying an object candidate having a score less than a second threshold, sets the data processing order of the remaining object candidates excluding the identified object candidate. Claim 6 A neural network system according to claim 1, wherein the metadata comprises at least one of the size of each of the plurality of object candidates, the position within the first image, the top-left coordinate, the bottom-right coordinate, the distance from a preset coordinate, and the depth. Claim 7 A neural network system according to claim 1, wherein the resource comprises a plurality of resources, and the processor matches the plurality of object candidates with the plurality of resources based on the metadata of the plurality of object candidates, and provides information used for data processing of the plurality of object candidates to the plurality of resources according to the matching result. Claim 8 A method of operation of a neural network system for processing a first image, comprising: a step of detecting a plurality of object candidates included in the first image; a step of performing a first object recognition for a first object candidate in a first order among the plurality of object candidates; a step of performing a first task corresponding to the result of the first object recognition; a step of, when the first task is completed, performing a second object recognition for a second object candidate in a second order following the first order among the plurality of object candidates; and a step of performing a second task corresponding to the result of the second object recognition. Claim 9 A method of operation according to claim 8, further comprising: a step of generating metadata for the plurality of object candidates based on the first image; a step of selecting the first object candidate among the plurality of object candidates based on the metadata; and a step of selecting a second object candidate among the remaining object candidates excluding the first object candidate based on the metadata. Claim 10 An electronic device comprising: a resource for performing operations; a sensor for acquiring data surrounding the electronic device and converting the acquired data into an image form to output a first image; a memory for storing program information including an object candidate detector for detecting a plurality of object candidates within the first image and a task manager for generating metadata of the plurality of object candidates based on the first image and selecting an object candidate to perform object recognition among the plurality of object candidates based on the metadata; and a processor for selecting a first object candidate based on the program information and providing information about the first object candidate to the resource; wherein the resource performs a first object recognition of the first object candidate in a first order based on the information about the first object candidate received and performs a first task corresponding to the result of the first object recognition, and the processor, when the first task is completed, provides information to the resource that is used for data processing of a second object candidate in a second order, which is the next order after the first order among the plurality of object candidates.

Images