A neural network system and method of operation thereof

Abstract

A neural network system includes a processor configured to detect a plurality of candidate objects included in a first image, generate metadata corresponding to the plurality of candidate objects based on the first image, and set a data processing order of the plurality of candidate objects based on the metadata, and at least one resource configured to perform data processing with respect to the plurality of candidate objects. The processor is configured to sequentially provide information about data processing of the plurality of candidate objects to the at least one resource according to the set data processing order, and the at least one resource is configured to sequentially perform data processing with respect to the plurality of candidate objects according to an order in which information about data processing of each of the plurality of candidate objects is received.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2019-0162881, filed with the Korean Intellectual Property Office on December 9, 2019, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The example embodiments relate to a neural network system and a method of operating the same, and more specifically, to a neural network system and a method of operating the same, wherein data processing for candidate objects included in an image is performed on an object-by-object basis. Background Technology

[0004] Artificial neural networks (ANNs) refer to computational architectures that use the biological brain as a model. Deep learning, machine learning, and other computational tasks can be implemented based on ANNs. Due to the rapid increase in the amount of computation required to process data using ANNs recently, there is a need to use ANNs to efficiently perform computational processing. Summary of the Invention

[0005] One or more example embodiments provide a neural network system and its operating method, wherein the neural network system sets a processing order for multiple candidate objects included in an image based on metadata of multiple candidate objects, performs object recognition for multiple objects according to the set order, and performs tasks corresponding to the object recognition results on an object-by-object basis. Therefore, bottlenecks caused by continuous object recognition operations can be prevented, and data processing can be prioritized for candidate objects of high importance, thereby improving the efficiency of data processing for object recognition.

[0006] According to one aspect of an example embodiment, a neural network system is provided, comprising: a processor configured to detect a plurality of candidate objects included in a first image, generate metadata corresponding to the plurality of candidate objects based on the first image, and set a data processing order for the plurality of candidate objects based on the metadata; and at least one resource configured to perform data processing on the plurality of candidate objects, wherein the processor is further configured to sequentially provide information relating to data processing of the plurality of candidate objects to the at least one resource according to the set data processing order of the plurality of candidate objects, and the at least one resource is further configured to sequentially perform data processing on the plurality of candidate objects according to the order in which information relating to data processing of each of the plurality of candidate objects is received.

[0007] According to one aspect of an example embodiment, a method for operating a neural network system is provided, the method comprising: detecting a plurality of candidate objects included in a first image; performing a first object recognition for a first candidate object among the plurality of candidate objects; performing a first task corresponding to the result of the first object recognition; after completing the first task, performing a second object recognition for a second candidate object among the plurality of candidate objects; and performing a second task corresponding to the result of the second object recognition.

[0008] According to one aspect of an example embodiment, an electronic device is provided, comprising: a sensor configured to acquire data about the vicinity of the electronic device and output a first image based on the acquired data; at least one resource configured to perform object recognition on the first image; a memory configured to store a program; and a processor configured to read the program and operate according to instructions of the program to detect a plurality of candidate objects in the first image, generate metadata of the plurality of candidate objects based on the first image, and provide information related to a first candidate object selected from the plurality of candidate objects based on the metadata to the at least one resource, wherein the at least one resource is further configured to perform a first object recognition on the first candidate object and perform a first task corresponding to the result of the first object recognition. Attached Figure Description

[0009] The above and other aspects, features, and advantages of specific exemplary embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0010] Figure 1 This is a block diagram of a neural network system according to an example embodiment;

[0011] Figure 2 This is a detailed block diagram of a neural network system according to an example embodiment;

[0012] Figure 3 This is a conceptual diagram illustrating an object recognition method according to an example embodiment;

[0013] Figure 4 Metadata of candidate objects according to an example embodiment is shown;

[0014] Figure 5 A plurality of candidate objects included in an image are shown according to an example embodiment;

[0015] Figure 6 This is a conceptual diagram illustrating a method for generating a sequence list according to an example embodiment;

[0016] Figure 7 This is a conceptual diagram illustrating a method for generating a sequence list according to an example embodiment;

[0017] Figure 8 This is a flowchart illustrating an example of the operation of a model processor and resources according to an example embodiment;

[0018] Figure 9 An example of the operation of the model processor and resources according to an example embodiment is shown;

[0019] Figure 10 This is a conceptual diagram illustrating a method for generating a sequence list according to an example embodiment;

[0020] Figure 11 This is a flowchart illustrating an example of the operation of a model processor and resources according to an example embodiment;

[0021] Figure 12 An example of the operation of the model processor and resources according to an example embodiment is shown;

[0022] Figure 13 This is a block diagram of a neural network system according to an example embodiment;

[0023] Figure 14 This is a block diagram of a neural network system according to an example embodiment;

[0024] Figure 15 This is a flowchart illustrating a method of operating a neural network system according to an example embodiment;

[0025] Figure 16 This is a block diagram of an electronic device according to an example embodiment; and

[0026] Figure 17 This is a block diagram of an autonomous driving device according to an example embodiment. Detailed Implementation

[0027] In the following description, exemplary embodiments will be illustrated in detail with reference to the accompanying drawings. However, the scope of this disclosure should not be construed as limited to the exemplary embodiments set forth herein. Throughout this disclosure, similar reference numerals in the drawings refer to similar elements.

[0028] Figure 1 This is a block diagram of a neural network system according to an example embodiment. (Reference) Figure 1 The neural network system 10 may include a model processor 100 and resources 200. The model processor 100 may include a candidate object detector 110 and a task manager 120.

[0029] The neural network system 10 can perform neural network-based neural tasks based on various neural networks. Neural networks can include, but are not limited to, various types of neural network models, such as convolutional neural networks (CNNs) (e.g., GoogLeNet, AlexNet, ResNet, VGG Network, etc.), regions with CNNs (R-CNN), fast R-CNN, region proposal networks (RPN), recurrent neural networks (RNNs), stack-based deep neural networks (S-DNNs), state-space dynamic neural networks (S-SDNNs), deconvolutional networks, deep belief networks (DBNs), restricted Boltzmann machines (RBMs), fully convolutional networks, long short-term memory (LSTM) networks, generative adversarial networks (GANs), Inception V3 (IV3), classification networks, etc. The neural network performing the task can include sub-neural networks that can be implemented using homogeneous or heterogeneous neural network models.

[0030] The neural network system 10 can receive an image IMG from an external source and perform data processing on the received image IMG. In an example embodiment, the neural network system 10 can perform data processing on the received image IMG by using a neural network model. In an example embodiment, the neural network system 10 can detect multiple candidate objects included in the received image IMG by using a neural network model such as RPN, R-CNN, Fast R-CNN, etc., perform object recognition on the multiple candidate objects, and perform a task corresponding to the recognition result.

[0031] Model processor 100 may include a candidate object detector 110 and a task manager 120. Components included in model processor 100 may be implemented using software and neural network platforms such as Evolutionary Deep Networks (EDEN). In another example, components included in model processor 100 may be implemented using hardware such as electronic circuits. In an example embodiment, model processor 100 may be implemented using a neural network framework.

[0032] Candidate object detector 110 can receive and analyze an image IMG from an external source, resulting in the detection of multiple candidate objects included in the image IMG. A candidate object can refer to a region in the image IMG where there is a high probability of the presence of an object of interest. A candidate object can refer to a region of interest (RoI). In an example embodiment, candidate object detector 110 can detect multiple candidate objects included in the image IMG using a neural network model such as RPN. Candidate object detector 110 can then provide the detected multiple candidate objects to task manager 120.

[0033] According to this disclosure, the task manager 120 can receive multiple candidate objects and set a data processing order for the multiple candidate objects. In an example embodiment, the task manager 120 can set the data processing order of the multiple candidate objects according to their priority. The task manager 120 can provide information Info_OC related to the data processing of the multiple candidate objects according to the set data processing order. For example, the task manager 120 can provide information related to the data processing of a first candidate object, which has a prior order among the multiple candidate objects of the image IMG, to the resource 200. When the task manager 120 receives a response from the resource 200 indicating that the task execution for the first candidate object is complete, the task manager 120 can provide information related to the data processing of a second candidate object in the next order to be processed to the resource 200.

[0034] Resource 200 may include computing resources capable of performing calculations based on data received from task manager 120. For example, resource 200 may include various computing processing devices such as a central processing unit (CPU), graphics processing unit (GPU), application processor (AP), digital signal processor (DSP), field-programmable gate array (FPGA), neural network processing unit (NPU), electronic control unit (ECU), image signal processor (ISP), etc. Resource 200 according to an example embodiment may include multiple computing resources that are homogeneous or heterogeneous. For example, resource 200 may include a first resource (not shown) and a second resource (not shown), both of which are NPUs, or a first resource (not shown) serving as an NPU and a second resource (not shown) serving as a GPU.

[0035] According to the present disclosure, resource 200 can perform data processing on multiple candidate objects in the order in which information related to data processing of multiple candidate objects is received from task manager 120. In an example embodiment, resource 200 can perform object identification on multiple candidate objects in the order in which information related to data processing of multiple candidate objects is received, and perform a task corresponding to the object identification result. The present disclosure is not limited thereto, and data processing may include separate computations in addition to object identification, and may include one computation or three or more computations.

[0036] According to the resource 200 of this disclosure, data processing can be performed on multiple candidate objects on a per-candidate-object basis. More specifically, when there are multiple candidate objects and multiple computational operations required for data processing of one candidate object, resource 200 can perform computational operations on the next candidate object after the computational operation for that candidate object is completed. For example, when resource 200 first receives information related to data processing of a first candidate object from task manager 120 and then receives information related to data processing of a second candidate object from task manager 120, resource 200 can perform first object identification and execute a first task corresponding to the result of the first object identification for the first candidate object, and perform second object identification and execute a second task corresponding to the result of the second object identification for the second candidate object.

[0037] The neural network system 10 according to this disclosure can set an order based on the priority of multiple candidate objects included in an image, perform object recognition on multiple candidate objects according to the set data processing order, and perform object recognition and tasks corresponding to the object recognition results on an object-by-object basis according to the set data processing order. Therefore, bottlenecks caused by continuous object recognition operations can be prevented, and data processing can be prioritized for candidate objects with high importance.

[0038] Figure 2 This is a detailed block diagram of a neural network system according to an example embodiment. More specifically, Figure 2 Detailed illustration Figure 1 The neural network system 10.

[0039] refer to Figure 2 The neural network system 10 may include a model processor 100 and resources 200, the candidate object detector 110 may include a feature map generator 111 and an RoI detector 112, and the task manager 120 may include a metadata generator 121 and a task scheduler 122.

[0040] Feature map generator 111 can generate feature maps using an image IMG received from an external source. In an example embodiment, feature map generator 111 can generate a feature map (FM) indicating features of the image IMG by performing convolution computation using the image IMG received from an external source. This disclosure is not limited thereto, and feature map generator 111 can generate feature map FM based on separate computation. Feature map generator 111 can provide the generated feature map FM to RoI detector 112.

[0041] RoI detector 112 can detect RoIs (e.g., candidate objects) by analyzing feature maps FM received from feature map generator 111. A RoI is a region where there is a high probability of the presence of an object of interest. In an example embodiment, RoI detector 112 can detect multiple RoIs (e.g., multiple candidate objects OC1 to OCn) using an RPN model. This disclosure is not limited thereto, and RoI detector 112 can detect multiple RoIs using a separate model. RoI detector 112 can provide the detected multiple candidate objects OC1 to OCn to task manager 120. According to an example embodiment, RoI detector 112 can provide the input image IMG along with the multiple candidate objects OC1 to OCn to task manager 120.

[0042] Task manager 120 can receive multiple candidate objects OC1 to OCn from candidate object detector 110, generate metadata MD1 to MDn of multiple candidate objects OC1 to OCn based on input image IMG, and set the data processing order for multiple candidate objects OC1 to OCn based on the generated metadata MD1 to MDn.

[0043] More specifically, the metadata generator 121 can receive multiple candidate objects OC1 to OCn from the candidate object detector 110, and generate metadata MD1 to MDn corresponding to the multiple candidate objects OC1 to OCn using the image IMG and the received multiple candidate objects OC1 to OCn. The metadata may include information about the size of each of the multiple candidate objects, its position in the image IMG, its upper-left coordinate, lower-right coordinate, distance from preset coordinates or preset area, depth, etc. This disclosure is not limited thereto, and the type of metadata may include additional information as well as the information described above. The metadata generator 121 can provide the generated metadata MD1 to MDn to the task scheduler 122.

[0044] Task scheduler 122 can receive metadata MD1 to MDn of multiple candidate objects OC1 to OCn from metadata generator 121, and set the data processing order of the multiple candidate objects based on the metadata MD1 to MDn. In an example embodiment, task scheduler 122 can use the metadata MD1 to MDn to calculate a score indicating the importance of the multiple candidate objects according to one or more criteria, and use the calculated score to set the data processing order of the multiple candidate objects OC1 to OCn.

[0045] Task manager 120 can provide information Info_OC1 to Info_OCn related to the data processing of multiple candidate objects OC1 to OCn according to the set data processing order. In an example embodiment, the information Info_OC1 to Info_OCn related to the data processing of multiple candidate objects OC1 to OCn may include multiple candidate objects OC1 to OCn and metadata MD1 to MDn. That is, task scheduler 122 can provide resource 200 with the RoIs and their metadata detected by RoI detector 112.

[0046] Resource 200 can perform object recognition on multiple candidate objects OC1 to OCn in the order of receiving information Info_OC1 to Info_OCn related to data processing of multiple candidate objects OC1 to OCn, and perform a task corresponding to the object recognition result. The task corresponding to the object recognition result may include one or more tasks, and can be variably set according to the type or purpose of the device including the neural network system 10, the user, designer, or manufacturer's settings. For example, when the neural network system 10 is included in an autonomous vehicle and the identified object is a "car", resource 200 can perform operations to calculate the distance between the identified "car" and the autonomous vehicle, and to generate control commands for controlling the speed of the autonomous vehicle based on the calculated distance, as a task corresponding to the identified "car".

[0047] Figure 3 This is a conceptual diagram illustrating an object recognition method according to an example embodiment. More specifically, Figure 3 This is a conceptual diagram illustrating an object recognition method for a neural network system according to an example embodiment.

[0048] refer to Figures 1 to 3 According to the example embodiment, the model processor 100 of the neural network system 10 can detect multiple candidate objects by analyzing the input image IMG. The method for detecting candidate objects can be the same as described above. Figure 1 and Figure 2 The method described is the same, so it will not be described again. The model processor 100 can generate metadata for multiple detected candidate objects and set the data processing order of the candidate objects based on the generated metadata.

[0049] The model processor 100 can sequentially provide information Info_OC1 to Info_OCn related to the data processing of multiple candidate objects OC1 to OCn to the resource 200 according to the set data processing order. For example, refer to Figure 3The model processor 100 can provide the resource 200 with information Info_OC1 related to the data processing of the first candidate object among the multiple candidate objects OC1 to OCn of the image IMG, which has a preceding order. When the task manager 120 receives a response from the resource 200 indicating that the task execution for the first candidate object is complete, the task manager 120 can provide the resource 200 with information Info_OC2 related to the data processing of the next candidate object OC2.

[0050] Resource 200 can receive information Info_OC1 to Info_OCn related to data processing of multiple candidate objects OC1 to OCn from model processor 100, and performs data processing on the multiple candidate objects OC1 to OCn according to the order in which the information Info_OC1 to Info_OCn related to data processing of the multiple candidate objects OC1 to OCn is received. For example, refer to Figure 3 When resource 200 receives information Info_OC1 for the first candidate object, resource 200 can perform a first object recognition OR1 on the first candidate object OC1 and execute a first task T1 corresponding to the result of the first object recognition OR1. When resource 200 receives information Info_OC2 for the second candidate object, resource 200 can perform a second object recognition OR2 on the second candidate object OC2 and execute a second task T2 corresponding to the result of the second object recognition OR2. In this way, resource 200 can similarly perform object recognition OR3…ORn on other candidate objects OC3…OCn, and then execute tasks T3…Tn corresponding to the corresponding object recognition results.

[0051] According to the prior art, the model processor 100 of a neural network system can detect multiple candidate objects OC1 to OCn by analyzing an input image IMG, and provides information related to the data processing of the multiple candidate objects OC1 to OCn to the resource 200. The resource 200 can perform object recognition for the multiple candidate objects OC1 to OCn based on the information received from the model processor 100 related to the data processing of the multiple candidate objects OC1 to OCn, and perform a task corresponding to the object recognition result. For example, refer to... Figure 3 Only after resource 200 performs object recognition OR1 to ORn on multiple candidate objects OC1 to OCn can resource 200 execute tasks T1 to Tn corresponding to the object recognition results.

[0052] In other words, existing neural network systems can perform object recognition on a frame-by-frame basis. Therefore, the time required for the neural network system to begin the task corresponding to the object recognition results may increase proportionally to the number of objects (or, candidate objects) included in the image IMG.

[0053] Recently, techniques for using object recognition in images have not only increased the number of images on which object recognition needs to be performed, but also increased the types and number of objects of interest to be identified in the images. Therefore, when using neural network systems according to existing techniques, the increased time required to perform object recognition on images may delay the start time of tasks corresponding to the object recognition results, and may cause bottlenecks, for example, due to the long execution time of object recognition. In particular, in techniques requiring precise control using object recognition, such as autonomous driving that controls a vehicle based on the identification of objects near the vehicle, the aforementioned problems are a serious limitation.

[0054] The neural network system 10 according to the example embodiment can perform object recognition on an object-by-object basis. Therefore, the time required until the neural network system 10 begins the task corresponding to the object recognition result (e.g., Figure 3 The time t1 in the equation may not include the time required to perform object recognition OR2 to ORn for other candidate objects. Therefore, the time t1 required until the task corresponding to the object recognition result begins can be less than the time required in neural network systems according to the prior art.

[0055] Therefore, when using the neural network system according to the example embodiment, bottlenecks caused by frame-based object recognition can be prevented, the start time of tasks corresponding to object recognition results can be advanced, and data processing can be prioritized for candidate objects with high importance. Thus, the neural network system according to the example embodiment can perform fine-grained control based on object recognition.

[0056] Figure 4 Metadata of candidate objects according to an example embodiment is shown. More specifically, Figure 4 It shows the result of Figure 2 Metadata of candidate objects generated by Task Manager 120.

[0057] refer to Figure 2 and Figure 4 Task Manager 120 can be accessed from the candidate object detector ( Figure 2110) Receives multiple candidate objects OC1 to OCn and generates a metadata table MT for the multiple candidate objects OC1 to OCn. The metadata table MT may include metadata MD1 to MDn for the multiple candidate objects OC1 to OCn. The metadata MD1 to MDn may include various information represented in the form of names (or labels, indexes, identifiers, etc.) and vectors for the multiple candidate objects OC1 to OCn. For example, the metadata MD1 to MDn may include information about the size of each of the multiple candidate objects OC1 to OCn, its position in the image IMG, its upper-left coordinates, lower-right coordinates, distance from preset coordinates or preset areas, depth, etc. This disclosure is not limited thereto, and the type of metadata may include additional information as well as the information described above. For example, although not in Figure 4 As shown, however, the metadata MD1 to MDn may also include metadata of objects that correspond to candidate objects in the current frame of the image among the objects identified in the previous frame of the image.

[0058] Task Manager 120 can generate a metadata table MT and store the generated metadata table MT in a memory area (not shown) included in Model Processor 100 or in an external memory area (not shown) accessible by Task Manager 120.

[0059] In the following text, reference will be made to Figures 5 to 7 This describes a method for setting the data processing order of multiple candidate objects using the metadata table MT generated by the model processor 100.

[0060] Figure 5 A plurality of candidate objects included in an image are shown according to an example embodiment. References Figure 5 The image, shown by IMG, depicts the scene ahead of an autonomous vehicle, captured using cameras attached to the vehicle.

[0061] refer to Figure 1 , Figure 2 and Figure 5 The candidate object detector 110 can receive an image IMG, analyze the received image IMG, and detect multiple candidate objects included in the image IMG. For example, refer to... Figure 5 The candidate object detector 110 can detect six candidate objects OC1 to OC6 included in the image IMG by using a neural network model such as RPN. The candidate object detector 110 can provide the detected candidate objects OC1 to OC6 to the task manager 120.

[0062] Figure 6 This is a conceptual diagram illustrating a method for generating a sequence list according to an example embodiment. (Reference) Figure 1 , Figure 2 , Figures 4 to 6 The metadata generator 121 of the task manager 120 can receive multiple candidate objects from the candidate object detector 110 and generate a metadata table MT for the received candidate objects. The metadata generator 121 can provide the generated metadata table MT to the task scheduler 122.

[0063] According to this disclosure, the task scheduler 122 can generate a score table ST for multiple candidate objects using a metadata table MT. The score table ST may include the name (or label, index, identifier, etc.) of each of the multiple candidate objects and a score indicating the importance of each of the multiple candidate objects according to one or more criteria. One or more criteria may be preset according to the application and purpose of the neural network system according to the example embodiment. In the example embodiment, the task scheduler 122 can calculate the scores of the multiple candidate objects by inputting metadata information into a function implemented using multiple inputs and multiple weights. For example, the preset criteria may be the distance and size of the candidate objects, and the task scheduler 122 can calculate the score of each of the multiple candidate objects using the following function:

[0064] Score = k1 × 1 / |distance| + k2 × size

[0065] (k1 represents the weight of the reciprocal of the distance, and k2 represents the weight of the size).

[0066] Functions that calculate scores using metadata are not limited to the examples above and can be implemented in various forms. For example, see reference... Figure 5 The candidate object with the highest score can be the candidate object OC6, which is located closest to the preset coordinates (e.g., the coordinates corresponding to an autonomous vehicle) and has a larger size. The candidate object with the lowest score can be the candidate object OC3, which is located furthest from the preset coordinates and has a smaller size.

[0067] According to this disclosure, the task scheduler 122 can generate a sequence list OT using a generated score table ST. In an example embodiment, the task scheduler 122 can generate a sequence list OT indicating the data processing order of multiple candidate objects by arranging the scores of multiple candidate objects included in the score table ST in descending order. For example, refer to... Figure 6 The task scheduler 122 can use the score table ST to generate a sequence table OT that indicates the order from candidate OC6 with the highest score to candidate OC3 with the lowest score.

[0068] Task scheduler 122 can sequentially provide information about multiple candidate objects to resource 200 using the generated sequence list OT. For example, refer to Figure 6Task scheduler 122 can identify the first-order candidate object OC6 using the sequence list OT and provide information about the identified candidate object OC6 to resource 200. Task scheduler 122 can identify the next-order candidate object OC1 using the sequence list OT and provide information about the identified candidate object OC1 to resource 200. Task scheduler 122 can repeat the above operations for other candidates in the sequence list OT.

[0069] According to an example embodiment of this disclosure, task scheduler 122 may provide a sequence list OT along with information about a plurality of candidate objects to resource 200. Resource 200 may perform object identification for candidate objects in a first order using the received sequence list OT, and execute a task corresponding to the object identification result. Resource 200 may perform object identification for candidate objects in a next order, and execute a task corresponding to the object identification result.

[0070] For example, refer to Figure 6 Resource 200 can perform object recognition on the first candidate object OC6 using the sequence list OT, and execute the task corresponding to the object recognition result. Resource 200 can also perform object recognition on the next candidate object OC1 using the sequence list OT, and execute the task corresponding to the object recognition result. Task scheduler 122 can repeat the above operations for other candidate objects in the sequence list OT.

[0071] exist Figure 6 Explanation and reference Figure 6 The description states that task scheduler 122 generates a sequence list OT, which can be implemented to generate a linked list, search tree, or first-in-first-out (FIFO) that indicates the data processing order of multiple candidate objects.

[0072] Figure 7 This is a conceptual diagram illustrating a method for generating a sequence list according to an example embodiment. More specifically, Figure 7 It shows Figure 6 A conceptual diagram of an example of a modified embodiment.

[0073] Figure 1 , Figure 2 and Figures 4 to 7 In the process of generating the sequence list OT using the score table ST, the task scheduler 122 can set the order for candidate objects other than those with scores less than a first threshold. The first threshold may refer to the minimum score based on which the object to be identified is to be identified.

[0074] For example, in Figure 7In an example embodiment, the first threshold may be 50. Task scheduler 122 can identify candidate object OC3 in the score table ST whose score is less than the first threshold 50, and arrange the scores of other candidate objects OC1, OC2, and OC4 to OC6 in descending order, thereby generating a sequence list OT. Task scheduler 122 can provide information about multiple candidate objects to resource 200 using the generated sequence list OT.

[0075] The neural network system according to the example embodiment can reduce computation and increase computation speed by performing object recognition on candidate objects other than those that do not require object recognition, based on the lower scores of the candidate objects.

[0076] Figure 8 This is a flowchart illustrating an example of the operation of a model processor and resources according to an example embodiment. More specifically, it shows... Figure 1 or Figure 2 The flowchart shows an example of the operation of the model processor 100 and resource 200.

[0077] refer to Figure 8 In operation S110, the candidate object detector 110 can detect multiple candidate objects OC1 to OCn by analyzing the input image. In operation S115, the candidate object detector 110 can provide the detected multiple candidate objects OC1 to OCn to the task manager 120. According to the example embodiment, the candidate object detector 110 can provide the input image together with the multiple candidate objects OC1 to OCn to the task manager 120.

[0078] Task manager 120 can generate metadata for multiple received candidate objects OC1 to OCn. In operation S125, task manager 120 can use the generated metadata to set the data processing order for the multiple candidate objects OC1 to OCn. In an example embodiment, as shown in reference... Figure 6 As described, Task Manager 120 can use metadata to calculate scores that indicate the importance of multiple candidate objects OC1 to OCn respectively, and generate a sequence list OT by using the calculated scores.

[0079] In the example embodiment, as referenced Figure 7The task manager 120 can generate a sequence list OT using candidate objects other than those whose scores are less than a first threshold in the calculated scores. The task manager 120 can provide information Info_OC1 related to the data processing of the first candidate object to the resource 200 according to the set data processing order. In an example embodiment, the task manager 120 can identify the set data processing order in the generated sequence list OT and provide information Info_OC1 related to the data processing of the first candidate object to the resource 200 according to the set data processing order. According to an example embodiment, the task manager 120 can provide the input image and the information Info_OC1 related to the data processing of the first candidate object together to the resource 200.

[0080] In operation S135, resource 200 can perform first object identification using the received information Info_OC1 related to data processing of the first candidate object. In operation S140, resource 200 can perform a first task corresponding to the first object identification. In operation S145, resource 200 can send a response indicating the completion of the first task to task manager 120.

[0081] When Task Manager 120 receives a response from Resource 200, in Operation S150, Task Manager 120 can identify a second candidate object corresponding to the next order of the first candidate object according to the set data processing order, and provide Resource 200 with information Info_OC2 related to the data processing of the second candidate object.

[0082] In operation S155, resource 200 can perform second object identification using the received information Info_OC2 related to the data processing of the second candidate object. In operation S160, resource 200 can perform a second task corresponding to the second object identification. In operation S165, resource 200 can send a response indicating the completion of the second task to task manager 120.

[0083] In operations S170 to S185, task manager 120 and resource 200 can repeatedly perform operations that provide information related to data processing of candidate objects included in the sequence list OT, perform object identification operations, perform tasks corresponding to the object identification results, and send responses indicating task completion for candidate objects included in the sequence list OT.

[0084] Figure 9 An example of the operation of the model processor and resources according to an example embodiment is shown. More specifically, Figure 9 It shows Figure 8 Examples of modified embodiments. Figure 9Operations S210 and S220 can be combined with Figure 8 Operations S110 and S120 are basically the same, so they will not be described again.

[0085] In operation S225, task manager 120 can generate a sequence list OT indicating the data processing order of multiple candidate objects OC1 to OCn by using the generated metadata. In an example embodiment, as referenced... Figure 6 As described, Task Manager 120 can calculate scores indicating the importance of multiple candidate objects OC1 to OCn respectively using metadata and generate a sequence list OT using the calculated scores. In an example embodiment, as referenced... Figure 7 The task manager 120 can generate a sequence list OT using candidate objects other than those whose scores are less than a first threshold in the calculated scores. In operation S230, the task manager 120 can provide the generated sequence list OT and information Info_OC1 to Info_OCn related to the data processing of the multiple candidate objects OC1 to OCn to the resource 200.

[0086] Resource 200 can identify the order using the received sequence list OT, perform object identification for each candidate object according to the identified order, and execute the task corresponding to the object identification result. That is, in operation S235, resource 200 can perform object identification for the first candidate object in the first order, and in operation S240, execute the task corresponding to the object identification result of the first candidate object. In operation S245, resource 200 can perform object identification for the second candidate object with the next order, and in operation S250, execute the task corresponding to the object identification result of the second candidate object. In operations S255 and S260, resource 200 can repeatedly perform the aforementioned operations of object identification and execute the tasks corresponding to the object identification results for the candidate objects included in the sequence list OT. In operation S265, resource 200 can send a response indicating the completion of the tasks corresponding to the multiple candidate objects OC1 to OCn to task manager 120.

[0087] Although reference Figure 9 The description describes how task manager 120 sends information related to the data processing of a candidate object to resource 200. However, this disclosure is not limited to this, and task manager 120 may send information related to the data processing of a preset number of candidate objects (e.g., two or more candidate objects) to resource 200, which resource 200 may then perform object identification operations on the preset number of candidate objects and execute tasks corresponding to the object identification results by using the received information related to the data processing of the preset number of candidate objects.

[0088] Figure 10 This is a conceptual diagram illustrating a method for generating a sequence list according to an example embodiment. More specifically, Figure 10 It shows Figure 6 A conceptual diagram of an example of a modified embodiment.

[0089] refer to Figure 10 According to this disclosure, the task scheduler 122 can generate a score table ST for multiple candidate objects using a metadata table MT, and can identify candidate objects whose scores exceed a second threshold by using the score table ST. The second threshold can be based on scores that indicate a high demand for performing object recognition. Candidate objects whose scores exceed the second threshold are likely to correspond to objects of interest and can be an important criterion for controlling devices that have applied neural network systems.

[0090] When task scheduler 122 identifies that there are no candidate objects with scores exceeding the second threshold, task scheduler 122 can refer to Figure 6 The scores are arranged in descending order to generate the sequential list OT.

[0091] On the other hand, when the task scheduler 122 identifies a candidate object whose score exceeds the second threshold, the task scheduler 122 can immediately provide information related to the data processing of the identified candidate object to the resource 200. The resource 200 can then use the received information related to the data processing of the candidate object to perform object recognition for the identified candidate object and execute the task corresponding to the object recognition result.

[0092] For example, in Figure 10 In an example embodiment, the second threshold can be 90. Task scheduler 122 can identify candidate objects OC6 with scores exceeding the second threshold 90 in the score table ST, and immediately provide information Info_OC6 related to data processing of the identified candidate objects OC6 to resource 200. Resource 200 can perform object recognition on candidate objects OC6 using the received information Info_OC6, and execute tasks corresponding to the object recognition results.

[0093] After providing information related to candidate objects whose scores exceed the second threshold to resource 200, task scheduler 122 can generate a sequence list OT by using the scores of other candidate objects. For example, refer to Figure 10 The task scheduler 122 can generate the sequence list OT by using the scores of other candidate objects OC1 to OC5.

[0094] Task Manager 120 can sequentially provide information Info_OC1 to Info_OC5 related to data processing of multiple candidate objects OC1 to OC5 to resource 200 using the generated sequence list OT. Resource 200 can perform object recognition for other candidate objects OC1 to OC5 according to the order of the received information Info_OC1 to Info_OC5, and perform tasks corresponding to the object recognition results.

[0095] According to a modifiable example embodiment of this disclosure, the task scheduler 122 may provide the resource 200 with a sequence list OT and information Info_OC1 to Info_OC5 related to the data processing of other candidate objects OC1 to OC5. The resource 200 may then perform object recognition for the other candidate objects OC1 to OC5 using the received sequence list OT and execute tasks corresponding to the object recognition results.

[0096] When the neural network system according to the example embodiment identifies a candidate object that is determined to have a high demand for performing object recognition, object recognition can be performed immediately for the identified candidate object, and a task corresponding to the object recognition result can be performed. Therefore, the neural network system according to this disclosure can perform fine control on devices in which a neural network system is applied.

[0097] Figure 11 This is a flowchart illustrating an example of the operation of a model processor and resources according to an example embodiment. More specifically, Figure 11 It shows Figure 8 Examples of modified embodiments. Figure 11 The operation of S310 and S320 can be combined with Figure 8 Operations S110 and S120 are basically the same, so they will not be described again.

[0098] In operation S325, Task Manager 120 can use metadata to calculate a score indicating the importance of each of a plurality of candidate objects OC1 to OCn. (See reference...) Figure 10 As described, Task Manager 120 can identify candidate objects whose calculated scores exceed a second threshold. When a first candidate object whose score exceeds the second threshold is identified, in operation S330, information Info_OC1 related to data processing of the first candidate object can be provided to resource 200.

[0099] In operation S335, resource 200 can perform first object identification for the first candidate object using the received information Info_OC1 related to data processing of the first candidate object. In operation S340, resource 200 can perform a first task corresponding to the first object identification. In operation S350, resource 200 can send a response indicating the completion of the first task to task manager 120.

[0100] After Task Manager 120 provides Information Info_OC1 related to data processing of the first candidate object to Resource 200 in operation S330, in operation S345, Task Manager 120 can set the data processing order of other candidate objects by using the scores of candidate objects OC2 to OCn other than the first candidate object. When Task Manager 120 receives a response from Resource 200 in operation S350, in operation S355, Task Manager 120 can provide the generated sequence list OT and Information Info_OC2 to Info_OCn related to data processing of other candidate objects to Resource 200.

[0101] In operations S360, S365, S370, and S375, resource 200 can identify the order of each of the other candidate objects OC2 to OCn using the received sequence list OT, perform object identification for each candidate object according to the identified order, and execute the task corresponding to the object identification result. In operation S380, resource 200 can send a response indicating task completion to task manager 120.

[0102] In a modified example embodiment, the task manager 120 may sequentially provide information Info_OC2 to Info_OCn related to the data processing of other candidate objects OC2 to OCn to the resource 200 according to a set data processing order. The resource 200 may perform object recognition in the order of receiving information Info_OC2 to Info_OCn related to the data processing of other candidate objects OC2 to OCn, and perform a task corresponding to the object recognition result.

[0103] Figure 12 An example of the operation of the model processor and resources according to an example embodiment is shown. More specifically, Figure 12 It shows Figure 8 Examples of modified embodiments.

[0104] The candidate object detector 110 can detect candidate objects by analyzing the input image. According to the example embodiment, the candidate object detector 110 can be implemented to send each candidate object to the task manager 120 each time it is detected by analyzing the input image. Its detailed operation will be described below.

[0105] refer to Figure 12 In operation S410, the candidate object detector 110 can detect a first candidate object OC1 by analyzing the input image. In operation S415, the candidate object detector 110 can send the detected candidate object OC1 to the task manager 120.

[0106] In operation S420, task manager 120 can generate metadata for the received first candidate object OC1. In operation S425, task manager 120 can calculate a score indicating the importance of the first candidate object OC1 based on the generated metadata. Task manager 120 can determine whether the calculated score exceeds a first threshold. The first threshold may refer to the minimum score indicating the degree of need for performing object recognition. When the score of the first candidate object OC1 exceeds the first threshold, in operation S430, task manager 120 can send information Info_OC1 related to data processing of the first candidate object OC1 to resource 200.

[0107] In operation S435, resource 200 may perform first object recognition based on received information related to data processing of the first candidate object OC1. In operation S440, resource 200 may perform a first task corresponding to the result of the first object recognition.

[0108] The candidate object detector 110 can send the first candidate object OC1 to the task manager 120 and continuously analyze the image, thereby detecting a second candidate object OC2 as the next candidate object in operation S445. In operation S450, the candidate object detector 110 can send the detected candidate object OC2 to the task manager 120.

[0109] In operation S455, task manager 120 can generate metadata for the received second candidate object OC2. In operation S460, task manager 120 can calculate a score indicating the importance of the second candidate object OC2 based on the generated metadata. Task manager 120 can determine whether the calculated score exceeds a first threshold. When the score of the second candidate object OC2 exceeds the first threshold, in operation S465, task manager 120 can send information Info_OC2 related to data processing of the second candidate object OC2 to resource 200.

[0110] In operation S470, resource 200 may perform second object recognition based on received information related to data processing of the second candidate object OC2. In operation S475, resource 200 may perform a second task corresponding to the result of the second object recognition.

[0111] In operations S480, S485, S490, S495, S500, S505 and S510, the candidate object detector 110, the task manager 120 and the resource 200 can repeat the above operations for other candidate objects OC3 to OCn.

[0112] In the neural network system according to the example embodiment, the candidate object detector 110, the task manager 120, and the resources perform computations in parallel, thereby improving the data processing speed of the input image.

[0113] Figure 13 This is a block diagram of a neural network system according to an example embodiment. More specifically, Figure 13 It shows Figure 1 A block diagram of an example of a modified embodiment of the neural network system 10.

[0114] refer to Figure 13 The neural network system 20 may include a model processor 300 and resources 400. The model processor 300 may include a candidate object detector 310 and a task manager 320, and the resources 400 may include a first task module 410 and a second task module 420. The first task module 410 and the second task module 420 can perform object recognition and perform tasks corresponding to the object recognition results. For example, the first task module 410 and the second task module 420 can respectively use various types of neural network models, such as CNNs like GoogLeNet, AlexNet, ResNet, VGG networks, R-CNN, Fast R-CNN, RPN, RNN, S-DNN, S-SDNN, deconvolutional networks, DBN, RBM, fully convolutional networks, LSTM networks, GAN, IV3, classification networks, etc.

[0115] The model processor 300's operations of analyzing the input image IMG to detect multiple candidate objects, generating metadata for multiple candidate objects, and setting the data processing order of multiple candidate objects based on the metadata can be the same as or similar to those in the foregoing embodiments, and therefore will not be described again.

[0116] According to the example embodiment, the task manager 320 of the model processor 300 can assign multiple candidate objects to the first task module 410 or the second task module 420 using metadata.

[0117] In an example embodiment, task manager 320 can determine the appropriate module between the first task module 410 and the second task module 420 by using metadata related to data processing for each of the plurality of candidate targets (e.g., reducing computational load, increasing computational speed, or reducing power consumption when processing data). Task manager 320 can then match each of the plurality of candidate objects with either the first task module 410 or the second task module 420 based on the determination result.

[0118] For example, task manager 320 may use an IV3 model to assign a first candidate object with a smaller size to a first task module 410, and use a ResNet model to assign a second candidate object with a larger size to a second task module 420. This disclosure is not limited to the foregoing example, and task manager 320 may assign multiple candidate objects to the first task module 410 and the second task module 420 according to various schemes.

[0119] The model processor 300 can send information related to the data processing of candidate objects among multiple candidate objects to the module that assigned the candidate objects, according to the set order of the candidate objects. For example, refer to Figure 13 When the first candidate object OC1 is assigned to the first task module 410 and the second candidate object OC2 is assigned to the second task module 420, the model processor 300 can send information Info_OC1 related to the data processing of the first candidate object OC1 to the first task module 410 and information Info_OC2 related to the data processing of the second candidate object OC2 to the second task module 420.

[0120] The first task module 410 and the second task module 420 can respectively perform object recognition and perform tasks corresponding to the object recognition results by using the received information.

[0121] Resource 400 Figure 13 The resource 400 is illustrated and described as including a first task module 410 and a second task module 420, but the resource 400 may include three or more task modules.

[0122] Thus, the neural network system according to the example embodiment can perform efficient computational processing by matching multiple candidate objects with multiple task modules based on the metadata of multiple candidate objects.

[0123] Figure 14 This is a block diagram of a neural network system according to an example embodiment. More specifically, Figure 14 It shows Figure 1 A block diagram of an example of a modified embodiment of the neural network system 10.

[0124] refer to Figure 14The neural network system 30 may include a model processor 500, a first resource 600_1, and a second resource 600_2, and the model processor 500 may include a candidate object detector 510 and a task manager 520. The first resource 600_1 and the second resource 600_2 may each include computing resources capable of performing object recognition and executing tasks corresponding to the object recognition results. For example, the first resource 600_1 and the second resource 600_2 may each include various computing processing devices, such as CPUs, GPUs, APs, DSPs, FPGAs, NPUs, ECUs, ISPs, etc.

[0125] The model processor 500's operations of analyzing the input image IMG to detect multiple candidate objects, generating metadata for multiple candidate objects, and setting the data processing order of multiple candidate objects based on the metadata can be the same as or similar to those in the foregoing embodiments, and therefore will not be described again.

[0126] According to the example embodiment, the task manager 520 of the model processor 500 can assign multiple candidate objects to a first resource 600_1 or a second resource 600_2 by using the metadata of multiple candidate objects.

[0127] In an example embodiment, task manager 520 can determine the appropriate module between the first resource 600_1 and the second resource 600_2 (e.g., reducing computation, increasing computation speed, or reducing power consumption when performing a task) by using metadata for each of a plurality of candidate objects. Task manager 520 can match each of the plurality of candidate objects with the first resource 600_1 or the second resource 600_2 based on the determination result.

[0128] For example, Task Manager 520 may assign a first candidate object of size NxN, which is a multiple of 32, to a first resource 600_1, which is an NPU, and assign candidate objects of other sizes to a second resource 600_2, which is a CPU or GPU. This disclosure is not limited to the foregoing example, and Task Manager 520 may assign multiple candidate objects to the first resource 600_1 and the second resource 600_2 according to various schemes.

[0129] According to a modifiable embodiment of this disclosure, the task manager 520 can assign multiple candidate objects to the first resource 600_1 and the second resource 600_2 by further considering the operational states of the first resource 600_1 and the second resource 600_2, respectively. Information related to the operational state of the resources may include various types of information, such as information indicating whether the resource is performing data processing, information about the data processing target, and information related to the ongoing data processing of the resource.

[0130] In an example embodiment, task manager 520 may receive information related to the operation state of first resource 600_1 from first resource 600_1 and information related to the operation state of first resource 600_2 from second resource 600_2, and assign multiple candidate objects to first resource 600_1 and second resource 600_2 by further considering information related to the operation state and metadata of multiple candidate objects. For example, when task manager 520 determines based on information about the operation state that first resource 600_1 is in a standby state and second resource 600_2 is performing data processing, task manager 520 may assign candidate objects with the previous order to first resource 600_1. The method of assigning multiple candidate objects to multiple resources based on information about the operation state in the neural network system according to this disclosure is not limited to the above example.

[0131] The model processor 500 can send information related to the data processing of candidate objects among multiple candidate objects to the module that assigned the candidate objects, according to the set order of the candidate objects. For example, refer to Figure 14 When the first candidate object OC1 is assigned to the first resource 600_1 and the second candidate object OC2 is assigned to the second resource 600_2, the model processor 500 can send the information Info_OC1 related to the data processing of the first candidate object to the first resource 600_1 and the information Info_OC2 related to the data processing of the second candidate object to the second resource 600_2.

[0132] The first resource 600_1 and the second resource 600_2 can use the received information to perform object recognition on the first candidate object OC1 and the second candidate object OC2 respectively, and perform tasks corresponding to the object recognition results.

[0133] Neural network system 30 in Figure 14 The system is shown as including a first resource 600_1 and a second resource 600_2, but the neural network system 30 can be implemented to include three or more resources.

[0134] Thus, the neural network system according to the example embodiment can match multiple candidate objects with multiple resources based on metadata or based on metadata and the operational states of multiple resources. Therefore, the neural network system according to the example embodiment can reduce computational load, increase computational speed, and reduce power consumption, thereby performing efficient computational processing.

[0135] Figure 15 This is a flowchart illustrating an operation method of a neural network system according to an example embodiment. More specifically, the operation method of a neural network system according to a current embodiment of this disclosure may include... Figure 1 , Figure 2, Figure 13 and Figure 14 The neural network systems 10, 20, and 30 perform operations in a time sequence.

[0136] refer to Figure 15 In operation S610, the neural network system can detect multiple candidate objects included in the first image. Candidate objects can refer to regions of interest (ROIs) with a high probability of being included in the first image. In an example embodiment, the neural network system can detect multiple candidate objects included in the first image using a neural network model such as RPN.

[0137] A neural network system can perform first object recognition on a first candidate object that corresponds to a first processing order among multiple candidate objects. In an example embodiment, the neural network system can set the data processing order of multiple candidate objects using metadata of the multiple candidate objects, and perform first object recognition on the first candidate object selected based on the set data processing order.

[0138] More specifically, the neural network system can generate metadata for multiple candidate objects based on the first image. The metadata may include information such as the size of each of the multiple candidate objects, their position in the image IMG, their top-left coordinates, bottom-right coordinates, distance from preset coordinates or preset regions, and depth. This disclosure is not limited thereto, and the type of metadata may include additional information as well as the information described above.

[0139] A neural network system can calculate scores indicating the importance of multiple candidate objects based on generated metadata, and sort the calculated scores in descending order. Based on the sorting result, the neural network system can set the data processing order for the multiple candidate objects. The neural network system can then identify the first candidate object in the first order according to the set data processing order, and perform first object recognition on the identified first candidate object.

[0140] In operation S630, the neural network system can perform a first task corresponding to the result of the first object recognition. For example, when the neural network system is applied to an autonomous vehicle and the first candidate object is recognized as a car, the neural network system can perform the operation of calculating the distance between the recognized car and the autonomous vehicle as a task corresponding to the recognized car.

[0141] After completing the first task, the neural network system can perform second object recognition on the second candidate object. In an example embodiment, the neural network system can perform second object recognition on the second candidate object having a next order. In operation S650, the neural network system can perform a second task corresponding to the result of the second object recognition.

[0142] The neural network system can repeat the above object recognition operation for the remaining candidate objects and perform the operation corresponding to the object recognition result.

[0143] The operating method of the neural network system according to this disclosure can perform object recognition on an object-by-object basis, and perform tasks corresponding to the object recognition results on multiple candidate objects included in an image. Therefore, the operating method of the neural network system according to this disclosure can prevent bottlenecks caused by continuous object recognition operations, and prioritize data processing for candidate objects with high importance.

[0144] Figure 16 This is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

[0145] refer to Figure 16 The electronic device 1000 may include a processor 1010, random access memory (RAM) 1020, a model processor 1030, a memory 1040, a sensor 1050, and a resource 1060, and the components of the electronic device 1000 may be connected to communicate with each other via a bus 1070. The model processor 1030 may correspond to any of the model processors 100, 300, and 500 in the foregoing embodiments, and the resource 1060 may correspond to any of the resources 200, 400, 600_1, and 600_2 in the foregoing embodiments. In some example embodiments of this disclosure, references can be used... Figures 1 to 15 One or more example embodiments described herein implement model processor 1030 and resource 1060.

[0146] Electronic device 1000 can extract useful information by analyzing input data in real time based on neural networks, and determine the status or control components of electronic devices equipped with electronic device 1000 based on the extracted information. For example, electronic device 1000 can be applied to drones, robotic devices such as advanced driver assistance systems (ADAS), smart TVs (TV), smartphones, medical devices, mobile devices, image display devices, measuring devices, Internet of Things (IoT) devices, etc., and can be installed on at least one of various types of electronic devices.

[0147] Processor 1010 can control the overall operation of electronic device 1000. For example, processor 1010 can control the functions of model processor 1030 by executing a program stored in RAM 1020. RAM 1020 can temporarily store programs, data, applications, or instructions.

[0148] Sensor 1050 can sense or receive signals (e.g., image signals, voice signals, magnetic signals, biosignals, touch signals, etc.) from outside the electronic device 1000 and convert those signals into data. According to an example embodiment, sensor 1050 may include an image sensor that receives image signals corresponding to the surroundings of the electronic device 1000 and converts those image signals into data in image form. The electronic device 1000 may include multiple sensors 1050.

[0149] The model processor 1030 can perform neural network calculations by controlling resource 1060 and generate information signals based on the execution results. In the example, this can be implemented using software stored in system memory (e.g., read-only memory (ROM)) or the model processor 1030 can be operated based on the control of processor 1010. Memory 1040 is a storage device for storing data and can store various data generated by the computational operations of model processor 1030 and resource 1060.

[0150] According to an example embodiment, model processor 1030 can receive an image from sensor 1050 and detect multiple candidate objects included in the received image. Model processor 1030 can generate metadata for the detected multiple candidate objects and set a data processing order for the multiple candidate objects based on the generated metadata. Model processor 1030 can sequentially provide information related to the data processing of the multiple candidate objects OC1 to OCn to resource 1060 according to the set data processing order.

[0151] Resource 1060 may include computing resources that perform multiple computations based on a neural network, or communication resources implemented using various wired or wireless interfaces capable of communicating with external devices. According to an example embodiment, resource 1060 may sequentially perform data processing on multiple candidate objects in the order in which information relating to data processing of multiple candidate objects is received from model processor 1030. According to an example embodiment, resource 1060 may include multiple resources that are homogeneous or heterogeneous.

[0152] Therefore, the electronic device 1000 according to this disclosure prioritizes data processing for candidate objects identified as having high importance, thereby using object recognition technology to perform fine control.

[0153] Figure 17 This is a block diagram of an autonomous driving device according to an example embodiment.

[0154] refer to Figure 17The autonomous driving device 2000 may include a processor 2010, RAM 2020, a model processor 2030, a memory 2040, a sensor 2050, resources 2060, a driver 2070, and a communication interface 2080, and the components of the autonomous driving device 2000 may be connected to communicate with each other via a bus 2090. The model processor 2030 may correspond to the model processors 100, 300, 500, and 1030 of the foregoing embodiments, and the resources 2060 may correspond to the resources 200, 400, 600_1, 600_2, and 1060 of the foregoing embodiments. In some exemplary embodiments of this disclosure, references may be used. Figures 1 to 16 The example embodiments described implement the model processor 2030 and resource 2060.

[0155] The autonomous driving device 2000 can determine the situation and control the vehicle's driving by analyzing environmental data around the autonomous vehicle in real time based on neural networks.

[0156] Processor 2010 can control the overall operation of autonomous driving device 2000. For example, processor 2010 can control the functions of model processor 2030 by executing programs stored in RAM 2020. RAM 2020 can temporarily store programs, data, applications, or instructions.

[0157] Sensor 2050 may include multiple sensors that receive image signals about the surrounding environment of the autonomous driving device 2000 and output the image signals as an image. For example, sensor 2050 may include an image sensor 2051 such as a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS), a light detection and ranging (LiDAR) sensor 2052, a radio detection and ranging (radar) sensor 2053, a depth camera 2054, etc. This disclosure is not limited thereto, and sensor 2050 may further include an ultrasonic sensor (not shown), an infrared sensor (not shown), etc.

[0158] The model processor 2030 can perform neural network calculations by controlling resource 2060 and generate information signals based on the execution results. The memory 2040 is a storage device for storing data, and can store various data generated by the calculations of the model processor 2030 and resource 2060.

[0159] According to an example embodiment, model processor 2030 can receive images captured by sensor 2050 of the surrounding environment of autonomous driving device 2000, and detect multiple candidate objects included in the received images. Model processor 2030 can generate metadata of the detected multiple candidate objects, and set a data processing order for the multiple candidate objects based on the generated metadata. Model processor 2030 can sequentially provide information related to the data processing of the multiple candidate objects to resource 2060 according to the set data processing order.

[0160] Resource 2060 may include computing resources that perform multiple computations based on a neural network, or communication resources implemented using various wired or wireless interfaces capable of communicating with external devices. According to an example embodiment, resource 2060 may sequentially perform data processing on multiple candidate objects in the order in which information relating to data processing of multiple candidate objects is received from model processor 2030. According to an example embodiment, resource 2060 may include multiple resources that are homogeneous or heterogeneous.

[0161] As a component for driving the autonomous driving device 2000, the driver 2070 may include an engine and motor 2071, a steering unit 2072, and a braking unit 2073. In an example embodiment, the driver 2070, under the control of the processor 2010, can adjust the propulsion, braking, speed, direction, etc. of the autonomous driving device 2000 by using the engine and motor 2071, the steering unit 2072, and the braking unit 2073.

[0162] The communication interface 2080 can communicate with external devices using wired or wireless communication. For example, the communication interface 2080 can perform communication using a wired communication scheme such as Ethernet, or it can perform communication using a wireless communication scheme such as WiFi or Bluetooth.

[0163] Resource 2060 generates information based on the results of data processing performed on candidate objects, and processor 2010 can use the information generated by resource 2060 to generate control commands for controlling the autonomous driving device 2000. For example, resource 2060 can identify fire as an object included in an image output from sensor 2050 and generate information about the emergency service number "119" as a task corresponding to the fire. Therefore, processor 2010 can control communication interface 2080 to call the emergency service number "119". In another example, resource 2060 can identify fire and perform a task to change the driving route of the autonomous driving device 2000 as a task corresponding to the fire. Then, processor 2010 can control driver 2070 to drive according to the changed autonomous driving.

[0164] The methods of the exemplary embodiments described herein can be recorded in a non-transitory computer-readable medium including program instructions to implement various operations specifically implemented by a computer. The medium may also include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the medium may be program instructions specifically designed and constructed for the purposes of the specific implementation herein, or may be known to those skilled in the art. Examples of non-transitory computer-readable media include: magnetic media, such as hard disks, floppy disks, and magnetic tapes; optical media, such as disc compactors (CDs), read-only memory (ROMs), and digital universal disks (DVDs); magneto-optical media such as floppy disks; and hardware devices specifically configured to store and execute program instructions, such as ROMs, random access memory (RAMs), flash memory, etc. Examples of program instructions include machine code (e.g., generated by a compiler) and files containing higher-level code that can be executed by a computer using an interpreter. The above-described device configuration can be configured to act as one or more software modules to perform the operations of the above-described exemplary embodiments, and vice versa.

[0165] According to exemplary embodiments, at least one of the components, elements, modules, or units described herein may be embodied in various numbers of hardware, software, and / or firmware structures that perform the various functions described above. For example, at least one of these components, elements, or units may use direct circuit structures, such as memory, processor, logic circuits, lookup tables, etc., which may perform various functions under the control of one or more microprocessors or other control devices. Furthermore, at least one of these components, elements, or units may be implemented by a module, program, or portion of code containing one or more executable instructions for performing a specific logical function and executed by one or more microprocessors or other control devices. Additionally, at least one of these components, elements, or units may also include, or be implemented by, a processor such as a central processing unit (CPU), a microprocessor, etc., that performs the various functions. Two or more of these components, elements, or units may be combined into a single component, element, or unit that performs all the operations or functions of the combined two or more components, elements, or units. Furthermore, at least a portion of the function of at least one of these components, elements, or units may be performed by another of these components, elements, or units. Furthermore, although a bus is not shown in the block diagram, communication between components, elements, or units may be performed via a bus. The functional schemes of the above example embodiments can be implemented as algorithms executed on one or more processors. Furthermore, the components, elements, units, or processing steps represented by the blocks can utilize any number of related techniques for electronic configuration, signal processing and / or control, data processing, etc.

[0166] Although the present disclosure has been specifically shown and described with reference to exemplary embodiments thereof, it will be understood that various changes in form and detail may be made therein without departing from the spirit and scope of the appended claims.

Claims

1. A neural network system, comprising: The processor is configured to detect a plurality of candidate objects included in a first image, generate metadata corresponding to the plurality of candidate objects based on the first image, and set a data processing order for the plurality of candidate objects based on the metadata; as well as At least one resource is configured to perform data processing on the plurality of candidate objects. The processor is further configured to sequentially provide multiple pieces of information related to the data processing of the plurality of candidate objects to the at least one resource according to the set data processing order of the plurality of candidate objects. The at least one resource is further configured to sequentially perform data processing on the plurality of candidate objects according to the order in which information relating to data processing of each of the plurality of candidate objects is received. The at least one resource is further configured to: perform a first object recognition for a first candidate object in a first order, and perform a first task corresponding to the result of the first object recognition, the first task relating to the generation of control commands for controlling a device applying the neural network system, and The processor is further configured to: based on the completion of the first task, provide the at least one resource with multiple pieces of information relating to the data processing of a second candidate object in the next order of the first order among the plurality of candidate objects.

2. The neural network system of claim 1, wherein, The at least one resource is also configured to: Based on the received information related to the data processing of the second candidate object, a second object recognition is performed on the second candidate object, and a second task corresponding to the result of the second object recognition is performed.

3. The neural network system of claim 2, wherein, The processor is also configured to determine a plurality of regions of interest (RoIs) in the first image and detect the plurality of RoIs as the plurality of candidate objects.

4. The neural network system of claim 1, wherein, The processor is further configured to: obtain scores for the plurality of candidate objects based on the metadata according to one or more criteria, and set the data processing order of the plurality of candidate objects by using the obtained scores.

5. The neural network system of claim 4, wherein, The processor is also configured to set the data processing order of the plurality of candidate objects based on the size of their score values.

6. The neural network system of claim 4, wherein, The processor is also configured to: identify candidate objects with a score value greater than a first threshold, and immediately provide the at least one resource with information related to the data processing of the candidate objects.

7. The neural network system according to claim 4, wherein, The processor is also configured to: identify candidate objects with scores less than a second threshold, and not set the data processing order for the identified candidate objects.

8. The neural network system of claim 1, wherein, The metadata includes information of at least one of the following: the size of each of the plurality of candidate objects, its position in the first image, its top-left coordinate, bottom-right coordinate, distance from preset coordinates, or depth.

9. The neural network system of claim 1, wherein, The processor is also configured to further set the data processing order of the plurality of candidate objects based on the metadata of the plurality of candidate objects included in a second image which is a frame preceding the first image.

10. The neural network system of claim 1, wherein, The at least one resource includes multiple resources, and The processor is further configured to: match the plurality of candidate objects with the plurality of resources based on the metadata corresponding to the plurality of candidate objects, and provide the plurality of resources with multiple pieces of information related to the data processing of the plurality of candidate objects based on the matching results.

11. The neural network system of claim 10, wherein, The processor is further configured to: identify the operational status of the plurality of resources, and further match the plurality of candidate objects with the plurality of resources based on the identified operational status.

12. The neural network system of claim 1, wherein, The at least one resource includes multiple task processing modules, and The at least one resource is further configured to: match the multiple task processing modules with the multiple candidate objects based on the metadata corresponding to the multiple candidate objects, and provide the multiple task processing modules with multiple pieces of information related to the data processing of the multiple candidate objects based on the matching results.

13. A method for operating a neural network system, the method comprising: The detection includes multiple candidate objects in the first image; Perform first object identification on the first candidate object in the first order among the plurality of candidate objects; Perform a first task corresponding to the result of the first object identification, the first task being related to the operation of generating control commands for controlling a device applying the neural network system; After completing the first task, perform second object identification on the second candidate object in the next order of the first order among the plurality of candidate objects; as well as Perform a second task corresponding to the result of the second object identification.

14. The operating method according to claim 13, further comprising: Based on the first image, generate metadata corresponding to the plurality of candidate objects; Based on the metadata, the first candidate object is selected from the plurality of candidate objects; as well as Based on the metadata, the second candidate object is selected from the remaining candidate objects other than the first candidate object.

15. The method of operation of claim 14, wherein, Selecting the first candidate object based on the metadata includes: Based on the metadata, scores for the plurality of candidate objects are obtained according to one or more criteria; and The first candidate is selected based on the obtained score.

16. The method of operation of claim 15, wherein, Selecting the first candidate object based on the metadata includes: selecting candidate objects whose scores are greater than a threshold from the obtained scores as the first candidate object.

17. The method of operation of claim 15, wherein, Selecting the first candidate object based on the metadata includes: selecting the candidate object with the highest score value from the plurality of candidate objects as the first candidate object.

18. The method of operation of claim 15, wherein, Selecting the second candidate object includes: selecting the candidate object with the highest score value from the remaining candidate objects other than the first candidate object as the second candidate object.

19. An electronic device comprising: A sensor is configured to acquire data about the vicinity of the electronic device and output a first image based on the acquired data; At least one resource is configured to perform object recognition for the first image; The memory is configured to store a program. as well as The processor is configured to read the program and operate according to the program's instructions to detect a plurality of candidate objects in the first image, generate metadata of the plurality of candidate objects based on the first image, and provide the at least one resource with information related to a first candidate object selected from the plurality of candidate objects based on the metadata. The at least one resource is further configured to perform a first object identification for the first candidate object in a first order and to perform a first task corresponding to the result of the first object identification, the first task relating to the generation of control commands for controlling the electronic device, and The processor is further configured to: based on the completion of the first task, provide information to the at least one resource regarding a second candidate object in the next order of the first order among the plurality of candidate objects.

20. The electronic device according to claim 19, wherein, The at least one resource is further configured to: perform a second object identification on the second candidate object based on the received information about the second candidate object, and perform a second task corresponding to the result of the second object identification.

Images