Image analysis device

By allocating the processor according to the inference time and usage frequency of the neural network model in the image analysis system, the object detection and recognition resource requirements when multiple cameras input images are solved, and efficient object recognition and cost reduction are achieved.

CN111860478BActive Publication Date: 2024-11-19AWL INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010248807.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-04-26
Filing Date
2020-04-01
Publication Date
2024-11-19
Estimated Expiration
2040-04-01

AI Technical Summary

Technical Problem

In the prior art, when multiple cameras input images, object detection and recognition require a large number of computing resources, resulting in a long recognition time in frame images with many objects, which may lead to frame loss problems. The cost of allocating multiple neural network models is high, making it difficult to deal with. Different cameras have different types of object identification.

Method used

Design an image analysis device and system, use a plurality of cameras to connect the image analysis unit and the processor allocation unit to allocate appropriate inference processors according to the inference time and frequency of each neural network model, and then use the processor allocation unit Dynamic allocation of inference processors is performed to ensure that each neural network model has appropriate resource allocation, including image accumulation to handle inference tasks that are not allocated in real time.

Benefits of technology

It realizes efficient object recognition when multiple cameras input images, reduces the number of processors for inference, improves the efficiency and cost-effectiveness of the system, and can adapt to different types of object recognition input by different cameras.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111860478B_ABST
    Figure CN111860478B_ABST
Patent Text Reader

Abstract

In an image analysis device and an image analysis system, even when the type of object recognition for an image input from a certain camera is different from the type of object recognition for an image input from another camera, an appropriate inference processor is allocated for the inference processing of each neural network model corresponding to these object recognition and object detection. According to the inference time and usage frequency of each neural network model for object detection and the neural network model for object recognition contained in the artificial intelligence inference instance, the allocation of chips used for the inference processing of the above-mentioned neural network models among multiple chips (inference processors) is performed (S3, S5, S9, S13, S16 and S19). Thus, even when the type of object recognition for an image input from a certain camera is different from the type of object recognition for an image input from another camera, an appropriate chip is allocated for the inference processing of each neural network model corresponding to these object recognition and object detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an image analyzing device and an image analyzing system. Background Art

[0002] In the past, the following devices or systems are known: using a neural network for object detection, etc. to detect objects such as people captured in a frame image taken by a surveillance camera or other camera, and using a neural network for object recognition to recognize the detected objects (for example, refer to Patent Document 1).

[0003] Prior Art Literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2017-224925 Summary of the invention

[0006] (Problems to be solved by the invention)

[0007] However, in the above-mentioned device or system for detecting and identifying objects using a neural network (hereinafter referred to as "device for detecting and identifying objects"), both object detection and object recognition are processes that require a large amount of computer resources. In addition, the time required to recognize all objects in one frame image depends on the number of (detected) objects contained in the frame.

[0008] Therefore, it takes a long time to recognize objects in a frame image in which many objects are detected, and thus a so-called frame drop problem may occur, that is, objects in a frame image input some time after a frame image with many objects cannot be recognized (recognition omission occurs).

[0009] As an existing method to deal with the above-mentioned problem, there is the following method: the thread used for object detection processing and the thread used for object recognition processing can be separated and processed in parallel. In addition, in order to speed up the object recognition processing, a large number of inference processors such as GPU (Graphics Processing Unit) are allocated in the inference processing of the neural network for object recognition.

[0010] However, when the device or system for detecting and identifying the above-mentioned object performs object detection and object recognition on an input image from one camera, the type of object detection and object recognition performed on the input image is determined, so it can be dealt with by the above-mentioned existing method, but when the device or system for detecting and identifying the above-mentioned object performs object detection and object recognition on an input image from multiple cameras, there are few cases where the types of object detection and object recognition performed on the input images from each camera are all the same, so it cannot be dealt with by the above-mentioned method. More specifically, when the device or system for detecting and identifying the above-mentioned object performs object detection and object recognition on an input image from multiple cameras, and the type of object recognition for an image input from a certain camera is different from the type of object recognition for an image input from another camera, if a large number of GPUs are allocated to the neural network (inference processing) corresponding to all types of object recognition, the cost becomes too high. Therefore, it is necessary to consider the processing time (inference time) and usage frequency of each neural network model corresponding to multiple types of object recognition and object detection, and allocate appropriate inference processors for the inference processing of each of these neural network models according to each of the above-mentioned neural network models.

[0011] The present invention is used to solve the above-mentioned problems, and its purpose is to provide an image analysis device and an image analysis system, which can allocate appropriate inference processors to the inference processing of each of the neural network models corresponding to these object recognition and object detection even when the type of object recognition for an image input from a camera among multiple cameras is different from the type of object recognition for images input from other cameras.

[0012] (Technical solutions to solve problems)

[0013] In order to solve the above-mentioned problems, a first form of an image analysis device of the present invention is connected to multiple cameras and comprises: an image analysis unit, which uses each instance of an image analysis program including a learned object detection neural network model and one or more learned object recognition neural network models to analyze images input from each of the multiple cameras, wherein the learned object detection neural network model is used to detect objects captured in each image input from the camera, and the one or more learned object recognition neural network models are used to recognize objects detected by the learned object detection neural network model; multiple inference processors, which are used to perform inference processing in the learned object detection neural network model and the learned object recognition neural network model; and a processor allocation unit, which allocates inference processors used for inference processing in the learned object detection neural network model and inference processing in each of the learned object recognition neural network models according to the inference time and usage frequency required for the respective inference processing of the learned object detection neural network model and the learned object recognition neural network model contained in each instance of the image analysis program.

[0014] It may be that, in the image analysis device, the processor allocation unit estimates the number of inference processors required for the inference processing of each of the learned object recognition neural network models based on the inference time required for the inference processing in each of the learned object recognition neural network models and the respective usage frequencies of the learned object recognition neural network models.

[0015] It may be that, in the image analysis device, the processor allocation unit estimates the number of inference processors required for inference processing of the learned object detection neural network model based on the inference time required for inference processing in the learned object detection neural network model and the number of cameras as input sources of images of objects to be detected based on the learned object detection neural network model.

[0016] It may be that, in the image analysis device, the processor allocation unit estimates the number of inference processors required for the inference processing of each of the learned object recognition neural network models based on the inference time required for the inference processing in each of the learned object recognition neural network models, the usage frequency of each of the learned object recognition neural network models, and the target number of frames for inference processing performed by each of the learned object recognition neural network models within a given time.

[0017] It may be that, in the image analysis device, the processor allocation unit estimates the number of inference processors required for inference processing of the learned object detection neural network model based on the inference time required for inference processing in the learned object detection neural network model, the number of cameras as input sources of images of objects to be detected based on the learned object detection neural network model, and the target number of frames for inference processing by the learned object detection neural network model within a given time.

[0018] It may be that the image analysis device further includes an image accumulation unit, which accumulates images input from each of the cameras, and at a point in time, the processor allocation unit fails to allocate the inference processor for the inference processing of the learned object detection neural network model or the learned object recognition neural network model. Thereafter, after the processor allocation unit becomes able to allocate the inference processor for the inference processing of the corresponding learned object detection neural network model or the learned object recognition neural network model, the image analysis device performs the inference processing of the corresponding learned object detection neural network model or the learned object recognition neural network model in non-real time based on the past images accumulated in the image accumulation unit.

[0019] It may be that, in the image analysis device, the cameras connected to the image analysis device are classified into a plurality of camera groups, and the image analysis programs corresponding to the plurality of camera groups are constructed by using a combination of different neural network models for learning to detect objects and neural network models for learning to recognize objects.

[0020] The image analysis system of the second aspect of the present invention comprises: a plurality of image analysis devices; a plurality of cameras connected to the image analysis devices respectively; and a management server for managing the image analysis devices and the cameras including installation of the image analysis program in the image analysis devices. Each of the image analysis devices comprises: an image analysis unit, which uses each instance of an image analysis program including a learned object detection neural network model and one or more learned object recognition neural network models to analyze images input from each of the multiple cameras, wherein the learned object detection neural network model is used to detect objects captured in each image input from the camera, and the one or more learned object recognition neural network models are used to recognize objects detected by the learned object detection neural network model; multiple inference processors, which are used to perform inference processing in the learned object detection neural network model and the learned object recognition neural network model; and a processor allocation unit, which allocates inference processors used for inference processing in the learned object detection neural network model and inference processing in each of the learned object recognition neural network models according to the inference time and usage frequency required for the respective inference processing of the learned object detection neural network model and the learned object recognition neural network model contained in each instance of the image analysis program.

[0021] Preferably, in the image analysis system, cameras connected to the multiple image analysis devices are classified into multiple camera groups, and the image analysis programs corresponding to the multiple camera groups are constructed using a combination of different neural network models for learning object detection and neural network models for learning object recognition.

[0022] (Effects of the Invention)

[0023] According to the image analysis device of the first aspect of the present invention, inference processors used for inference processing in the learned object detection neural network model (included in the instance) and inference processing in each learned object recognition neural network model (included in the instance) are allocated among a plurality of inference processors according to the respective inference times and usage frequencies of the learned object detection neural network model and the learned object recognition neural network model contained in each instance of the image analysis program. Thus, even when the type of object recognition for an image input from a certain camera among a plurality of cameras is different from the type of object recognition for an image input from another camera, the processing time (inference time) and usage frequency of each neural network model corresponding to a plurality of object recognition and object detection can be considered, and appropriate inference processors can be allocated for the inference processing of each of the neural network models corresponding to each of these object recognition and object detection. Therefore, efficient object recognition can be performed for each of the images input from a plurality of cameras using a limited number of inference processors.

[0024] Furthermore, according to the image analysis system of the second aspect of the present invention, in addition to the above-mentioned effects, the image analysis device can be managed using the management server, including installation of the image analysis program to the image analysis device. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a block diagram showing a schematic configuration of an image analysis system including an analysis cartridge according to one embodiment of the present invention.

[0026] Figure 2 This is a block diagram showing a schematic hardware configuration of the analysis box.

[0027] Figure 3 This is a functional block diagram of the CPU in the same analysis box.

[0028] Figure 4 This is a diagram showing the main software components in the analysis kit.

[0029] Figure 5 This is an explanatory diagram of a method for calculating the number of chips required for inference processing of the NN model for identifying each object in the same analysis box.

[0030] Figure 6 This is a diagram illustrating the communication between the artificial intelligence (AI) reasoning instance in the same analysis box and the GPU server.

[0031] Figure 7 This is a flowchart of the allocation process of the NN model to the chip performed by the above-mentioned GPU server.

[0032] Figure 8 This diagram is an illustration of the camera group and application group in the image analysis system.

[0033] Fig. 9 It is an explanatory diagram showing an example of the above-mentioned application group.

[0034] Fig.10 yes Fig. 9 Illustration of the vectorization process performed by the vectorization model in .

[0035] Fig.11 This is a block diagram showing the connection between each camera and each analysis box in a store and a management server.

[0036] Fig.12 This is an explanatory diagram showing an example of a management unit of cameras connected to each analysis box in the image analysis system. DETAILED DESCRIPTION

[0037] Hereinafter, an image analysis device and an image analysis system according to embodiments embodying the present invention will be described with reference to the drawings. Figure 1 1 is a block diagram showing a schematic structure of an image analysis system 10 including an analysis box 1 as an image analysis device of the present embodiment. In the present embodiment, an example is described in which the analysis box 1 and a network camera (IP (Internet Protocol) camera) 2 connected to the analysis box 1 are arranged in a store S such as a chain store. Figure 1 As shown, the image analysis system 10 includes an analysis box 1, a plurality of network cameras 2 (hereinafter referred to as "cameras 2"), a hub 3 and a router 5, and an AI analysis server 6 and a management server 7 on the cloud C, which are arranged in a store S. The plurality of cameras 2 and the analysis box 1 are arranged in a plurality of stores, so the image analysis system 10 includes a plurality of analysis boxes 1 and a plurality of cameras 2 connected to each of the analysis boxes 1.

[0038] The network camera 2 has an IP address and can be directly connected to the network. Figure 1 As shown, the analysis box 1 is connected to a plurality of cameras 2 via a LAN (Local Area Network) 4 and a hub 3, and analyzes images input from each of these cameras 2. Specifically, object detection processing is performed on images input from each of the cameras 2, and object recognition processing is performed on images of objects detected by the object detection processing.

[0039] In addition, if Figure 1As shown, the analysis box 1 is connected to the AI ​​analysis server 6 and the management server 7 on the cloud C via the hub 3 and the router 5. The AI ​​analysis server 6 analyzes the behavior of people in each store based on the object recognition results from the analysis box 1, and converts the information of the analysis results into data that can be easily used by applications for various purposes such as marketing and anti-theft, and outputs it.

[0040] The management server 7 manages the plurality of analysis boxes 1 and the cameras 2 connected to the analysis boxes 1 arranged in each store. Specifically, the management server 7 controls the installation of the application package in the analysis boxes 1 of each store, the start and stop of the cameras 2 connected to the analysis boxes 1, etc. In addition, the application package is equivalent to the "image analysis program" in the technical solution. Fig. 9 The application 51 is an example of the application package. The application package will be described in detail later. The application package is a packaged program including a neural network model for learning object detection, one or more neural network models for learning object recognition, and a control script that records the use method (processing order) of these neural network models.

[0041] Next, refer to Figure 2 The hardware structure of the analysis box 1 is described below. The analysis box 1 includes: a CPU 11, which controls the entire device and performs various calculations; a hard disk 12, which stores various data and programs; a RAM (Random Access Memory) 13; an inference chip (hereinafter referred to as a "chip") 14a to 14h as a processor for DNN inference; and a communication control IC 15. The CPU 11 is a general-purpose CPU, or a CPU designed to improve the parallel processing performance for simultaneously processing multiple video streams. In addition, the data stored in the hard disk 12 includes video data (frame image data) obtained by decoding the video streams (data) input from each camera 2, and the programs stored in the hard disk 12, in addition to the above-mentioned application packages, also include Figure 4 The programs of VMS21, analysis box OS24 and GPU server 25 described in the description.

[0042] The above-mentioned (inference) chips 14a~14h are preferably processors (inference-dedicated chips) optimized for DNN (Deep Neural Networks) inference, but can also be general-purpose GPUs (Graphics Processing Units) or other processors for general purposes. In addition, each of the above-mentioned chips 14a~14h can be a device in which multiple chips (inference processors) are integrated (mounted) on a plug-in computer. In addition, multiple types of chips can be mounted on one analysis box 1. For example, four inference-dedicated chips made by company A, eight inference-dedicated chips made by company B, and one GPGPU made by company C can be mounted on one analysis box 1. Among them, in the description of the estimation processing of the number of chips required for the inference processing in the neural network model for learning to complete object recognition and the neural network model for learning to complete object detection described later, in order to simplify the description, an example of the case where multiple chips of the same type are mounted on one analysis box 1 is described.

[0043] like Figure 2 As shown, the (inference) chips 14a to 14h are connected to the CPU 11 via PCI Express or USB. In addition, some of the chips 14a to 14h may be connected to the CPU 11 via PCI Express, and the other chips may be connected to the CPU 11 via USB.

[0044] The communication control IC 15 includes a LAN port 16 which is a port for connection to a LAN conforming to Ethernet standards.

[0045] Figure 3 1 and 2 show the functional blocks of the CPU 11 in the above-mentioned analysis box 1. The analysis box 1 includes an image analysis unit 18 and a processor allocation unit 19 as functional blocks.

[0046] The image analysis unit 18 uses each instance of an application package including a learned object detection neural network model (hereinafter referred to as “object detection NN model”) for detecting an object captured in each image input from the camera 2 and one or more learned object recognition neural network models (hereinafter referred to as “object recognition NN models”) for recognizing an object detected by the object detection NN model. Figure 4The image analysis unit 18 is mainly executed by the CPU 11 of the analysis box 1. Figure 4 The functions of the processor allocation unit 19 are realized by the CPU 11 of the analysis box 1 executing Figure 4 The GPU server 25 (program) shown is implemented.

[0047] Next, refer to Figure 4 The main software components in the analysis box 1 are described below. Figure 4 As shown, the main software in the analysis box 1 is VMS (Video Management System) 21, AI reasoning instances 23a-23c, analysis box OS 24 and GPU server 25. Figure 4 , the processes of each software are shown. VMS 21 decodes the video stream input from cameras 2a to 2c via hub 3, stores the decoded frame image (data) in storage device 22, and outputs it to AI reasoning instances 23a to 23c via the bus in analysis box 1. VMS 21 and storage device 22 together constitute VMS server 20.

[0048] In addition, the above AI reasoning instances 23a to 23c are each the above application packages (equivalent to Fig. 9 The AI ​​reasoning instances 23a, 23b, and 23c perform object detection and object recognition for the images input from the cameras 2a, 2b, and 2c, respectively. The AI ​​reasoning instances for object recognition processing for the images input from the cameras 2a, 2b, and 2c are divided in this way because the type of application package (NN model for object detection and NN model for object recognition contained therein) to be used may be different for each of the cameras 2a, 2b, and 2c.

[0049] In addition, the analysis box OS 24 controls the applications of the AI ​​reasoning instances 23a to 23c and the like in the analysis box 1, and performs transmission and reception with the management server 7. In addition, the main processing performed by the GPU server 25 is as follows: when each AI reasoning instance 23a to 23c is started, the plurality of chips 14a to 14h (see Figure 2 ) among the chips used for the inference processing in the NN model for object detection and the inference processing in the NN model for object recognition. Among them, the usage frequency of the NN model for object detection and the NN model for object recognition contained in each AI inference instance 23a~23c can also change according to the time period, so the GPU server 25 can change the allocation of chips used for the inference processing in the NN model for object detection and the inference processing in the NN model for object recognition according to the time period.

[0050] Next, a basic policy for the case where the GPU server 25 distributes the NN model for object detection and the NN model for object recognition to each of the chips 14 a to 14 h will be described.

[0051] 1. Although the NN model (NN model for object detection or NN model for object recognition) assigned to each chip 14a to 14h can be replaced, the replacement of the NN model is time-consuming (cost), so it is best not to replace it. In addition, the time required for replacing the above NN model varies depending on the chip. In addition, generally speaking, there are cases where only one NN model is assigned to the (inference) chip, and there are also cases where multiple NN models can be assigned.

[0052] 2. Allocate each NN model to each chip 14a to 14h so as to appropriately respond to the analysis requirements (object recognition requirements) within the store. Specifically, the following 2-1 and 2-2 are assumed.

[0053] 2-1. At the frame rate at which all recognition objects can be recognized with the current chip configuration (type and number of chips), the NN model for object detection is assigned to the minimum chip that can detect objects for images from all cameras. The minimum chip refers to the minimum chip from the perspective of quantity and performance.

[0054] 2-2. Based on the inference time of recognition (classification, etc.) (the inference time required for the inference processing of the NN model for object recognition) and the necessity of the recognition (frequency of use and priority), each NN model for object recognition is assigned to an appropriate number of chips.

[0055] 2-3. When the above 2-1 and 2-2 are constantly changing, the time (cost) required for replacing the NN model in 1 above may be taken into consideration, and the NN model may be replaced (exchanged) to the minimum extent possible.

[0056] 3. For identification processing with a low priority, it can be postponed (not completed in real time). That is, it can be processed in an idle time period according to the information from the VMS server 20 (refer to Figure 4 )’s frame image (data) is used to perform the above-mentioned recognition processing.

[0057] The result of assigning each NN model to each chip 14a~14h in accordance with the above basic principles is that if the chips are assigned appropriately, the behavior of the GPU server 25 is very simple. It only inputs the frame image (data) obtained by decoding the video stream input from the cameras 2a~2c to the chip assigned to the NN model for object detection, detects the identification object, and flows all detected identification objects into the chip assigned to the NN model for object recognition.

[0058] Next, an example of how the GPU server 25 allocates NN models to chips is described. In this allocation example, the extent to which the recognition objects appear (are detected) as a result of object detection and the extent to which the object recognition NN model is applied to these recognition objects are calculated, and the number of chips allocated to the object recognition NN model is calculated by reverse calculation from these calculation results. The reason for performing such a chip allocation method is that the number of chips required for inference processing in each object recognition NN model when the frame rate of object detection is constant is different depending on the number of recognition objects as a result of object detection and the type of object recognition NN model applied to these recognition objects.

[0059] Regarding the above points, refer to Figure 5 For explanation. Figure 5 In the example, the object detection NN model 34 is an NN model that can detect both a person and a face. Figure 5As shown, when the object detection NN model 34 detects a person such as 1-1 and a person such as 1-2 from the first frame image 33, and detects a face such as 2-1 from the second frame image 33, the first object recognition NN model 36 for performing object recognition processing such as classification of persons is used twice for the two frame images, and the second object recognition NN model 37 for performing object recognition processing (gender, age estimation, etc.) for faces is used once. Here, the inference time required for the inference processing of object recognition in the first object recognition NN model 36 and the second object recognition NN model 37 are different, so the number of chips required for the inference processing of the first object recognition NN model 36 and the second object recognition NN model 37 are different according to the number of recognition objects as the object detection results and the inference time of the object recognition NN model applied to the recognition objects. That is, the number of chips required for the inference processing in each object recognition NN model when the frame rate of object detection is constant is different according to the number of recognition objects as the object detection results and the type of object recognition NN model applied to the recognition objects.

[0060] Among them, the number of identification objects as a result of object detection for each frame image 33 and the time for inference processing of each object identification NN model for these identification objects vary according to the time period. For example, the number of identification objects is different in the busy time period in the evening and the idle time period after noon, so the time required for object recognition processing for these identification objects is different. Therefore, in anticipation of this situation, when the analysis box 1 is started, it is necessary to allocate each NN model (for example, the NN model 34 for object detection, the NN model 36 for the first object recognition, and the NN model 37 for the second object recognition) to each chip so that it can be dealt with with the minimum number of NN model replacement (change) times (the number of changes in the allocation method of each NN model (NN model for object detection and NN model for each object recognition) to each chip).

[0061] The management server 7 (see Figure 1 ) The manager allocates each NN model required for the analysis (object recognition) to be performed to each chip in a simple process (by inputting the model performance, number of cameras, target performance, etc. used in the "needed chip number estimation" process described later). Specifically, when the manager inputs the model performance, number of cameras, target performance, etc. from the management server 7, the management server 7 sends the input information to the analysis box 1, and the CPU 11 of the analysis box 1 uses the GPU server 25 (refer to Figure 4 ) to assign the NN models required for analysis to each chip.

[0062] Next, an example of how the GPU server 25 allocates NN models to chips is described in detail. However, before that, refer to Figure 6 To illustrate the above AI reasoning example 23 ( Figure 4 The inter-process communication between the AI ​​reasoning instance 23 (the general term for the AI ​​reasoning instances 23a to 23c) and the GPU server 25. The inter-process communication between the AI ​​reasoning instance 23 (the process) and the GPU server 25 (the process) is performed using, for example, shared memory, sockets, named pipes (fifo), etc. Figure 6 In, an example of using fifo is shown. Figure 6 In the description, the processing after each NN model has been assigned to each chip 14a to 14h is described. Figure 6 In the inter-process communication in the AI ​​inference instance 23, the sending and receiving processing is precisely carried out through the script in the AI ​​inference instance 23 (equivalent to Fig. 9 The script 55 in the application 51 is used to perform the operation.

[0063] exist Figure 6 In the AI ​​reasoning example 23a, the object detection NN model of model ID1 and the object recognition NN model of model ID2 are included, and the object detection NN model of model ID1 and the object recognition NN model of model ID3 are included in the AI ​​reasoning example 23b. Figure 6 , illustrating the processing performed between the AI ​​reasoning instances 23a, 23b and the GPU server 25 after each NN model has been assigned to each chip 14a~14h.

[0064] First, if Figure 6 As shown, when the AI ​​reasoning instance 23a sends information such as the NN model to be executed is model ID1, and the frame (image) data used in the reasoning processing based on the NN model to the GPU server 25, the GPU server 25 incorporates (inputs) the received frame data into the reasoning data queue 42a in the first reasoning thread 41a corresponding to the NN model for object detection of model ID1. The reasoning data queue 42a outputs the input frame data to the chips 14a~14c assigned to the NN model for object detection of model ID1 in the order of input. Then, object detection is performed on the input frame (image) data by these reasoning chips 14a~14c. Thereafter, the first reasoning thread 41a of the GPU server 25 returns the result of the above-mentioned object detection to the AI ​​reasoning instance 23a.

[0065] When the AI ​​inference instance 23a receives the above-mentioned object detection result from the first inference thread 41a, it sends the information that the NN model to be executed is model ID2 together with the frame data of the identification object (for example, a person, a face) based on the received object detection result to the GPU server 25. The GPU server 25 incorporates (inputs) the received frame data into the inference data queue 42b in the second inference thread 41b corresponding to the object recognition NN model of model ID2. The inference data queue 42b outputs the input frame data of the identification object to the chip 14d assigned to the object recognition NN model of model ID2 in the order of input. Then, object recognition is performed on the input frame (image) data by these inference chips 14d. Thereafter, the second inference thread 41b of the GPU server 25 returns the result of the above-mentioned object recognition to the AI ​​inference instance 23a.

[0066] The processing performed by AI reasoning example 23b is basically the same as that performed by AI reasoning example 23a, but the NN model for object recognition processing performed on the frame data of the recognition object is different in that the NN model of model ID3 is used. Figure 6 As shown, in the GPU server 25, the threads for performing inference processing corresponding to each NN model are divided into individual threads (a first inference thread 41a, a second inference thread 41b, and a third inference thread 41c). In addition, these inference threads 41a~41c have respective inference data queues 42a~42c.

[0067] Next, refer to Figure 7 To explain the above reference in more detail Figure 5 In the following description, the GPU server 25 is described as performing the processing, but more precisely, Figure 3 The processor allocation unit 19 in the CPU 11 performs allocation processing of allocating NN models to chips. That is, the CPU 11 and the GPU server 25 (such software) cooperate to perform allocation processing of allocating NN models to chips.

[0068] Figure 7 2 is a flowchart of the allocation process of the NN model to the chip performed by the GPU server 25 (to be precise, the processor allocation unit 19). This allocation process is executed as a batch process based on a timer at a given time (for example, once an hour). Figure 7 Before the processing shown in the flowchart, the GPU server 25 obtains information on the model ID, model path, model performance (value), priority of the NN model itself, and priority of the AI ​​reasoning instance 23 itself for all NN models contained in all AI reasoning instances 23 that may be used. Here, the above-mentioned model path refers to the hard disk 12 (refer to Figure 2 ) in the corresponding NN model (showing the path to the location where the target file is stored). In addition, the above priority is based on (in application 51 (refer to Fig. 9 ) is registered by the administrator when the NN model is registered) The priority of the NN model itself × the priority of the AI ​​reasoning instance 23 itself (the priority of the AI ​​reasoning instance 23 corresponding to the camera is calculated based on the criterion that the camera that does not capture many people has a lower priority).

[0069] Then, the GPU server 25 performs the following processes (1) to (3).

[0070] (1) Based on the information obtained above (model ID, model path, model performance (value), priority of the NN model itself, and information on the priority of the AI ​​reasoning instance 23 itself), a parameter group of (model ID, model path, model performance (value), priority) is generated for all NN models contained in all AI reasoning instances 23, thereby obtaining a list of parameter groups, and creating a list L = (l1, l2, ..., lN) by rearranging the list in order of priority.

[0071] (2) Reset (release) the allocation of the NN model to the chip at that point in time.

[0072] (3) For each element li in the above list L, execute in order from the first element (from the element with the highest priority) Figure 7 After S1, the NN model is allocated to the chip.

[0073] Here, when the priority is a negative value (= when a negative value is set for the priority of the NN model itself), it is allowed not to assign the NN model to a chip in the following processing. Furthermore, when the NN model is not assigned to a chip, the reasoning processing based on the NN model is postponed as described in paragraph (0036) using the frame images accumulated in the VMS server 20.

[0074] In addition, the priority of the above-mentioned AI reasoning instance 23 is calculated based on the past reasoning performance of the AI ​​reasoning instance 23 (all NN models contained therein). For example, the priority of the AI ​​reasoning instance 23 can be calculated based on the cumulative value of the reasoning time of the AI ​​reasoning instance 23 at the previous time, or the reasoning time of the AI ​​reasoning instance 23 at the next time can be predicted and prioritized based on the cumulative value of the reasoning time of the AI ​​reasoning instance 23 on each day of the week / time period in the past month or so.

[0075] If the chip allocation process for a certain element li (corresponding NN model) in the list L of (3) above is started ("Yes" in S1), the GPU server 25 determines whether the chip has been allocated to the corresponding NN model. Specifically, the GPU server 25 confirms whether the model ID (contained in the parameter group) of the above element li is stored (exists) in the model ID array (array in the program of the GPU server 25) managed by the self-process, thereby confirming whether the chip has been allocated to the NN model corresponding to the model ID (S2). If the result is that the above model ID has not been stored in the above model ID array ("No" in S2), the number of chips required for the inference processing of the NN model corresponding to the model ID is estimated (S3). The details of the required chip number estimation process will be described later.

[0076] Next, in Figure 2 When there are (NN model) chips that have not been allocated among the (inference) chips 14a to 14h shown ("Yes" in S4), the GPU server 25 selects the number of chips estimated as required in S3 from the unallocated chips, and allocates these chips to the NN models corresponding to the model IDs that have not been stored in S2 (S5). Then, the GPU server 25 adds (stores) the model IDs that have not been stored in S2 to the model ID array (S6).

[0077] As a result of the chip allocation process in S5, the NN model to which the chip is allocated through the allocation process can be used in the analysis box 1 (becomes in a "True" state) (S7). Figure 7 In the above, "True" means that at least one chip is allocated to the corresponding NN model, which means that the NN model can be used. In contrast, "False" means that no chip is allocated to the corresponding NN model, which means that the NN model cannot be used. In the case of "False", the reasoning processing based on the NN model will be postponed as described in paragraph (0036).

[0078] In the determination process of S4, if there are no unallocated chips left (No in S4), the GPU server 25 determines whether the priority of the element li is greater than 0 and whether it can be allocated from other inference threads (equivalent to Figure 6The first inference thread 41a~41c) calls the chip (S8). When the priority of the above-mentioned element li is greater than 0 and the chip can be called from other inference threads ("Yes" in S8), the GPU server 25 calls the chip from other inference threads and assigns the called chip to the NN model corresponding to the model ID determined as not yet stored in the above-mentioned S2 (S9). Then, the GPU server 25 appends the model ID determined as not yet stored in the above-mentioned S2 to the above-mentioned model ID array (S10). In addition, the specific content of the processing of calling the chip from the above-mentioned other inference threads will be described later.

[0079] As a result of the process of calling the chip assigned to the above S9, the NN model to which the chip is assigned through the assignment process can be used in the analysis box 1 (becoming a "true" state) (S11). Among them, when the result of the judgment of the above S8 is that the priority of the above element li is less than 0 (negative), or the chip cannot be called from other reasoning threads ("No" in S8), the model ID determined to be not yet stored in the above S2 cannot be used in the analysis box 1 (becoming a "false" state) (S12). In addition, when it is determined in the judgment of the above S8 that the priority of the above element li is less than 0 (negative), no attempt is made to call the chip from the above other reasoning threads, and a "false" value is returned (becoming a "false" state). In the case of becoming this "false" state, there are no unassigned chips left, and the priority of the subsequent element li is only negative, or it becomes a state that the chip cannot be called from other reasoning threads (the number of NN models to which the chip has been assigned is the same as the number of chips). Therefore, the chip assignment process for each element in the list L is terminated here.

[0080] In the determination process of S2, if the chip has been allocated to the NN model of the corresponding element li (the model ID of the element li is already stored in the model ID array) ("Yes" in S2), the number of NN models allocated to the chip will not increase, but the instances of the corresponding NN model will increase. Therefore, the GPU server 25 performs the same estimation process of the number of required chips as in S3 for the corresponding NN model (S13).

[0081] In the estimation process of S13, the number of instances of the corresponding NN model increases compared to when the estimation process of the number of required chips for the corresponding NN model was initially performed (estimation process of S3), so the number of cameras as the basis for estimating the number of required chips for the NN model for object detection and the NN model for object recognition, and the average number of people captured in each camera as the basis for estimating the number of required chips for the NN model for object recognition (average number of objects to be recognized) will increase. Therefore, the GPU server 25 observes the number of required chips obtained in the estimation process of S13 and determines whether it is necessary to add chips to be allocated to the corresponding NN model (S14).

[0082] If the result of the determination in S14 is that it is necessary to add chips to the corresponding NN model ("Yes" in S14), and there are unallocated chips (to the NN model) remaining ("Yes" in S15), the GPU server 25 selects the required number of chips to be added from the unallocated chips, and allocates these chips to the NN model determined to need to be added in S14 (S16). As a result, the corresponding NN model is in a state where two or more chips are allocated, and the corresponding NN model naturally becomes in the "true" state (S17).

[0083] In contrast, when the result of the determination in S15 is that there are no unallocated chips left ("No" in S15), the GPU server 25 determines whether the priority of the above-mentioned element li is greater than 0 and whether the chip can be called from other reasoning threads (S18). When the priority of the above-mentioned element li is greater than 0 and the chip can be called from other reasoning threads ("Yes" in S18), the GPU server 25 calls the chip from other reasoning threads and allocates the called chip to the NN model determined to be required to be added in the above-mentioned S14 (S19). In addition, when the thread corresponding to the NN model determined to be required to be added in the above-mentioned S14 will become the thread with the lowest recognition rate described later if the chip is not called from other reasoning threads, the GPU server 25 performs the additional allocation processing of S19 (chip calling processing of calling the chip from other reasoning threads), but when the chip will not become the thread with the lowest recognition rate even if it is not called from other reasoning threads, the GPU server 25 does not have to perform the calling processing of calling the chip from other reasoning threads.

[0084] When the additional allocation processing of S19 is performed, the corresponding NN model is in a state where two or more chips are allocated to the corresponding NN model, so the corresponding NN model naturally becomes the above-mentioned "true" state (S20). In addition, when the result of the judgment of S18 is that the priority of the above-mentioned element li is less than 0 (negative), or when the chip cannot be called from other inference threads ("No" in S18), additional chip allocation cannot be performed to the corresponding NN model. However, even in this case, according to the judgment result of S2 (the judgment result of "the chip has been allocated to the corresponding NN model"), the chip should be allocated to the NN model corresponding to the corresponding model ID, so the corresponding NN model naturally becomes the above-mentioned "true" state (S21).

[0085] In the above description, after all chip allocation processing for NN models with a priority of 0 or higher is completed, as long as there are still unallocated chips ("Yes" in S4 and "Yes" in S15), chips are allocated even to NN models with a negative priority. However, the present invention is not limited to this. Even if there are no unallocated chips, if there is a thread of an NN model that exceeds the target performance and has spare capacity as described later, the chip call allocated to the NN model may be allocated to the NN model with a negative priority.

[0086] When the result of the judgment of the above S14 is that there is no need to add chips to be allocated to the corresponding NN model ("No" in S14), that is, even if the instances of the corresponding NN model increase, the current number of allocated chips is sufficient, the corresponding NN model will naturally become the above-mentioned "true" state (S22).

[0087] Next, the details of the required number of chips estimation process of the above-mentioned S3 and S13 are explained. The content of the required number of chips estimation process differs depending on whether the NN model as the object is an NN model for object detection or an NN model for object recognition. Among them, regardless of the type of NN model, the GPU server 25 (processor allocation unit 19) estimates the number of inference processors required for the inference processing of each NN model based on the inference time and usage frequency required for the inference processing of each NN model. In addition, in the following description of the required number of chips estimation process, in order to simplify the description, an example of a case where one analysis box 1 is equipped with multiple chips of the same type is described.

[0088] First, the estimation process of the number of chips required for the NN model for object detection is described. In this case, the GPU server 25 (processor allocation unit 19) estimates the number of chips (necessary performance) required for the inference processing of the NN model for object detection based on the number K of cameras that are the input source of the image of the object to be detected based on the NN model for object detection, the model performance T of the NN model for object detection (the inference time (seconds) required for the inference processing of the NN model for object detection), and the target performance F of the NN model for object detection (the target number of frames (FPS (frame per second)) for the inference processing of the NN model for object detection within a given time (1 second)) by the following formula.

[0089] Required performance (number of chips) = K*F*T

[0090] For example, if the number of cameras K = 3 (cameras), the model performance T = 0.05 (seconds), and the target performance F = 6 (FPS), the required performance of the NN model for object detection (the number of chips required for inference processing) is calculated by the following formula.

[0091] Required performance (number of chips) = 3*6*0.05 = 0.9

[0092] Therefore, in the case of the above example, one chip is required. The above target performance F is required as a reference value for estimating the number of required chips. In addition, the target performance F is also required when comparing the performance and resource abundance between other threads (corresponding to the NN model).

[0093] Next, the estimation process of the number of chips required for the NN model for object recognition is described. In this case, the GPU server 25 (processor allocation unit 19) estimates the number of chips required for the inference processing of the NN model for object recognition (necessary performance) based on the average number of people N1, N2, ... captured in each camera as the input source of the image as the object recognition object based on the NN model for object recognition (that is, the frequency of use of the NN model for object recognition), the model performance T of the NN model for object recognition (the inference time (seconds) required for the inference processing of the NN model for object recognition), and the target performance F of the NN model for object recognition (the target number of frames (FPS) for the NN model for object recognition to perform inference processing within a given time), using the following formula.

[0094] Required performance (number of chips) = sum (N1, N2, ...) * F * T

[0095] (Wherein, sum(N1, N2, ...) represents the sum (total) of N1, N2, ...)

[0096] For example, if there are three cameras as the input source of the image to the NN model for object recognition, the average number of people captured by each camera is 5, 2, and 3, the model performance T = 0.03 seconds, and the target performance F = 6FPS, then the required performance of the NN model for object recognition (the number of chips required for inference processing) is calculated by the following formula.

[0097] Required performance (number of chips) = (5 + 2 + 3) * 6 * 0.03 = 1.8

[0098] Therefore, in the case of the above example, two chips are required. As in the case of the NN model for object detection, the target performance F is required as a reference value for estimating the number of required chips. In addition, the target performance F is also required when comparing the performance and resource abundance between threads (corresponding to the NN model) of other threads.

[0099] Next, the calling process of calling a chip from other reasoning threads described in the above S8, S9, S18 and S19 will be described in more detail. Before describing the specific chip calling process, the basic principle when calling a chip from other reasoning threads will be described. In the above S4 and S15, there are no unallocated chips left. When calling a chip from other reasoning threads (corresponding NN models), the allocation of multiple chips should not be changed at one time, but only the allocation of one chip should be changed. This is because, in the case of changing the allocation of multiple chips at one time, the possibility of increasing or decreasing the number of chips allocated to the NN model (thread) for object detection becomes higher to a certain extent, and in the case of increasing or decreasing the number of chips allocated to the NN model (thread) for object detection, the amount of data (mainly frame data of the object to be recognized) flowing into the NN model (thread) for object recognition that performs object recognition processing after object detection processing will increase or decrease significantly, so it is necessary to estimate (re-estimate) the number of chips required for each NN model (thread).

[0100] Next, the specific process of calling the chip processing from other inference threads is described.

[0101] 1. First, list the threads of the NN model that are assigned to multiple chips.

[0102] 2. If there are threads with a data loss rate of 0 described later among the threads listed in 1 above, re-list these threads. On the other hand, if there are no threads with a data loss rate of 0 among the threads listed in 1 above, the threads listed in 1 above are first sorted in ascending order of priority. If there are multiple threads of the NN model with the same priority, they are further sorted in descending order of recognition rate described later (recognition rate descending order).

[0103] 3. In the case where there is a thread with a data loss rate of 0 in the above 2, one chip is released from the top (first) thread among the listed threads with a data loss rate of 0. In addition, to be precise, in the case where there are multiple threads with a data loss rate of 0, it is preferred to release one chip from the thread with the lowest priority (using the NN model). In addition, in the above 2, in the case where there is no thread with a data loss rate of 0 and the threads are sorted in descending order of recognition rate, one chip is released from the top (first) thread.

[0104] 4. Allocate the chips released in step 3 above to the threads (of the NN model) that need the chips (the threads of the NN model that are not allocated chips or the threads with the lowest recognition rate).

[0105] Next, the data loss rate is described, which refers to the ratio of data that is discarded due to failure to detect or recognize, among the data that flows into each thread corresponding to the object detection NN model and the object recognition NN model.

[0106] In addition, when the recognition rate R is represented by the target performance Fa (FPS) and the actual performance Fr (FPS), it is as follows.

[0107] R=Fr / Fa

[0108] In the above formula, the actual performance Fr represents the measured value of the number of data (number of frames) inferred and processed by the corresponding (NN model) thread within a given time (1 second). The actual performance Fr is only meaningful when the above data loss rate is greater than 0. This is because, when the data loss rate = 0, the number of data (number of frames) that can be inferred by the corresponding thread within a given time (1 second) is greater than the above measured value (i.e., the actual performance Fr). In addition, the target performance Fa in the above formula is essentially the same as the target performance F of the above-mentioned NN model for object detection and the NN model for object recognition, and represents the target number of frames (FPS) of inference processing performed by the thread corresponding to each NN model within a given time (1 second).

[0109] Next, refer to Figure 8 as well as Fig. 9 The application group (hereinafter referred to as "application group") in the above-mentioned image analysis system 10 is described. Figure 8 As shown, the cameras 2 (cameras 2a to 2d, etc.) connected to the analysis boxes 1 in the image analysis system 10 are classified into a plurality of camera groups (a first camera group, a second camera group, etc.). The applications 51 corresponding to each of these camera groups (see Fig. 9 : The "image analysis program" in the technical proposal is composed of a combination of different NN models (NN model for object detection and NN model for object recognition). Figure 8Each application group 24a, 24b in the example is composed of instances (AI reasoning instances) of the same type of application 51. That is, the AI ​​reasoning instances 23a, 23b belonging to the first application group 24a (two AI reasoning instances that process images from the camera 2a at the entrance and the camera 2b at the exit belonging to the first camera group) are instances of the same type of application 51, and each AI reasoning instance belonging to the second application group 24b (AI reasoning instances 23c, 23d, etc. that process images from cameras 2c, 2d, etc. disposed at various locations in the passages within the store) are instances of the same type of application 51.

[0110] Fig. 9 An example of the above-mentioned application group is shown. Fig. 9 The entrance and exit face recognition application group 50 is as follows Figure 8 As shown in the first application group 24a in FIG. 1 , there is a group of AI reasoning instances 23 that process images from the camera 2a located at the entrance of each store and the camera 2b located at the exit. Fig. 9 The application 51 is an application of a class equivalent to the above-mentioned AI reasoning instances 23a and 23b, and is a packaged program including a person and face detection model 52, a vectorization model 53, a face recognition model 54, and a script 55. The person and face detection model 52 is an object detection NN model that detects people and faces contained in images acquired from the camera 2a located at the entrance of each store and the camera 2b located at the exit. The vectorization model 53 is an object recognition NN model that vectorizes the image of the person detected by the person and face detection model 52. The face recognition model 54 is an object recognition NN model that estimates the gender and age of the detected person based on the face detected by the person and face detection model 52. The script 55 is a simple control program that describes the processing procedures of the above-mentioned person and face detection model 52, the vectorization model 53, and the face recognition model 54.

[0111] In addition, in the above Figure 7 The registration of the priority of each NN model itself used in the allocation process of each NN model to the chip described in the above is performed as follows. That is, if the administrator performs the above application 51 (refer to Fig. 9 ) contains the input of the priority of each NN model 52 to 54 (itself), the management server 7 registers the priority of each of the input NN models. Fig.11 When the management server 7 described in is downloading each application to the analysis box 1 , the information of the priority of each of the NN models is transmitted from the management server 7 to the analysis box 1 .

[0112] On the other hand, it is equivalent to Figure 8An application 51 of a class of AI reasoning instances 23c, 23d, etc. (AI reasoning instances that process images from cameras 2c, 2d, etc. disposed at various locations in the aisles of the store) of the second application group 24b is, for example, a packaged program that includes an object detection NN model for detecting people, an object recognition NN model for identifying a person holding a shopping basket based on an image of the person detected by the object detection NN model, an object recognition NN model for identifying a store clerk in charge of a store based on an image of the person detected by the object detection NN model, and a script 55.

[0113] The GPU server 25 (processor allocation unit 19) performs the above Figure 7 The processing of each element li of the list L described in the description is performed, thereby allocating chips 14a, 14b, etc. to the NN model for object detection and the NN model for object recognition, respectively, so that object detection and object recognition can be performed with minimal data loss (data that is discarded without recognition). The GPU server 25 (processor allocation unit 19) allocates chips 14a, 14b, etc. to each NN model so that high-priority object recognition processing can be performed to the maximum extent possible while keeping the frame rate of object detection constant (without reducing the frame rate of object detection). Here, as Fig. 9 As shown, CPU 11 uses VMS server 20 (refer to Figure 4 ) is used to store the frame images, and then perform object recognition processing that cannot be processed in real time. In addition, as mentioned above, the frame rate of object detection is kept constant in order to maintain the accuracy of the processing (analysis, etc.) performed by the AI ​​analysis server 6 using the object recognition results of the analysis box 1.

[0114] Here, refer to Fig.10 To illustrate the above Fig. 9 Vectorization processing is performed by the vectorization model 53 in . Figure 3 The image analysis unit 18 of the CPU 11 shown uses the vectorization model 53 (see Fig. 9 ) to perform the following vectorization processing. That is, the image analysis unit 18 is as follows Fig.10 As shown in FIG. 1 , the image 57a, 57b of the person detected by the person and face detection model 52 from the frame images 33a, 33b taken by the cameras 2a, 2b is input to the vectorization model 53 (DNN model), so that the vectorization model 53 is as follows: Fig.10 As shown, the frame images 33a and 33b are converted into vectors V1 and V3 and output. These vectors V1 and V3 are, for example, 128-dimensional vectors.

[0115] If the vector V1 obtained from the image 57a of the person detected in the frame image 33a taken by the camera 2a and the vector V3 obtained from the image 57b of the person detected in the frame image 33b taken by the camera 2b are input to the Dist function, then when the person 58a captured in the image 57a is the same as the person 58b captured in the image 57b, the output value of the Dist function is less than a given value (for example, 15). Fig.10 In the example shown, the output value (32) of the Dist function is greater than the given value (15), so the person 58a captured in the image 57a and the person 58b captured in the image 57b are different persons. Here, the above-mentioned Dist function is a function for obtaining the distance between input vectors.

[0116] Therefore, the image analysis unit 18 of the CPU 11 detects at least one image of a person (for example, Fig.10 The above-mentioned vectorization processing is performed on the images of the person in camera 2a and camera 2b, so that it can be determined whether the person photographed by camera 2a is the same person as the person photographed by camera 2b.

[0117] Fig.11 2 is a diagram showing the connection between the management server 7, each analysis box 1 in the store, and each camera 2. Fig.11 In the figure, only the connection between the management server 7 and each analysis box 1a, 1b and each camera 2a to 2f in a certain store S is shown. In addition, in order to make the correspondence between each application 51a to 51c in the analysis box 1a, 1b and each camera 2a to 2f easy to understand, each application 51a to 51c and each camera 2a to 2f are connected with a dotted line and a solid line. When connecting the management server 7, each analysis box 1a, 1b and each camera 2a to 2f, first, it is necessary to register the analysis box 1a, 1b to be connected in the management server 7, and on this basis, register each camera 2a to 2f in the store S to any one of the analysis boxes 1a, 1b. Then, when registering each of the above-mentioned cameras 2a to 2f to each of the analysis boxes 1a, 1b, a process is performed to establish a correspondence between each camera 2a to 2f and each application 51 (the first application 51a, the second application 52b and the third application 52c) (that is, each camera and the above-mentioned application group (refer to Figure 8 as well as Fig. 9 )'s correspondence relationship)'s application selection processing and other setting processing.

[0118] At this time, if the application 51 selected for each camera 2a to 2f is an application already installed in the analysis box 1 (analysis box 1a or analysis box 1b), only the association between the corresponding camera 2 and the selected application 51 is established. On the other hand, if the selected application 51 is a new application 51 that has not been installed in the corresponding analysis box 1, the analysis box 1 (CPU 11 thereof) downloads the application 51 from the management server 7 and installs it, and then the association between the application 51 and the corresponding camera 2 is established.

[0119] In addition, Fig.11 In the example, the first application 51a of the analysis box 1a is associated with the camera 2a, and the second application 51b of the analysis box 1a is associated with the camera 2b. In addition, the first application 51a of the analysis box 1b is associated with the cameras 2c and 2d, and the third application 51c of the analysis box 1b is associated with the cameras 2e and 2f.

[0120] The above applications 51a to 51c and Figure 8 In addition, the cameras 2a to 2f connected to the analysis boxes 1a and 1b are classified into three camera groups corresponding to the above-mentioned applications 51a to 51c. Specifically, Fig.11 The cameras 2a to 2f shown are classified into three camera groups: cameras 2a, 2c, and 2d associated with a first application 51a, camera 2b associated with a second application 51b, and cameras 2e and 2f associated with a third application 51c.

[0121] Although in the above Fig.11 In the figure, the correspondence between each camera 2a~2f and each application 51a~51c in a store S is explained, but the cameras connected to all the analysis boxes 1 in the image analysis system 10 are classified into camera groups corresponding to each application 51, and the applications 51 corresponding to these camera groups are constructed using a combination of different NN models for object detection and NN models for object recognition.

[0122] In addition, although in the above Figure 8 , Fig. 9 as well as Fig.11 In the descriptions of FIG. 1 and FIG. 2 , an example is shown in which the management server 7 manages the cameras 2 in the image analysis system 10 in units of applications 51 . However, the units in which the management server 7 manages the cameras 2 are not limited to this. Fig.12 The example of the unit in which the management server 7 manages the cameras 2 connected to the analysis boxes 1 in the image analysis system 10 is shown. The management server 7 can manage the cameras 2 connected to the analysis boxes 1 in the image analysis system 10 in the unit of area (A area group 61a, B area group 61b), in the store unit (A store group 62a to D store group 62d), in the application unit (entrance and exit face recognition application group 63a (equivalent to Fig. 9The cameras 2 within the image analysis system 10 are managed in units (groups) such as the entrance and exit face recognition application group 50), or the theft detection application group 63b). Fig.12 Each group in corresponds to the above-mentioned camera group. When the administrator changes the unit of management of the camera 2, the management server 7 performs a switching operation in groups.

[0123] Although the management server 7 can manage each camera 2 as a group or individually, it is preferred to manage the cameras 2 as a group in consideration of management efficiency. The management server 7 manages the cameras 2 connected to each analysis box 1 as a group, so that the management server 7 can, for example, uniformly send the updated application 51 to each analysis box 1 of the cameras 2 in the management group, and can instruct the cameras 2 in the group to start or stop at the same time.

[0124] As described above, according to the analysis box 1 of the present embodiment, the chips 14 used for the respective inference processing of the NN models for object detection and the NN models for object recognition (included in each AI inference instance 23) are allocated among the plurality of chips, according to the respective inference time and usage frequency of the NN models (NN models for object detection and NN models for object recognition) included in each AI inference instance 23. Thus, even when the type of object recognition for an image input from a certain camera 2 among the plurality of cameras 2 is different from the type of object recognition for an image input from another camera, it is possible to consider the processing time (inference time) and usage frequency of each NN model corresponding to a plurality of object recognitions and object detections, and allocate appropriate chips 14 for the respective inference processing of each of the NN models corresponding to these object recognitions and object detections. Therefore, efficient object recognition for images input from each of the plurality of cameras 2 can be performed using a limited number of chips 14.

[0125] In addition, according to the analysis box 1 of the present embodiment, the number of chips 14 required for the inference processing of each NN model for object recognition is estimated based on the inference time required for the inference processing of each NN model for object recognition and the usage frequency of each NN model for object recognition. Therefore, even when the type of object recognition for an image input from a certain camera 2 among a plurality of cameras 2 is different from the type of object recognition for an image input from another camera, the processing time (inference time) and usage frequency of each NN model corresponding to a plurality of object recognitions can be considered, and an appropriate number of chips 14 can be allocated to the inference processing of each of these NN models corresponding to each of these object recognitions.

[0126] In addition, according to the analysis box 1 of the present embodiment, the number of chips 14 required for the inference processing of the NN model for object detection is estimated based on the inference time required for the inference processing in the NN model for object detection and the number of cameras 2 as input sources of images of objects to be detected based on the NN model for object detection. Here, the frequency of use of the NN model for object detection varies depending on the number of cameras 2 as input sources of images of objects to be detected based on the NN model for object detection. Therefore, as described above, by estimating the number of chips 14 required for the inference processing of the NN model for object detection based on the number of cameras 2 as input sources of images of objects to be detected based on the NN model for object detection and the inference time required for the inference processing in the NN model for object detection, an appropriate number of chips 14 can be allocated to the NN model for object detection.

[0127] In addition, according to the analysis box 1 of the present embodiment, the number of chips 14 required for the inference processing of each object recognition NN model is estimated based on the inference time required for the inference processing of each object recognition NN model, the usage frequency of each object recognition NN model, and the target number of frames for the inference processing of each object recognition NN model within a given time. Here, for example, by changing the above-mentioned target number of frames according to the priority of the object recognition processing performed by each object recognition NN model, it is possible to allocate an appropriate number of chips 14 to the inference processing of each of these NN models in addition to the effects described above, taking into account the priority of the object recognition processing performed by each of the object recognition NN models.

[0128] In addition, according to the analysis box 1 of the present embodiment, the number of chips 14 required for the inference processing of the NN model for object detection is estimated based on the inference time required for the inference processing in the NN model for object detection, the number of cameras 2 that are input sources of images to be detected by the NN model for object detection, and the target number of frames for the NN model for object detection to perform inference processing within a given time. Here, for example, by changing the above-mentioned target number of frames based on the priority of the object detection processing performed by each NN model for object detection (the priority compared with the object recognition processing performed by each NN model for object recognition and the object detection processing performed by other types of NN models for object detection), it is possible to allocate an appropriate number of chips 14 to the object detection processing of the NN model for object detection in consideration of the priority of the object detection processing performed by the NN model for object detection, based on the effects described above.

[0129] In addition, according to the analysis box 1 of the present embodiment, the VMS server 20 (the storage device 22) that accumulates each image input from the camera 2 is further provided, and for the inference processing of the object detection NN model or the object recognition NN model to which the processor allocation unit 19 failed to allocate the inference chip 14 at a certain point in time, after the processor allocation unit 19 becomes able to allocate the chip 14 for the inference processing of the corresponding object detection NN model or the object recognition NN model, the CPU 11 performs the inference processing of the corresponding object detection NN model or the object recognition NN model in non-real time based on the past images accumulated in the VMS server 20. Thus, even if the processor allocation unit 19 fails to allocate the inference chip 14 for the inference processing of a certain object detection NN model or the object recognition NN model at a certain point in time, the CPU 11 can perform the inference processing of the corresponding object detection NN model or the object recognition NN model in addition later based on the past images accumulated in the VMS server 20.

[0130] Furthermore, according to the image analysis system 10 of the present embodiment, in addition to the above-mentioned effects, the management server 7 can be used to manage the analysis box 1 including installing the application 51 in the analysis box 1 .

[0131] Variations:

[0132] In addition, the present invention is not limited to the configurations of the above-described embodiments, and various modifications are possible within the scope of the gist of the invention. Next, modifications of the present invention will be described.

[0133] Modification 1:

[0134] In the above-mentioned embodiment, an example is shown in which the processor allocation unit 19 (GPU server 25) estimates the number of chips 14 (inference processors) required for the inference processing of each NN model (each of the object detection NN model and the object recognition NN model) and allocates the estimated number of chips 14 to each NN model (inference processing). However, in the case where one analysis box is equipped with multiple types of chips (inference processors) and these chips are composed of chips with different performances, the processor allocation unit may also determine the type and number of chips used for the inference processing of each NN model and allocate the determined type and number of chips to each NN model, or may only determine the type of chip used for the inference processing of each NN model and allocate the determined type of chip to each NN model.

[0135] Modification 2:

[0136] In the estimation process of the number of chips required for the NN model for object detection in the above-mentioned embodiment, the processor allocation unit 19 (GPU server 25) estimates the number of chips required for the inference processing of the NN model for object detection (necessary performance) based on the number K of cameras that are the input source of the image as the object to be detected by the NN model for object detection, the model performance T of the NN model for object detection (the inference time required for the inference processing of the NN model for object detection), and the target performance F of the NN model for object detection (the target number of frames for the NN model for object detection to perform inference processing within a given time). However, this is not limited to this. For example, the processor allocation unit 19 can estimate the number of chips required for the inference processing of the NN model for object detection based only on the number K of cameras that are the input source of the above-mentioned image and the model performance T of the NN model for object detection.

[0137] In addition, in the estimation process of the number of chips required for the NN model for object recognition in the above-mentioned embodiment, the processor allocation unit 19 (GPU server 25) estimates the number of chips required for the inference processing of the NN model for object recognition based on the average number of people N1, N2, ... captured by each camera of the input source of the image as the object recognition object based on the NN model for object recognition (that is, the use frequency of the NN model for object recognition), the model performance T of the NN model for object recognition (the inference time (seconds) required for the inference processing of the NN model for object recognition), and the target performance F of the NN model for object recognition (the target number of frames for the inference processing of the NN model for object recognition within a given time). However, this is not limited to this. For example, the processor allocation unit 19 can estimate the number of chips required for the inference processing of the NN model for object recognition based only on the average number of people N1, N2, ... captured by each camera of the input source of the above-mentioned image and the model performance T of the NN model for object recognition.

[0138] Modification 3:

[0139] In the above embodiment, an example is shown in which the image analysis system 10 includes the AI ​​analysis server 6 and the management server 7 on the cloud C. However, the configuration of the image analysis system is not limited thereto, and for example, only the management server may be included on the cloud.

[0140] (Description of the label)

[0141] 1Analysis box (image analysis device)

[0142] 2 Network Camera (Camera)

[0143] 7 Management Server

[0144] 10 Image Analysis System

[0145] 14a~14h chips (processors for inference)

[0146] 18 Image Analysis Department

[0147] 19 Processor Allocation Unit

[0148] 20 VMS Server

[0149] 22 Storage device (image storage unit)

[0150] 23, 23a, 23b, 23c AI reasoning examples (examples of image analysis programs)

[0151] 32 decoder

[0152] 51 Applications (Image Analysis Programs)

[0153] Number of K cameras

[0154] T Model performance (the inference time required for the inference processing in each learned neural network model for object recognition and the inference time required for the inference processing in each learned neural network model for object detection)

[0155] F Target performance (the target number of frames that each neural network model for learning to complete object recognition can perform inference processing within a given time, and the target number of frames that each neural network model for learning to complete object detection can perform inference processing within a given time).

Claims

1. An image analysis device connected to a plurality of cameras, characterized in that: The image analysis device comprises: an image analysis unit that analyzes an image input from each of the plurality of cameras using instances of an image analysis program including a learned object detection neural network model for detecting an object captured in each of the images input from the cameras and one or more learned object recognition neural network models for recognizing an object detected by the learned object detection neural network model; A plurality of inference processors for performing inference processing in the learned object detection neural network model and the learned object recognition neural network model; as well as A processor allocation unit allocates inference processors for inference processing in the learned object detection neural network model and inference processing in each of the learned object recognition neural network models among the multiple inference processors based on the inference time and usage frequency required for the respective inference processing of the learned object detection neural network model and the learned object recognition neural network model contained in each instance of the image analysis program.

2. The image analysis device according to claim 1, characterized in that The processor allocation unit estimates the number of inference processors required for the inference processing of each of the learned object recognition neural network models based on the inference time required for the inference processing in each of the learned object recognition neural network models and the respective usage frequencies of the learned object recognition neural network models.

3. The image analysis device according to claim 1 or 2, characterized in that: The processor allocation unit estimates the number of inference processors required for inference processing of the learned object detection neural network model based on the inference time required for inference processing in the learned object detection neural network model and the number of cameras that are input sources of images of objects to be detected based on the learned object detection neural network model.

4. The image analysis device according to claim 2, characterized in that: The processor allocation unit estimates the number of inference processors required for the inference processing of each of the learned object recognition neural network models based on the inference time required for the inference processing in each of the learned object recognition neural network models, the usage frequency of each of the learned object recognition neural network models, and the target number of frames for inference processing performed by each of the learned object recognition neural network models within a given time.

5. The image analysis device according to claim 3, characterized in that: The processor allocation unit estimates the number of inference processors required for inference processing of the learned object detection neural network model based on the inference time required for inference processing in the learned object detection neural network model, the number of cameras that are input sources of images that are objects of object detection based on the learned object detection neural network model, and the target number of frames for inference processing by the learned object detection neural network model within a given time.

6. The image analysis device according to claim 1, characterized in that: The image analysis device further includes an image storage unit that stores images input from each of the cameras. At a point in time, the processor allocation unit fails to allocate the inference processing of the learned object detection neural network model or the learned object recognition neural network model to the inference processor. Thereafter, after the processor allocation unit becomes able to allocate the inference processor for the inference processing of the corresponding learned object detection neural network model or the learned object recognition neural network model, the image analysis device performs the inference processing of the corresponding learned object detection neural network model or the learned object recognition neural network model in non-real time based on past images accumulated in the image accumulation unit.

7. The image analysis device according to claim 1, characterized in that: The cameras connected to the image analysis device are classified into multiple camera groups, and the image analysis programs corresponding to the multiple camera groups are constructed by using a combination of different neural network models for learning object detection and neural network models for learning object recognition.