Information processing method, information processing device, and computer program

Abstract

To provide a method for processing information, an information processing device, and a computer program capable of appropriately pipeline-processing operations using a learning model.SOLUTION: The method for processing information includes a process in which a computer acquires performance data during execution of a process using one of a plurality of trained neural network models for each of the plurality of trained neural network models trained to output one of a detection result of an object appearing in an image and a recognition result of the object when image data is input, and determines a distribution of processes using the plurality of trained neural network models by using the acquired performance data for a plurality of edge devices.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an information processing method, an information processing device, and a computer program using a trained model (hereinafter referred to as a "trained model"). [Background technology]

[0002] Image recognition systems are used that use a camera that captures the inside of a store or a specific area within its field of view to detect people and other objects in the images captured by the camera and output the recognition results of the detected objects. Such image recognition systems use a learning model that uses a neural network (hereinafter referred to as NN: Neural Network), which is trained to input target image data and output the detection and recognition results of objects in the image data.

[0003] Thanks to improvements in computing power and hardware technology, it is now possible to process images using learning models on edge devices with relatively limited computing resources, rather than on a server with abundant computing resources that collects data and processes the images. For example, AI (Artificial Intelligence) cameras are now being used that incorporate learning models into the camera itself and output image data along with the recognition results. Processing using learning models on edge devices makes it possible to maintain functionality even if there are problems with communication with the server.

[0004] Even with improvements in the hardware performance of edge devices, image recognition systems that require real-time recognition, such as displaying advertisements based on the attributes of customers in images captured by cameras installed in stores, require faster processing speeds.

[0005] Patent Document 1 discloses a technology for performing pipeline processing on images. Pipeline processing is a process in which, when processing of one piece of data is divided into multiple steps, once the first step for one piece of data is completed, a step for the next piece of data is started, and the steps proceed simultaneously in parallel.

[0006] For example, if the processing related to a learning model in image recognition is divided into four steps, such as preprocessing, calculation using the first model, calculation using the second model, and postprocessing, when preprocessing for the first of consecutive frame images ends and calculation using the first model for the first image starts, preprocessing for the second image starts. When calculation using the first model for the first image ends and calculation using the second model for the first image starts, calculation using the first model for the second image and preprocessing for the third image start. This type of pipeline processing is expected to increase speed. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] International Publication No. 2005 / 025230 Summary of the Invention [Problem to be solved by the invention]

[0008] If multiple edge devices can appropriately pipeline processing using multiple learning models, taking into account the performance of each device, such as the processing load, it will be possible to perform processes such as highly accurate image recognition at high speed.

[0009] According to the present invention, an object is to provide an information processing method, an information processing device, and a computer program for a system in which multiple learning models are implemented in multiple edge devices, in which processing using multiple learning models is appropriately pipelined and executed by each edge device, thereby performing processing such as highly accurate image recognition at high speed. [Means for solving the problem]

[0010] In an information processing method according to one embodiment of the present disclosure, when image data is input, a computer acquires performance data for each model during execution of a process using one of a plurality of trained neural network models, each of which has been trained to output a detection result of an object appearing in the image or a recognition result of the object, and determines the allocation of processes using the plurality of trained neural network models to a plurality of edge devices using the acquired performance data.

[0011] In an information processing method according to an embodiment of the present disclosure, the computer is a server connected to the edge devices and acquires results of processes executed by the edge devices. The server stores input data for each trained neural network model obtained when a plurality of processes using each of the plurality of trained neural network models is executed in a predetermined order for at least one image data. For each combination of a plurality of processes using one of the plurality of trained neural network models and a respective one of the edge devices, the server executes a trial process of deploying the input data for each of the stored trained neural network models, together with a process using the trained neural network model to which the input data is input, to one of the edge devices. The server also executes a trial process of acquiring performance data from the deployed edge device when the edge device, to which the process using the trained neural network model has been deployed on a trial basis, executes the process deployed on the edge device using the input data. The server may use the performance data obtained by the process executed for the combination to determine the allocation of the processes using the plurality of trained neural network models to the plurality of edge devices.

[0012] In an information processing method according to an embodiment of the present disclosure, the computer may be a specific edge device among the plurality of edge devices that can be connected to each other for communication, and the specific edge device may store input data for each trained neural network model obtained when the specific edge device executes a plurality of processes using each of the plurality of trained neural network models in a predetermined order on at least one image data. For each combination of a plurality of processes using one of the plurality of trained neural network models and a respective one of the plurality of edge devices, the computer may execute a trial process of deploying the input data for each of the stored trained neural network models, together with a process using the trained neural network model to which the input data is input, to another of the plurality of edge devices. The computer may also execute a trial process of acquiring, from the deployed edge device, performance data obtained by the process executed for the combination when the edge device executes the process using the trained neural network models deployed on the edge device itself. The computer may use the performance data obtained by the process executed for the combination to determine the allocation of the processes using the plurality of trained neural network models to the plurality of edge devices.

[0013] In an information processing method according to one embodiment of the present disclosure, some or all of the edge devices are connected to a camera, and the computer may acquire performance data when the edge device that acquires image data from the camera executes a process using one of the plurality of trained neural network models for each model, and determine the allocation of the processes based on the acquired performance data for each process.

[0014] In the information processing method according to an embodiment of the present disclosure, the computer may acquire, as the performance data, at least one of a processing time, a processing load, a power consumption amount, and a consumed memory amount for each of the processes.

[0015] In an information processing method according to one embodiment of the present disclosure, the edge devices may execute each of a plurality of processes using any of the plurality of trained neural network models in a predetermined order for one piece of image data, and when each of the plurality of processes completes processing for one piece of image data, continue to execute processing for the next input image data.

[0016] In an information processing method according to one embodiment of the present disclosure, the computer may determine the allocation of processing loads based on the number of the edge devices, the processing capabilities of each of the edge devices, and the performance data, so as to minimize the difference in processing load among the edge devices.

[0017] In the information processing method according to an embodiment of the present disclosure, the computer may output the determined process allocation results to a display unit.

[0018] In an information processing method according to an embodiment of the present disclosure, the plurality of edge devices are composed of a first type edge device that executes a process with a uniform computational load on an image and a plurality of second type edge devices that execute a process with a non-uniform computational load on data included in the image, and the computer may change the allocation of processes using the plurality of trained neural network models to the plurality of second type edge devices based on results of the processes in the first type edge devices.

[0019] In the information processing method according to an embodiment of the present disclosure, each of the edge devices may change the content of the processing based on performance data of the process executed by the edge device itself.

[0020] In an information processing method according to one embodiment of the present disclosure, the process includes a storage process for storing data output from a trained neural network model corresponding to the process, and a transmission process for transmitting the data to a server connected for communication, and when each of the edge devices determines that the process load is heavy based on performance data of the process executed by the edge device itself, the edge device may omit at least one of the storage process and the transmission process, or may change the storage destination of the storage process from a non-temporary storage medium to a temporary storage medium.

[0021] An information processing device according to one embodiment of the present disclosure includes a processing unit that, when image data is input, acquires data on the processing load of each process using a plurality of trained neural network models, each of which is trained to output a detection result of an object appearing in the image or a recognition result of the object, and executes a process of determining, for a plurality of edge devices, the allocation of processes using the plurality of trained neural network models using the acquired processing load data.

[0022] A computer program according to one embodiment of the present disclosure causes a computer to, when image data is input, acquire data on the processing load of each process using a plurality of trained neural network models, each of which has been trained to output a detection result of an object appearing in the image or a recognition result of the object, and to execute a process of determining, for a plurality of edge devices, the allocation of processes using the plurality of trained neural network models using the acquired processing load data. [Effects of the Invention]

[0023] According to the present disclosure, in a system in which multiple learning models are implemented in multiple edge devices, it is possible to appropriately pipeline processes using the multiple learning models and have each edge device execute them, taking into consideration the performance of each device, such as the processing load, thereby enabling high-speed execution of processes such as highly accurate image recognition.Furthermore, it is possible to execute processes such as highly accurate image recognition that have undergone location-specific learning in the edge devices. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 is a schematic diagram of an image analysis system. [Figure 2] FIG. 2 is a block diagram showing a configuration of an edge device. [Figure 3] FIG. 2 is a block diagram showing the configuration of a server. [Figure 4] FIG. 2 is a block diagram showing the configuration of a client; [Figure 5] FIG. 1 is an overview of a group of learning models. [Figure 6] FIG. 10 is an explanatory diagram of pipeline processing using a group of learning models. [Figure 7] FIG. 10 is an explanatory diagram of process sharing among a plurality of edge devices. [Figure 8] 10 is a flowchart illustrating an example of a process performed by a server. [Figure 9] 10 is a flowchart illustrating an example of a performance measurement process in an edge device. [Figure 10] FIG. 10 is a diagram showing an example of a display on a client. [Figure 11] FIG. 10 is a diagram illustrating an example of a monitor screen. [Figure 12] FIG. 10 is a diagram showing an example of display on a client in a modified example. [Figure 13] FIG. 10 is a diagram showing an example of a map screen in a modified example. [Figure 14] 10 shows an example of a monitor screen in a modified example. [Figure 15] 10 is a flowchart showing an example of a procedure for determining a share in the second embodiment. [Figure 16] FIG. 10 is an explanatory diagram of process allocation in the second embodiment. [Figure 17] 11 is a flowchart illustrating an example of a processing procedure in an edge device according to the third embodiment. [Figure 18] 11 is a flowchart illustrating an example of a processing procedure in an edge device according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0025] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present disclosure will be specifically described with reference to the drawings illustrating embodiments thereof. In the following embodiments, an image analysis system to which the information processing method of the present disclosure is applied will be described.

[0026] (First embodiment) 1 is a schematic diagram of an image analysis system 100. The image analysis system 100 in the first embodiment includes one or more cameras 4 installed in a space to be analyzed, such as a store, multiple edge devices 1 connected to the cameras 4, a server 2 that can be connected to the edge device 1 for communication, and a client 3 that can be connected to the server.

[0027] One or more cameras 4 and multiple edge devices 1 can be connected to each other via a wireless or wired local network LN. The local network LN may be a wired LAN or a wireless network such as WiFi or Bluetooth (registered trademark).

[0028] One or more cameras 4 use image elements that are responsive to visible light and / or near-infrared light and output image data. The cameras 4 output image data in time series at a rate of several fps to several tens of fps. The cameras 4 are installed so as to look down from the top of the space to be analyzed, such as on the ceiling or shelves. The cameras 4 may be ceiling-mounted cameras that can cover the entire space in a 360-degree field of view. The cameras 4 sequentially transmit the image data to the edge device 1 via the local network LN.

[0029] Some or all of the multiple edge devices 1 can be connected to a server 2 via a network N. The network N is a wired or wireless communication network that may include a public communication network, a dedicated line, or a carrier network. The client 3 can be connected to the server 2 via the network N.

[0030] The edge device 1 performs image recognition processing on image data acquired from the camera 4, detects objects within the field of view of the camera 4 by distinguishing them from the background, and outputs text indicating the object's attributes and data on its position within the image. The edge device 1 may also identify the movement of objects within the field of view of the camera 4 and output text indicating the identification result. Based on the position of the camera 4 and the position of the object within the image, the edge device 1 calculates data (time-series coordinate data within the space) indicating the path of the object detected in the space to be analyzed where the camera 4 is installed.

[0031] The edge device 1 stores text indicating the attributes of the detected object obtained by calculation and coordinate data indicating the object's trajectory in association with identification data that enables the space to be analyzed to be identified, and also transmits the same to the server 2 as necessary.

[0032] The server 2 records the data sent from the edge device 1 in a database 210 and can perform analysis processing such as aggregating the attributes of detected objects and statistical processing of object paths for each space being analyzed. The results of the analysis processing by the server 2 can be viewed from the client 3 for each space being analyzed. The manager of the space being analyzed can use the client 3 to design the space according to the paths of objects detected within the space. More specifically, by installing the image analysis system 100 in a store, the store manager can check from the client 3 the results of aggregating the age group, gender, etc. of customers visiting the store, as well as suggestions for product lineups and placement based on the paths customers take within the store.

[0033] The image analysis system 100 of the first embodiment applies the information processing method of the present disclosure and optimizes the distribution of processes using each of multiple learning models according to the performance of each of multiple edge devices 1 placed in the space to be analyzed. Performance may vary depending on the specifications of each edge device 1, the specifications of the installed cameras 4, and the attributes of objects to be detected in the space to be analyzed. The configurations of the edge devices 1 and server 2 for realizing such an image analysis system 100, as well as details of the processes, are described below.

[0034] 2 is a block diagram showing the configuration of the edge device 1. The edge device 1 is a box-shaped device that can be installed together with cameras 4 in a space to be analyzed, such as a store. The edge device 1 includes a processing unit 10, a storage unit 11, a first communication unit 12, and a second communication unit 13.

[0035] The processing unit 10 includes one or more processors such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), etc. The processing unit 10 includes a memory that is a temporary storage medium such as a static random access memory (SRAM) or a dynamic random access memory (DRAM). The processing unit 10 includes a timer and can acquire time information at each point in time from data from the timer. The processing unit 10 may be configured as a single piece of hardware (SoC: System On a Chip) that integrates a processor, a memory, a storage unit 11, a first communication unit 12, and a second communication unit 13. The specifications of the processing unit 10 may be the same or different between edge devices 1.

[0036] The processing unit 10 reads the information processing program P1 stored in the storage unit 11 into the memory and executes it, thereby causing the processor to execute various processes described below and function as the edge device 1 of the present disclosure.

[0037] The storage unit 11 is a relatively large-capacity non-transitory storage medium such as a hard disk, a flash memory, etc. A part of the storage unit 11 may be removable.

[0038] The storage unit 11 stores a program (program product) required for the processing unit 10 to execute processing, the results of processing by the processing unit 10, and reference setting data. The setting data includes an edge device identifier, identification data of the connected camera 4, etc. The program product includes an OS (Operating System) program, an information processing program P1 that runs on the OS, and a learning model group M1. The learning model group M1 will be described in detail later.

[0039] The information processing program P1 or the learning model group M1 stored in the memory unit 11 may be the information processing program P9 and the learning model M9 stored in a computer-readable non-transitory storage medium 9 that are read by the processing unit 10 and stored in the memory unit 11, or may be pre-stored at the time of shipment. The information processing program P1 or the learning model group M1 stored in the memory unit 11 may be the information processing program P9 or the learning model M9 that the processing unit 10 downloads from the server 2 or from another download server via the second communication unit 13 and stores in the memory unit 11.

[0040] The storage unit 11 stores image data acquired from the connected camera 4 in association with time information. The storage unit 11 preferably stores only image data in which the detection target has been detected by the processing unit 10. The storage unit 11 may continue to store data by overwriting it according to the FIFO (First In First Out) method to a range according to the capacity. The storage unit 11 may store data of the detection results (such as text indicating attributes) in association with time information. The storage unit 11 stores performance data of its own device obtained by processing described below.

[0041] The first communication unit 12 is a communication device that realizes communication via the local network LN. The first communication unit 12 may be a LAN network card or a CAN communication device. The first communication unit 12 may be a communication device compatible with wireless networks such as Wi-Fi or Bluetooth (registered trademark). The first communication unit 12 may include multiple communication devices compatible with various types of cameras 4. The first communication unit 12 may include an interface such as a USB (Universal Serial Bus) connected to the camera 4. The first communication unit 12 can be replaced by an interface connected to the camera 4 via a coaxial cable or another serial bus. The processing unit 10 acquires image data from the camera 4 via the local network LN through the first communication unit 12. The processing unit 10 can transmit and receive data to and from other edge devices 1 through the first communication unit 12. The first communication unit 12 may be the same device as the second communication unit 13.

[0042] When the processing unit 10 receives image data from different cameras 4 via the first communication unit 12, it is preferable that the processing unit 10 distinguishes the data port to identify which camera 4 the data is from, or distinguishes the interface physically, and stores the data in the memory unit 11 in association with the identification data of the camera 4.

[0043] The second communication unit 13 is a communication device that realizes communication via an external network N. The second communication unit 13 may be a network card for a wired LAN, a communication device that realizes carrier communication via a carrier network, or a communication device that supports a wireless network such as WiFi or Bluetooth (registered trademark). The second communication unit 13 may support encrypted communication with the server 2 using SSL or the like. The second communication unit 13 may be an interface that realizes connection with the server 2 via a dedicated line.

[0044] 3 is a block diagram showing the configuration of the server 2. The server 2 may be configured as a single server computer, or may be configured to distribute processing among multiple server computers. The server 2 includes a processing unit 20, a storage unit 21, and a communication unit 22.

[0045] The processing unit 20 includes one or more processors such as a CPU, an MPU, a GPU, etc. The processing unit 20 includes a memory that is a temporary storage medium such as an SRAM or a DRAM.

[0046] The storage unit 21 is a relatively large-capacity non-temporary storage medium such as a hard disk, a flash memory, etc. The storage unit 21 stores a program (program product) required for the processing unit 20 to execute processing.

[0047] The program product stored in the storage unit 21 includes a server program P2. The server program P2 includes a module that functions as a web server, and can output the results of processing in the server 2 to the client 3 via a web page. The storage unit 21 stores login account data for distinguishing users who use the client 3, making it possible to identify whether the logged-in user is an account with authority to operate the server 2, or an account of a customer who will view the processing results of the server 2. The program for outputting data to the client 3 included in the server program P2 is not limited to a web server program, and may be a program compatible with other protocols as long as it can output data to a screen displayed on the client 3.

[0048] The program product stored in the storage unit 21 includes a learning model group M2. The learning model group M2 is stored for distribution to the edge device 1, and if it is stored in another download server, it does not need to be stored in the storage unit 21. The learning model group M2 will be described in detail later.

[0049] The server program P2 or the learning model group M2 stored in the memory unit 21 may be the server program P8 and the learning model M8 stored in the computer-readable non-transitory storage medium 8 that are read by the processing unit 20 and stored in the memory unit 21, or may be pre-stored at the time of shipment. The server program P2 or the learning model group M2 stored in the memory unit 21 may be the server program P8 and the learning model M8 that are stored in the computer-readable non-transitory storage medium 8 that are read by the processing unit 20 and stored in the memory unit 21 via the communication unit 22 from a download server.

[0050] The storage unit 21 has a database 210 configured to store space definition data and administrator data for the analysis target space in association with the identification data of the analysis target space. The storage unit 21 may store the account of the client 3 in association with each analysis target space in the database 210, thereby narrowing down the analysis results that can be referenced by the client 3. The storage unit 21 stores data such as the number and specifications of edge devices 1 arranged in the space, and the number and specifications of cameras 4, in association with the identification data of the analysis target space. Specifically, the storage unit 21 stores layout data of the store floor and shelves, installation positions and angle of view data of the cameras 4 corresponding to the floor, and identification data of the cameras 4 in association with the identification data of the store in the database 210. The storage unit 21 stores detection and recognition results transmitted from the edge devices 1 in association with the identification data of the analysis target space in the database 210. The storage unit 21 stores performance data transmitted from the edge devices 1 in association with the identification data of the edge devices 1.

[0051] The communication unit 22 is a communication device that realizes a communication connection with the client 3 and the edge device 1 via the network N.

[0052] 4 is a block diagram showing the configuration of the client 3. The client 3 is a personal computer, a smartphone, or a tablet terminal. The client 3 may be used by an administrator of the space to be analyzed, or by an operator of the server 2.

[0053] The client 3 includes a processing unit 30, a storage unit 31, a communication unit 32, a display unit 33, and an operation unit 34. The processing unit 30 includes one or more processors such as a CPU, an MPU, a GPU, etc. The processing unit 30 includes a memory that is a temporary storage medium such as an SRAM or a DRAM.

[0054] The storage unit 31 is a memory of a non-temporary storage medium such as a hard disk or a flash memory. The storage unit 31 stores a client program corresponding to an information provision protocol from the server 2. The client program is, for example, a web browser program. The client program is not limited to a web browser program, and may be a special program that causes the processing unit 30 to execute a process of displaying data transmitted from the server 2 on a screen.

[0055] The communication unit 32 is a communication device that realizes a communication connection with the server 2 via the network N. The communication unit 32 may be a communication device that realizes a communication connection with the server 2 via a dedicated line.

[0056] The display unit 33 uses a display such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 33 displays a web page including text and images through processing based on the client program of the processing unit 30. The display unit 33 may use a display with a built-in touch panel.

[0057] The operation unit 34 is a user interface such as a keyboard or a pointing device that accepts operations from a user or an operator. The operation unit 34 may be a touch panel built into the display of the display unit 33, or may be physical buttons. The operation unit 34 may be a voice input unit that accepts operations by voice using a voice recognition function. The operation unit 34 can notify the processing unit 30 of operation information by the user or operator.

[0058] FIG. 5 is a schematic diagram of the learning model groups M1 and M2. Each of the learning model groups M1 and M2 includes multiple learning models related to image recognition. The learning model groups M1 and M2 are model groups trained to output multiple types of recognition results for input image data, and are configured to be divided into models and deployed according to the type of recognition. For example, as shown in FIG. 5, the learning model group M2 includes a head detection model M21, a head region extraction model M22, a head direction discrimination model M23, an age discrimination model M24, and a mask recognition model M25.

[0059] The head detection model M21 is a model trained using a neural network (NN), such as a single-shot multibox detector (SSD), to output, for an input image, coordinate data of an area containing a head in the image and a score indicating the likelihood that the area is a head if a person is captured in the image. Each model may employ a model without a convolutional layer, such as a Transformer, or other algorithms. For example, when a time-series frame image, i.e., video data, is input, the head detection model M21 outputs, as a feature, coordinate data of a rectangular area containing the head for each frame image. The frame images may be subjected to predetermined preprocessing, such as filtering, before being input to the head detection model M21. If the image contains multiple heads, the head detection model M21 outputs coordinate data of rectangular areas containing the heads for each head. If the input frame image does not contain a person, the head detection model M21 may output "no detection" or may output 0 (zero) as coordinate data. The head detection model M21 has been trained using a general-purpose training data set regardless of the detection target space. The head detection model M21 may be a distilled and lightweight model. As shown in FIG. 5, the head detection model M21 may be selected from a plurality of stored patterns, each of which may be provided by a different model provider or have a different training data set.

[0060] The head region extraction model M22 is a model trained using a neural network (NN) so that when an image containing a head and coordinate data of a rectangular region including the head output as features from the head detection model M21 are input, the model outputs a (rectangular) image of the head region as features. The head region extraction model M22 has also been trained using a general-purpose training data set, regardless of the detection target space. The head region extraction model M22 may be a lightweight model that has been distilled. The head region extraction model M22 may be selected from multiple patterns stored, for example, based on different model providers or different training data sets.

[0061] As with the head region extraction model M22, the input data to the head direction discrimination model M23 may be an image containing a head and coordinate data of a rectangular region including the head output as a feature from the head detection model M21, or an image of the head region output as a feature from the head region extraction model M22. The head direction discrimination model M23 is a model trained using a neural network to output, as a feature, vector data indicating the direction in which the head is facing when the above data is input. The output data is a vector indicating the orientation relative to the vertical direction in the space captured in the image and the horizontal direction in the image. The head direction discrimination model M23 is also trained using a general-purpose training dataset, regardless of the detection target space. The head direction discrimination model M23 may be a lightweight model that has been distilled. The head direction discrimination model M23 may be selected from multiple patterns stored, for example, due to different model providers or different training datasets.

[0062] The input data to the age discrimination model M24 may be an image of a head and coordinate data of a rectangular image including the head output as a feature from the head detection model M21, or an image of the head region output as a feature from the head region extraction model M22. However, in either case, it is desirable to also add vector data indicating the direction the head is facing, output as a feature from the head direction discrimination model M23, to the input data to the age discrimination model M24. The age discrimination model M24 is a model trained using a neural network to estimate the age of the person whose head is shown in the image when the above data is input and output as a feature. The data output from the age discrimination model M24 is a number indicating the age. The age discrimination model M24 may be a distilled and lightweight model. The age discrimination model M24 may be selected from multiple patterns stored, such as those from different model providers or different training datasets.

[0063] The input data to the mask recognition model M25 may be an image of a head and coordinate data of a rectangular area including the head output as a feature from the head detection model M21, or an image of the head area output as a feature from the head area extraction model M22. In either case, it is desirable to add vector data indicating the direction the head is facing, output as a feature from the head direction discrimination model M23, to the input data to the mask recognition model M25. The mask recognition model M25 is a model trained using a neural network to output, as a feature, logical data indicating whether the person whose head is shown in the image is wearing a mask. The mask recognition model M25 may be a lightweight model that has been distilled. The mask recognition model M25 may be selected from multiple patterns stored, for example, by different model providers or different training datasets.

[0064] The learning model group M2 may include a gender discrimination model, a movement discrimination model, etc. in addition to the models shown in Fig. 5. The movement discrimination model can automatically execute a process of recording a video of several seconds to several tens of seconds before and after a frame image stored in the memory unit 11 when a person makes a suspicious movement.

[0065] The learning model group M1 is selected from the learning model group M2 according to each analysis target space. The image analysis system 100 of the first embodiment corresponding to one analysis target space includes, as the learning model group M1, a head detection model M11, a head region extraction model M12, a head direction discrimination model M13, an age discrimination model M14, and a mask recognition model M15. The head detection model M11 is a model selected, distilled, and lightweight from among multiple patterns of head detection models M21 stored in the server 2. Similarly, the head region extraction model M12, the head direction discrimination model M13, the age discrimination model M14, and the mask recognition model M15 are each models selected, distilled, and lightweight from among models stored in the server 2.

[0066] 6 is an explanatory diagram of pipeline processing using the learning model group M1. The edge device 1 receives image data in time series from the camera 4. Each piece of image data received in time series is called a frame image, and will be explained below.

[0067] The processing unit 10 of the edge device 1 can execute the processing of the head detection model M11, head region extraction model M12, head direction discrimination model M13, age discrimination model M14, and mask recognition model M15 in parallel in different process units (instances, blocks). In the process using the head detection model M11, the processing unit 10 inputs the first frame image of consecutive frame images received from the camera to the head detection model M11. When the processing unit 10 obtains a detection result from the head detection model M11, if a person is captured in the frame, the processing unit 10 passes the image containing the head and the coordinate data of the rectangular region including the head output as a feature from the head detection model M11 to the process using the head region extraction model M12. If a person is not captured in the first frame image, the processing unit 10 ends the processing there. The process using the head region extraction model M12 is not executed. For example, if a head is not detected in the fifth frame image, the next process is not executed.

[0068] When a person is present in the first frame image, the processing unit 10 can execute a process using the head detection model M11 for the second frame image in parallel with the start of processing of the process using the head area extraction model M12 for the first frame image.

[0069] 6, the processing unit 10 acquires an image of the head region through a process using the head region extraction model M12 for the first frame image, and then passes this image to a process using the head direction discrimination model M13. The processing unit 10 can then acquire the detection result for the second frame image by the head detection model M11, which was being performed in parallel at the same time.

[0070] At the same time that the processing unit 10 starts a process using the head direction discrimination model M13 for the first frame image, it can execute a process using the head area extraction model M12 for the second frame image and a process using the head detection model M11 for the third frame image in parallel.

[0071] 6, the processing unit 10 acquires vector data indicating the direction in which the head is facing determined by a process using the head direction discrimination model M13 for the first frame image, and then passes the image of the head region output from the head region extraction model M12 and the vector data indicating the direction in which the head is facing output from the head direction discrimination model M13 to a process using the age discrimination model M14. The processing unit 10 can acquire the extraction result of the head region image by the head region extraction model M12 for the second frame image and the detection result by the head detection model M11 for the third frame image, which were performed in parallel at the same time.

[0072] At the same time that the processing unit 10 starts a process using the age discrimination model M14 for the first frame image, it can simultaneously execute a process using the head direction discrimination model M13 for the second frame image, a process using the head area extraction model M12 for the third frame image, and a process using the head detection model M11 for the fourth frame image.

[0073] 6, the processing unit 10 acquires a number indicating the age of the person appearing in the first frame image through a process using the age discrimination model M14 for the first frame image, and then passes the image of the head region output from the head region extraction model M12 and the vector data indicating the direction the head is facing output from the head direction discrimination model M13 to a process using the mask recognition model M15. The processing unit 10 can acquire the vector data indicating the head direction for the second frame image, the extraction result of the head region image for the third frame image, and the head detection result for the fourth frame image, which were all performed in parallel.

[0074] At the same time as starting a process using the mask recognition model M15 for the first frame image, the processing unit 10 can simultaneously execute a process using the age discrimination model M14 for the second frame image. At the same time, the processing unit 10 can simultaneously execute a process using the head direction discrimination model M13 for the third frame image, a process using the head region extraction model M12 for the fourth frame image, and a process using the head detection model M11 for the fifth frame image.

[0075] The processing unit 10 acquires logical data indicating whether or not the person appearing in the first frame image is wearing a mask through a process using the mask recognition model M15 for the first frame image, and ends the processing using the learning model M1 for the first frame image. The processing unit 10 can acquire, in parallel, the number indicating the age of the person appearing in the second frame image, vector data indicating the head direction of the person appearing in the third frame image, the extraction result of the head region image for the fourth frame image, and the head detection result for the fifth frame image.

[0076] In this way, the processing unit 10 can increase the speed by pipelined processing of frame images that can be acquired in time series. Here, the process shown in FIG. 6 can be performed by different edge devices 1. By distributing the pipeline processing among multiple edge devices 1, high-speed processing as a whole can be achieved even if each edge device 1 does not have abundant hardware resources.

[0077] FIG. 7 is an explanatory diagram of process allocation among multiple edge devices 1. The explanatory diagram in FIG. 7 shows different patterns of which edge device 1 executes the pipelined process shown in FIG. 6. FIG. 7A shows an example in which a first edge device 1 executes a process using a head detection model M11, a second edge device 1 executes a process using a head region extraction model M12 and a process using a head direction discrimination model M13, and a third edge device 1 executes a process using an age discrimination model M14 and a process using a mask recognition model M15. FIG. 7B shows an example in which a first edge device 1 executes a process using the head detection model M11 and a process using the head region extraction model M12, and a second edge device 1 executes a process using the head direction discrimination model M13, a process using the age discrimination model M14, and a process using the mask recognition model M15.

[0078] 7, the method of allocating processes using each model can be varied in various ways depending on the number of edge devices 1 to be installed, the specifications of each edge device 1, and the weight of each process. Therefore, in the first embodiment, each process of the learning model group M1 is trial-deployed to each edge device 1, and the process of measuring the processing speed, processing load, and power consumption of each process on the edge device 1 is repeated, and the process allocation method is determined based on the results.

[0079] 8 is a flowchart showing an example of the processing of the server 2. In the first embodiment, the server 2 executes the following processing, but it may also be a specific edge device 1, in particular, an edge device 1 connected to the camera 4. Alternatively, a device with a higher processing capacity than the edge device 1 may execute the following processing.

[0080] The processing unit 20 of the server 2 acquires image data (step S201). In step S201, the server 2 acquires image data that can be acquired by the image analysis system 100 for which allocation is to be determined.

[0081] The processing unit 20 inputs the acquired image data to the head detection model M21 and acquires coordinate data of a rectangular area including the head, which is output as a feature from the head detection model M21 (step S202). The processing unit 20 stores input data (image data) necessary for calculations based on the head detection model M21 (step S203).

[0082] If the detection result by the head detection model M21 indicates that the head is not shown, the processing unit 20 does not execute the subsequent processes and ends the process.

[0083] The processing unit 20 inputs the coordinate data of the rectangular area including the head acquired by the head detection model M21 into the head area extraction model M22 to acquire a (rectangular) image of the head area (step S204). The processing unit 20 stores the input data (coordinate data of the rectangular area including the head) required for calculation based on the head area extraction model M22 (step S205).

[0084] The processing unit 20 inputs the image of the head region acquired in step S204 into the head direction discrimination model M23 to acquire vector data indicating the head direction (step S206). The processing unit 20 stores input data (image of the head region) required for calculation based on the head direction discrimination model M23 (step S207).

[0085] The processing unit 20 inputs the image of the head region acquired in step S204 and the vector data indicating the head direction acquired in step S206 into the age discrimination model M24 to acquire the age number (step S208). The processing unit 20 stores the input data (the image of the head region and the vector data indicating the head direction) required for calculations based on the age discrimination model M24 (step S209).

[0086] Processing unit 20 inputs the image of the head region acquired in step S204 and the vector data indicating the head direction acquired in step S206 into mask recognition model M25 to acquire logical data indicating whether or not a mask is being worn (step S210). Processing unit 20 stores the input data (image of the head region and vector data indicating the head direction) required for calculation based on mask recognition model M25 (step S211), and ends the process.

[0087] In step S201, the server 2 may acquire from the database 210 image data of the same type as the image data captured in the space to be analyzed, rather than data (camera image data) that can actually be acquired by the image analysis system 100 (space to be analyzed). The server 2 may also acquire a data set from a provider site. In this case, for example, it is preferable to acquire image data from a group of image data captured by a camera 4 with the same or similar specifications and resolution as a camera 4 installed in a store, which is the space to be analyzed, and which is captured in a store of the same type of format.

[0088] 8 may be executed by a specific edge device 1. In this case, the server 2 does not need to acquire image data from the camera 4 that is taken in the space to be analyzed.

[0089] FIG. 9 is a flowchart showing an example of a performance measurement process in the edge device 1.

[0090] The server 2 selects and deploys a model to each edge device 1 in response to an operation from an operator of the server 2 via the client 3 or by calculation by the processing unit 20 itself (step S221). At this time, the server 2 selects and transmits input data corresponding to the deployed model to each edge device 1 from the input data stored by the processing of FIG. 8 (step S222).

[0091] The processing unit 10 of the edge device 1 receives a model and input data to be deployed to the device from the learning model group M1 (step S101) and stores them (step S102). The processing unit 10 provides the input data received in step S101 (input data corresponding to the deployed learning model) to one of the learning model group M1 deployed to the device, and executes a process including this learning model (step S103).

[0092] In step S103, each edge device 1 in which the model is deployed performs a calculation using the transmitted feature amounts without transmitting or receiving the recognition results or discrimination results to or from each other.

[0093] During step S103, the processing unit 10 acquires (step S104) and stores (step S105) performance data of the processes based on the model deployed to the device itself. In step S104, the processing unit 10 acquires at least one of the processing time, processing load, power consumption, and memory consumption of each process. Some or all of the edge devices 1 may each have a display, and each may output the contents of the performance data as text or an image. For example, the processing unit 10 of the edge device 1 outputs the processor utilization rate in text to the display.

[0094] The processing unit 10 associates the acquired performance data with the identification data of the edge device 1 and data indicating one of the deployed learning models M1, and transmits the data to the server 2 (step S106).

[0095] The processing unit 20 of the server 2 receives the performance data transmitted from each edge device 1 (step S223), stores the performance data in association with data indicating one of the learning models M1 and the identification data of the edge device 1 (step S224), and terminates the processing.

[0096] In response to the selection of a model to be deployed to the edge device 1 in S221, the server 2 may cause each edge device 1 to execute the measurement process (S103, S104) multiple times, and calculate statistical values ​​such as the average, median, and mode for at least one of the processing time, processing load, power consumption, and memory consumption included in the performance data.

[0097] For each model included in the pipeline to be executed in the actual installation environment (such as a store) of multiple edge devices 1, the server 2 performs a process of trial deployment of the model and input data (features) corresponding to the model to each edge device 1 (deployment process of the model to the edge device 1 in S221) for all edge devices 1, and performs a process of acquiring performance data shown in Fig. 9 (acquisition process of performance data when each model is deployed (placed) in each edge device 1). This makes it possible to obtain performance data when each of all models included in the pipeline to be executed in the actual installation environment is deployed to all edge devices 1, and therefore the server 2 can propose an optimal placement of each model to each edge device 1 based on the processing time or the (total) power consumption, based on this performance data.

[0098] By recording actual performance data for each analyzed space, performance can be output in response to a request from client 3, as shown below.

[0099] FIG. 10 is a diagram showing an example of a display on the client 3. FIG. 10 shows an example of a screen provided by the web server function of the server 2 in response to access from the processing unit 30 of the client 3, displayed by a web browser. An output screen 330 shown in FIG. 10 displays a list 331 of spaces to be analyzed that the operator of the server 2 is permitted to view. The list 331 includes an interface 332 with a link to a performance monitor screen for each edge device 1. In the example of FIG. 10, a list 331 of "Store (X Chain Store A)," "Store (X Chain Store B)," and "Store (X Chain Store C)" is displayed.

[0100] Fig. 11 is a diagram showing an example of a monitor screen 333. The monitor screen 333 shown in Fig. 11 is displayed when an interface 332 for "Store (X Chain Store A)" is selected by the operation unit 34 from the list 331 shown in Fig. 10.

[0101] The monitor screen 332 is a screen that visualizes the performance data of each edge device 1. The monitor screen 333 shown in Fig. 11 displays a map image 334 of the store as the space to be analyzed. Icons 335 indicating the locations of the cameras 4 are displayed on the map image 334.

[0102] The monitor screen 333 includes, on a map image 334, the arrangement of edge devices 1 and objects 336 that visualize the performance of the edge devices 1. In the example of FIG. 11, the objects 336 are circular, and the diameter indicates the magnitude of the load. Next to the objects 336, text 337 of performance data for each edge device 1 is displayed. In FIG. 11, the text 337 outputs the measured process speed (how many frames can be processed per second) and the amount of power consumption.

[0103] The monitor screen 333 includes an interface 338 for switching the display content of performance data. In the example shown in Fig. 11, the interface 338 allows the user to select whether the proposed placement of models for each edge device 1 displayed in a window 339 should be displayed based on "power" or speed. In the example shown in Fig. 11, "power" is selected.

[0104] The monitor screen 333 includes a window 339 showing a proposal for the placement of models on the edge device 1. Based on the criteria selected in the interface 338 and on performance data when all of the models included in the pipeline to be executed in the above-mentioned actual installation environment are deployed to all of the edge devices 1, the window 339 proposes that, for image data obtained from the first camera 4, the edge device 1 of "Device 1" should execute a first model (e.g., head detection model M11) and a second model (head region extraction model M12), and the edge device 1 of "Device 4" should execute a third model (head direction discrimination model M13), a fourth model (age discrimination model M14), and a fifth model (mask recognition model M15).

[0105] In this way, by making the monitor screen 333 as shown in FIG. 11 visible, the operator of the server 2 of the image analysis system 100 or the administrator of the space to be analyzed can check the performance balance of each edge device 1 and consider the allocation of processes corresponding to each model depending on the actual operational results in the space to be analyzed.

[0106] 11 or based on the performance information output on the display provided in each edge device 1, the operator of the server 2 of the image analysis system 100 or the administrator of the space to be analyzed deploys a model from the group of learning models M1 to each edge device 1. Alternatively, the server 2 deploys each model to each edge device 1 based on performance data when each of the models included in the pipeline executed in the above-mentioned actual installation environment is deployed to each of the edge devices 1. This makes it possible to distribute processing to the edge devices 1 appropriately based on performance.

[0107] (Variation) After appropriate distribution, the distribution of processing may be optimized based on the performance of the edge device 1. In a modified example, even after operation has begun, the edge device 1 acquires performance data for processes based on the models deployed thereon, and transmits the performance data when transmitting detection results and recognition results to the server 2. As a result, the processing unit 20 of the server 2 associates the performance data with the identification data of the analysis target space and the identification data of the edge device 1 that transmitted the detection results and recognition results transmitted from each edge device 1, and stores the performance data in association with data indicating one of the learning models M1 and the identification data of the edge device 1.

[0108] During analysis by the image analysis system 100, the processing unit 20 of the server 2 can compile data on the number of people who have visited the space being analyzed (for example, a store), their ages, and whether they are wearing masks, based on the detection and recognition results. The processing unit 20 can also create an analysis map showing which shelves are attracting the most attention, based on the data on the direction the head is facing, and on the angle of view and position within the space associated with the identification data of each camera 4.

[0109] In a modified example, actual performance data can be recorded during the operation of the analysis by image analysis system 100 in the space to be analyzed, and can be output in response to a request from client 3.

[0110] FIG. 12 is a diagram showing an example of a display on a client 3 in a modified example. FIG. 12 shows an example of a screen that can be confirmed by the administrator of the space to be analyzed in an image analysis system 100 that has started operation. An output screen 340 shown in FIG. 12 displays a list 341 of spaces to be analyzed that the user of the client 3 (user ID: 012345) is permitted to view. The list 341 includes, for each space to be analyzed, an interface 342 with a link to a screen including an analysis map, and an interface 343 with a link to a monitor screen of the performance of each edge device 1. In the example of FIG. 12, lists 341 of "Store (X Chain Store A)," "Store (X Chain Store B)," and "Store (X Chain Store C)" are displayed.

[0111] Fig. 13 is a diagram showing an example of a map screen 344 in a modified example. The map screen 344 shown in Fig. 13 is displayed when the manager of the space to be analyzed selects, via the operation unit 34, an interface 342 for "Store (X Chain Store A)" from the list 341 shown in Fig. 12.

[0112] A map screen 344 displays a map image 345 of the store as the space to be analyzed. Icons 346 indicating the placement and angle of view of the cameras 4 are displayed on the map image 345. A graphic 347 showing the path of customers who visited the store (Store A of X Chain) is superimposed on the map image 345. The path is identified by analyzing whether the detection results and recognition results of customers for images captured by each camera 4 match and whether there are any temporal inconsistencies. The server 2 may acquire feature amounts from the edge device 1 using the head detection model M11, and derive the path by treating detected features as the head of the same person.

[0113] The map screen 344 includes a screen 348 that displays statistical results on the attributes of customers who visited the store (Store A of X Chain). The screen 348 includes graphs showing a weekly visitor count, age distribution, and recognition results of whether or not a mask was worn. In response to a request from the client 3 to display the map screen 344, the server 2 extracts data from the database 210 and draws it.

[0114] When an interface 343 is selected on the output screen 340 shown in Fig. 12, a monitor screen 333 showing the performance of each edge device 1 after operation has started is displayed. Fig. 14 shows an example of the monitor screen 333 in a modified example. The screen 333 shown in Fig. 14 is similar to the monitor screen 333 in Fig. 11 described in the first embodiment, and therefore the same reference numerals are used to designate common components, and detailed description thereof will be omitted.

[0115] The text displayed on the monitor screen 333 in FIG. 14 may be displayed based on the most recent performance data obtained from the edge device 1, or may be displayed based on statistical data for one day's worth of performance data. The monitor screen 333 in FIG. 14 includes an interface 349 for requesting the server 2 to create a report based on the performance data. When the interface 349 is selected, the server 2 receives the request to create a report and statistically processes the performance data regarding the performance of the edge device 1 in the analysis target space currently in operation to create a document file. The server 2 calculates information identifying the process performed by each edge device 1 (e.g., identification data for the model performed by that edge device 1), the average processing time and time zone distribution for each device, and the average power consumption and time zone distribution. The server 2 outputs a report including the calculation results. The server 2 may include in the report a proposal for allocating processes corresponding to each learning model.

[0116] In this way, in the modified example, by making the map screen 344 as shown in FIG. 13 visible, the administrator of the space to be analyzed can check statistical data on the paths of detected objects and aggregated attributes, and can also check the performance balance of each edge device 1 on a screen such as that shown in FIG. 14, and can consider reallocating the processes corresponding to each learning model or relocating the cameras 4 depending on the actual operational results in the space to be analyzed, seasonal differences, and changes in fashion.

[0117] (Second embodiment) In the second embodiment, it is possible to dynamically change the allocation of processes from the server 2 or a specific edge device 1. The hardware configuration of the image analysis system 100 of the second embodiment is similar to the hardware configuration of the image analysis system 100 of the first embodiment, so the same reference numerals are used for the common components and detailed descriptions will be omitted.

[0118] In the image analysis system 100 of the second embodiment, a process using a head detection model M11, which is executed with a uniform amount of calculation regardless of whether detection is present or not, is executed by a first edge device 1 (referred to as a "first type edge device" in the claims). When the first edge device 1 executes a process using a head region extraction model M12 or later as a result of the process using the head detection model M11, the first edge device 1 dynamically determines which of the second, third, fourth, ..., edge devices 1 (referred to as a "second type edge device" in the claims) will execute those processes, and executes the processes by calling the processes. Each edge device 1 stores both models in its storage unit 11, and reads and uses the specified model when it is called.

[0119] 15 is a flowchart showing an example of a procedure for determining allocation in the second embodiment. In the second embodiment, the first edge device 1 executes the following processing procedure. For simplicity of explanation, the following description will be given assuming that one camera 4 is connected, but in a state where inputs from multiple cameras 4 are provided to the first communication unit 12 of the first edge device 1, processing is performed to identify which camera 4 the data comes from and distinguish the detection results, etc.

[0120] The processing unit 10 of the first edge device 1 acquires image data from the corresponding camera 4 (step S301). The processing unit 10 inputs the acquired image data to the head detection model M11 (step S302). The processing unit 10 acquires a detection result from the head detection model M11 (step S303). The processing unit 10 may perform appropriate preprocessing on the acquired image data before executing the head detection model M11 in step S302.

[0121] The processing unit 10 determines whether the detection result indicates that a head has been detected (step S304), and if it determines that the detection result does not indicate that a head has been detected (S304: NO), it ends the processing, and when the next image data is acquired, it starts from step S301.

[0122] If it is determined that the detection result indicates that a head has been detected (S304: YES), the processing unit 10 determines, based on the detection result acquired in step S301, the devices to be executed for each of the processes using the head region extraction model M12, the head direction discrimination model M13, the age discrimination model M14, and the mask recognition model M15 (step S305). In step S305, the processing unit 10 makes the determination based on, for example, whether the number of detected heads is equal to or greater than a predetermined number. In step S305, the processing unit 10 makes the determination based on the number and specifications of other edge devices 1 included in the design data. For example, if the number of heads is two or less, the processing unit 10 determines that the second edge device 1 should execute two processes: the process using the head region extraction model M12 and the process using the head direction discrimination model M13. Conversely, if the number of heads is three or more, it is estimated that the processing load of the process using the head region extraction model M12 and the process using the head direction discrimination model M13 will be heavy, so it is decided that the process will be executed by the second edge device 1, and the process using the head direction discrimination model M13 will be executed by the third edge device 1.

[0123] Based on the determination made in step S305, the processing unit 10 provides the second edge device 1 that executes each process with instructions for the process to be executed, as well as data such as the necessary image data and extracted features (step S306), and terminates processing of the image data acquired in S301.

[0124] In response to the instructions given in step S306, the target second edge device 1 reads the specified model from the learning model group M1 stored in the memory unit 11, executes the process, stores the detection result or recognition result obtained as a result of the process in the memory unit 11, and transmits data intended for the server 2 to the server 2.

[0125] Fig. 16 is an explanatory diagram of process allocation in the second embodiment. A in Fig. 16 shows the allocation determined when the processing load by the head region extraction model M12, head direction discrimination model M13, age discrimination model M14, and mask recognition model M15 is estimated to be relatively light, and B in Fig. 16 shows the allocation determined when the processing load by the head region extraction model M12, head direction discrimination model M13, age discrimination model M14, and mask recognition model M15 is estimated to be heavy.

[0126] The first edge device 1 receives image data from the camera 4 and performs a fixed process using the first head detection model M11. The first edge device 1 performs the same processing for any image data, including input of image data from the camera and detection using the head detection model M11. In other words, the process performed by the first edge device 1 results in a uniform amount of calculation regardless of the image data.

[0127] After the detection process using the head detection model M11, the amount of calculation may not be constant depending on the number of heads in the image (i.e., the number of objects to be detected), etc. Therefore, the first edge device 1 appropriately selects a device to execute the subsequent process based on the detection result of the process using the head detection model M11.

[0128] 16A, for example, the first edge device 1 causes the second edge device 1 to execute a process using a head region extraction model M12 and a process using a head direction discrimination model M13 for a certain frame image. The first edge device 1 causes the third edge device 1 to execute a process using an age discrimination model 14 and a process using a mask recognition model 15 for a certain frame image.

[0129] If the first edge device 1 detects three or more heads in other frame images through a process using the head detection model M11 and predicts that the processing load on the second edge device 1, which was executing a process using the head region extraction model M12 and a process using the head direction discrimination model M13, and the processing load on the third edge device 1, which was executing a process using the age discrimination model M14 and a process using the mask recognition model M15, will become heavy, the first edge device 1 will have a new fourth edge device 1 execute the process using the head direction discrimination model M13 and a new fifth edge device 1 execute the process using the mask recognition model M15.

[0130] As described above, in the image analysis system 100 of this embodiment, the multiple edge devices 1 included in the system are divided into first-type edge devices 1 (e.g., the first edge device 1 described above) that execute processes with a uniform computational load on an input image, and multiple second-type edge devices 1 (e.g., the second to fifth edge devices 1 described above) that execute processes with a non-uniform computational load on data included in the input image. The allocation of processes using multiple trained neural network models to the multiple second-type edge devices 1 is changed based on the results of the processes in the first-type edge devices 1 (e.g., the number of detected people or heads). Note that if the results of the processes in the first-type edge devices 1 impose too heavy a processing load on the multiple second-type edge devices 1 (as a whole) (i.e., if there are insufficient second-type edge devices 1), the allocation of processes to each of the multiple second-type edge devices 1 is determined by focusing on important processes (models). Furthermore, the instruction to change the process allocation to the plurality of second-type edge devices 1 may be given by a first-type edge device 1 (for example, the first edge device 1) or by the server 2.

[0131] As a result of the above, the image analysis system 100 in this embodiment can perform overall processing without placing too much load on a single device, even for a second type edge device 1 that executes models whose processing load varies depending on the input image (the number of detected objects in the input image) (for example, the above-mentioned head area extraction model M12, head direction discrimination model M13, age discrimination model M14, and mask recognition model M15).

[0132] The change in the allocation of processes to the plurality of second-type edge devices 1 is displayed on the display of each edge device 1, server 2, or client 3. This display includes the process (which may be the model name) to be executed by each edge device 1 and the operating rate (CPU load rate) of each edge device 1.

[0133] In this way, even if the number of people detected in the space to be analyzed varies depending on the time of day, the processing can be appropriately divided and carried out so that the performance of the edge device 1 is more even.

[0134] (Third embodiment) In the third embodiment, each edge device 1 changes the content of the process it is responsible for depending on the processing load. The hardware configuration of the image analysis system 100 of the third embodiment is similar to the hardware configuration of the image analysis system 100 of the first embodiment, so the same reference numerals are used for the common components and detailed description will be omitted.

[0135] In the image analysis system 100 of the third embodiment, as described in the first embodiment, the server 2 determines in advance how processes are shared among the multiple edge devices 1 according to performance. Then, each edge device 1 limits transmission and data retention according to the processing load during operation as follows:

[0136] 17 and 18 are flowcharts showing an example of a processing procedure in the edge device 1 in the third embodiment.

[0137] The processing unit 10 of the edge device 1 receives a data collection instruction from the server 2 (step S131). The processing unit 10 acquires data to be input to one of the deployed learning models M1 (step 132), and provides the acquired data to a process using one of the deployed learning models M1 for execution (step S133). In step S132, the processing unit 10 acquires an extracted image, a result of a previous process in the pipeline processing, and the like.

[0138] The processing unit 10 acquires performance data during process execution (step S134), and determines whether at least one of the processing time, processing load, power consumption, and memory consumption of the performance data is higher than a predetermined first set value, and whether the processing load is extremely heavy (step S135).

[0139] If it is determined in step S135 that the result is higher than the first set value (S135: YES), the processing unit 10 passes the data obtained by the process result in step S133 to the next process (step S136) and ends the process. In this case, since the processing load is the highest, the process of storing the process result in the memory unit 11 and the process of transmitting it to the server 2 are both omitted. The storage of the performance data may also be omitted. However, the buffering of the process result and performance data in the memory in the processing unit 10, which is a temporary storage medium (storage process with a low processing load), may be performed without being omitted. At least the process of transmitting this data to the server 2 and the storage of this data in the hard disk of the memory unit 11 are omitted.

[0140] If it is determined in step S135 that the processing time is equal to or less than the first set value (S135: NO), the processing unit 10 determines whether at least one of the processing time, processing load, power consumption, and memory consumption of the performance data is higher than a second set value that is lower than the first set value, and whether the processing load is heavy (step S137).

[0141] If it is determined in step S137 that the result is higher than the second set value (S137: YES), the processing unit 10 passes the data obtained as a result of the process in step S133 to the next process (step S138), temporarily stores the performance data and the data obtained as a result of the process in a memory in the processing unit 10, which is a temporary storage medium (step S139), and ends the process. The storage destination in step S139 may be storage on the hard disk of the storage unit 11. In this case, since the processing load is somewhat high, at least the process of transmitting the result of the process to the server 2 is omitted. This is because transmission to the server 2 can be executed later if the data is stored. Storing the performance data on the hard disk of the storage unit 11 may also be omitted.

[0142] If it is determined in step S137 that the difference is equal to or less than the second set value (S137: NO), the processing load is relatively light, so the processing unit 10 passes the data obtained as a result of the process in step S133 to the next process (step S140). In this case, the processing unit 10 temporarily stores the performance data and the data obtained as a result of the process in the memory of the processing unit 10, which is a temporary storage medium (step S141), and then stores them on the hard disk of the storage unit 11 (step S142). In this case, the processing unit 10 transmits to the server 2, among the detection results or recognition results obtained as a result of the process, data intended for the server 2 (step S143). The processing unit 10 associates the performance data acquired in step S134 with the identification data of the edge device 1 and data indicating one of the deployed learning models M1, and transmits them to the server 2 (step S144), and ends the process.

[0143] 17 and 18, the changes to the processing content are changes related to transmission to the server 2 and storage in the storage unit 11. However, it is also possible to make the resolution of the rectangular image coarser, skip some of the processing, etc.

[0144] As described above, the processing content may be changed by each edge device 1 based on its own performance, or may be changed in response to an instruction from a specific edge device 1 or the server 2. When changing in response to an instruction from the server 2, a priority may be set for processes using the learning model group M1 based on the overall load of the image analysis system 100 and statistics on the time period distribution of the processing load, and an instruction may be given from the server 2 or a specific edge device 1 not to execute processes with low priority.

[0145] The change in the content of a process according to the processing load by each edge device 1 described in the third embodiment can also be applied when the edge device 1 shown in the second embodiment executes an instructed process.

[0146] As explained in the third embodiment, by changing the processing content depending on the performance, processing can be carried out appropriately so that the performance in the edge device 1 is more uniform, even if the characteristics of the images captured in each space to be analyzed or the number of people detected differ depending on the time of day.

[0147] The change in process content according to the processing load of each edge device 1 described in the third embodiment is applied to, for example, the following system: That is, a learning dataset generation system and / or a re-learning system in which each edge device 1 receives an instruction to collect data that meets predetermined conditions from a server 2 or a device (high-performance device) with higher processing power than the edge device 1, each edge device 1 selects data that meets the predetermined conditions (e.g., selects data of people of a predetermined gender and age) based on the inference results of each model, and returns the selected data (including the image that is the source of this data (i.e., the image to be input to each model when this data is the inference result of each model)) to the server 2 or a high-performance device (hereinafter referred to as "server 2, etc.") to reflect the selected data in re-learning (fine-tuning, etc.) of each model used in the edge device 1. In this system, if there is a difference between the statistical tendency of the attributes of data collected from all of the edge devices 1 (the attributes of the images that form the basis of the collected data (the inference results of each model)) and the statistical tendency of the attributes of data collected from a certain edge device 1, the server 2 etc. changes the attributes (characteristics) of the data to be collected for the certain edge device 1 so as to balance this with the statistical tendency of all of the edge devices 1. For example, if the statistical tendency of the attributes of data collected from a certain edge device 1 clearly shows more data on males and more data on young people than the statistical tendency of the attributes of data from all of the edge devices 1, the server 2 etc. changes the data that it instructs the edge device 1 to collect to data on females and data on middle-aged and elderly people.

[0148] Furthermore, in the above system, an example has been given in which the server 2 or the like instructs a certain edge device 1 to change the attributes (characteristics) of the data it collects when there is a bias in the statistical tendency of the attributes of the data collected from the certain edge device 1. However, each edge device 1 may store statistical values ​​of the attributes of the data already transmitted to the server 2 or the like, and each edge device 1 may determine whether there is a bias in the attributes of the data already transmitted to the server 2 or the like based on the statistical values. If there is a bias, each edge device 1 may change the attributes (characteristics) of the data to be transmitted to the server 2 or the like to eliminate the bias. In this way, in order to change the attributes (characteristics) of the data to be transmitted to the server 2 or the like, each edge device 1 may prioritize the attributes of the data (including the image from which this data is based) stored on the hard disk of the storage unit 11 in step S142. That is, when each edge device 1 determines that there is a bias in the attributes of the data already transmitted to the server 2 or the like, data with attributes that have been transmitted less frequently may be preferentially stored on the hard disk of the storage unit 11. Furthermore, if the processing unit 10 of each edge device 1 continues to determine in step S137 of FIG. 18 that at least one of the performance data (processing load, etc.) is higher than the second set value (S137: YES), the processing unit 10 may update the data (including the image that is the source of this data) temporarily stored (buffered) in the memory within the processing unit 10 from the existing data to data that is more suitable for eliminating the bias in the attributes, in order to eliminate the bias in the attributes (characteristics) of the data to be sent to the server 2, etc.

[0149] The embodiments disclosed above are illustrative in all respects and are not restrictive. The scope of the present invention is defined by the claims, and includes all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0150] 100 Image Analysis System 1. Edge Devices 10 Processing section 11 Storage section 2 Server 20 Processing section 21 Memory section 210 databases 3 Client 33 Display section M1, M2 learning model group M11, M21 head detection model M12, M22 head region extraction model M13,M23 head direction discrimination model M14,M24 age discrimination model M15,M25 mask recognition model

Claims

1. The computer When image data is input, performance data is acquired for each model during execution of a process using one of a plurality of trained neural network models, each trained to output a detection result of an object appearing in the image or a recognition result of the object; Using the acquired performance data, a process allocation using the plurality of trained neural network models is determined for a plurality of edge devices. Information processing methods.

2. the computer is a server connected to the plurality of edge devices and acquiring results of processes executed by the plurality of edge devices; The server: storing input data to each trained neural network model obtained when a plurality of processes using the plurality of trained neural network models are executed in a predetermined order on at least one image data; For each combination of a plurality of processes using any of the plurality of trained neural network models and each of the plurality of edge devices, a trial process of deploying input data for each of the stored trained neural network models to any of the plurality of edge devices together with a process using the trained neural network model to which the input data is input; and acquiring, from the deployed edge device, performance data obtained when the edge device, on which a process using the trained neural network model has been deployed on a trial basis, executes the deployed process using the input data; Using the performance data obtained by the processing executed on the combination, a process allocation using the plurality of trained neural network models is determined for the plurality of edge devices. The information processing method according to claim 1 .

3. the computer is a specific edge device among the plurality of edge devices that can be communicatively connected to each other, The specific edge device storing input data to each trained neural network model obtained when a plurality of processes using the plurality of trained neural network models are executed in a predetermined order on at least one image data; For each combination of a plurality of processes using any of the plurality of trained neural network models and each of the plurality of edge devices, a trial process of deploying input data for each of the stored trained neural network models, together with a process using the trained neural network model to which the input data is input, to another of the plurality of edge devices; and acquiring, from the deployed edge device, performance data obtained when the edge device, on which a process using the trained neural network model has been deployed on a trial basis, executes the deployed process using the input data; Using the performance data obtained by the processing executed on the combination, a process allocation using the plurality of trained neural network models is determined for the plurality of edge devices. The information processing method according to claim 1 .

4. Some or all of the plurality of edge devices are connected to a camera, The computer an edge device that acquires image data from the camera acquires performance data when a process using one of the plurality of trained neural network models is executed for each model; Based on the performance data of each process, the allocation of the processes is determined. The information processing method according to claim 1 .

5. The computer acquires at least one of the processing time, processing load, power consumption, and memory consumption amount of each of the processes as the performance data. The information processing method according to any one of claims 1 to 4.

6. the plurality of edge devices executing a plurality of processes using any of the plurality of trained neural network models in a predetermined order for one image data; For each of the plurality of processes, when processing of one image data is completed, processing of the next input image data is continued. The information processing method according to any one of claims 1 to 3.

7. The computer determines the allocation of the processing loads of the edge devices based on the number of the edge devices, the processing capabilities of each of the edge devices, and the performance data, so as to minimize differences in processing loads among the edge devices. The information processing method according to any one of claims 1 to 4.

8. The computer outputs the determined process allocation results to a display unit. The information processing method according to any one of claims 1 to 4.

9. the plurality of edge devices are comprised of a first type edge device that executes a process having a uniform computational complexity on an image, and a plurality of second type edge devices that executes a process having a non-uniform computational complexity on data included in the image; The computer changes the allocation of processes using the plurality of trained neural network models to the plurality of second-type edge devices based on results of the processes in the first-type edge devices. The information processing method according to any one of claims 1 to 4.

10. Each of the plurality of edge devices The content of the process is changed based on the performance data of the process executed by the device itself. The information processing method according to any one of claims 1 to 3.

11. The process includes a storage process for storing data output from a trained neural network model corresponding to the process, and a transmission process for transmitting the data to a server connected to the server, Each of the plurality of edge devices When it is determined that the load of the process is heavy based on the performance data of the process executed by the device itself, at least one of the storage process and the transmission process is omitted, or the storage destination of the storage process is changed from a non-temporary storage medium to a temporary storage medium. The information processing method according to claim 10.

12. When image data is input, outputting a detection result of an object appearing in the image or acquiring data on the processing load of each process using a plurality of trained neural network models each trained to output a recognition result of the object; Using the acquired processing load data, a process allocation using the plurality of trained neural network models is determined for the plurality of edge devices. An information processing device including a processing unit that executes processing.

13. On the computer, When image data is input, outputting a detection result of an object appearing in the image or acquiring data on the processing load of each process using a plurality of trained neural network models each trained to output a recognition result of the object; Using the acquired processing load data, a process allocation using the plurality of trained neural network models is determined for the plurality of edge devices. A computer program that executes a process.

Images