Method for processing information, information processing apparatus, and computer program

Abstract

To provide a method for processing information, an information processing apparatus, and a computer program that enable appropriate execution of processing using a learning model on an edge device with limited resources.SOLUTION: The method for processing information causes a computer to use a plurality of submodels obtained by dividing a trained model, which is trained to output a recognition result of an object appearing in an image when image data is input, for each feature extracted within the model. For image data input in time series, the computer performs, in parallel, processes for extracting features on the basis of the submodels. Among the plurality of submodels, the execution frequency of a process based on a submodel having a larger computational load than others is changed with respect to the frame rate of the image data, and recognition processing is executed using respective features extracted by the processes based on the plurality of submodels.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing method, an information processing device, and a computer program using a trained model (hereinafter referred to as a "trained model"). [Background technology]

[0002] Image recognition systems are used that use a camera that captures the inside of a store or a specific area within its field of view to detect people and other objects in the images captured by the camera and output the recognition results of the detected objects. Such image recognition systems use a learning model that uses a neural network (hereinafter referred to as NN: Neural Network), which is trained to input target image data and output the detection and recognition results of objects in the image data.

[0003] Thanks to improvements in computing power and hardware technology, it is now possible to process images using learning models on edge devices with relatively limited computing resources, rather than on a server with abundant computing resources that collects data and processes the images. AI (Artificial Intelligence) cameras are also being used, which are equipped with learning models and output image data along with the recognition results. Processing using learning models on edge devices makes it possible to maintain functionality even if there are problems with communication with the server.

[0004] Patent Document 1 also discloses that each edge device performs a different process. For example, one edge device may be configured to process images obtained from a video camera, another edge device may be configured to perform processing using a thermostat, another edge device may be configured to control an automated teller machine, and another edge device may be configured to recognize items placed in a cart. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2020-129390 Summary of the Invention [Problem to be solved by the invention]

[0006] Image recognition systems that require real-time recognition, such as displaying advertisements based on the attributes of customers in images captured by cameras installed in stores, are required to perform processing faster and with higher accuracy.

[0007] Even with improvements in the hardware performance of edge devices, resources are limited, so when performing computationally intensive processing on all frame images of the target video, real-time processing may not be possible in some cases. Also, in a configuration where processing is performed in parallel for each function in order to speed up processing, if the timing at which each distributed process finishes differs, inconsistencies may occur when integrating the processing results.

[0008] The present invention aims to provide an information processing method, an information processing device, and a computer program that allow an edge device with limited resources to appropriately perform processing using a learning model. [Means for solving the problem]

[0009] In an information processing method according to one embodiment of the present disclosure, a computer uses a trained model that is trained to output recognition results for objects appearing in an image when image data is input, and divides the trained model into multiple sub-models based on features extracted within the model. The computer then performs a process of extracting features based on each of the multiple sub-models for image data that is input in a time series. The computer changes the execution frequency of a process based on a sub-model that requires more computation than the others, and performs recognition processing using each of the features extracted by the process based on the multiple sub-models.

[0010] In an information processing method according to an embodiment of the present disclosure, a learning model is divided into submodels for processing, thereby achieving high speed. Among processes based on the divided submodels, processes based on submodels with a larger amount of computation can be changed in frequency, thereby preventing the processing of the process from becoming stalled.

[0011] In an information processing method according to one embodiment of the present disclosure, the trained model is divided into a plurality of sub-models corresponding to a backbone portion that extracts features from image data, and sub-models corresponding to head portions that are trained to output recognition results for objects appearing in the image data when the features of the image data extracted by each of the plurality of sub-models are input, and the computer may perform recognition processing based on the sub-models corresponding to the head portions using at least some of the features extracted by the plurality of sub-models corresponding to the backbone portion, including a sub-model that requires a greater amount of calculation than the others.

[0012] In an information processing method according to one embodiment of the present disclosure, when a learning model is divided into submodels for processing, if the frequency of a process based on a submodel that requires a large amount of calculation is changed, that process may not be executed. Therefore, the results of that process are retained, and recognition processing is performed using all features.

[0013] In an information processing method according to an embodiment of the present disclosure, the trained model is divided into a plurality of sub-models corresponding to a backbone portion that extracts feature amounts from image data, and sub-models corresponding to head portions that are trained to output a recognition result of an object appearing in an image of the image data when feature amounts of the image data extracted by each of the plurality of sub-models are input, and the computer selects, from the sub-models corresponding to the head portions, either a model that uses the feature amounts extracted by the sub-models with a larger amount of calculation than the others, or a model that does not use the feature amounts extracted by the sub-models with a larger amount of calculation than the others; The recognition process may be performed based on the selected sub-model.

[0014] In an information processing method according to one embodiment of the present disclosure, when a learning model is divided into submodels for processing, if the frequency of a process based on a submodel that requires a large amount of calculation is changed, that process may not be executed. Therefore, it is possible to switch between performing recognition processing without using the features extracted by that process and performing recognition processing using all features.

[0015] In the information processing method according to an embodiment of the present disclosure, the computer may execute a process based on a sub-model that requires more computation than other sub-models at a predetermined rate relative to the frame rate of the image data.

[0016] In an information processing method according to an embodiment of the present disclosure, processes based on sub-models with a larger amount of calculation than others are thinned out and executed at a rate of once every few times the frame rate, thereby preventing the processing of the sub-models with a larger amount of calculation than others from becoming stuck.

[0017] In the information processing method according to an embodiment of the present disclosure, the computer may change the execution frequency of a process based on a sub-model having a larger amount of calculation than the other sub-models in accordance with a load on the computer.

[0018] In an information processing method according to an embodiment of the present disclosure, the frequency of a process based on a submodel with a larger amount of calculation is changed depending on whether the actual load is heavy or not. By omitting the process only when the load is heavy, it is possible to maintain as high accuracy as possible.

[0019] In the information processing method according to the embodiment of the present disclosure, the computer may use different resolutions of image data input to the plurality of sub-models for each sub-model.

[0020] In the information processing method according to an embodiment of the present disclosure, the resolution of image data input to a plurality of sub-models is differentiated for each sub-model, thereby further reducing the amount of calculation and the processing load.

[0021] In an information processing method according to one embodiment of the present disclosure, the computer determines whether the time difference between the target image data and the previous image data among the image data input in chronological order is greater than or equal to a predetermined amount, and if it is determined that the time difference is greater than or equal to the predetermined amount, executes a process based on a submodel in which the amount of calculation is greater than others, and if it is determined that the time difference is less than the predetermined amount, executes a process based on another submodel in which the amount of calculation is replaced by a submodel in which the amount of calculation is greater than others.

[0022] In an information processing method according to an embodiment of the present disclosure, when there is a large difference from the previous frame image in the time series, that is, when it is better to perform a process in a sub-model with a larger amount of calculation than others, the process is selected and executed, thereby making it possible to improve the efficiency of calculations and to balance a reduction in processing load with high accuracy.

[0023] In an information processing method according to one embodiment of the present disclosure, the computer may determine whether the time difference between the target image data and the previous image data among the image data input in chronological order is equal to or greater than a predetermined amount, and if it is determined that the time difference is equal to or greater than the predetermined amount, increase the execution frequency of the process based on the submodel with a larger amount of calculation than others, and if it is determined that the time difference is less than the predetermined amount, decrease the execution frequency of the process based on the submodel with a larger amount of calculation than others.

[0024] In an information processing method according to an embodiment of the present disclosure, the frequency of a process is increased during periods of time when there are large differences from the previous frame image in time series, i.e., when it is determined that a process in a sub-model with a larger amount of calculation is better to be performed, thereby improving the efficiency of calculations and achieving a balance between reduced processing load and high accuracy.

[0025] In one embodiment of the information processing method of the present disclosure, processes based on the multiple submodels are distributed to different computers, and a computer that executes a process based on a submodel with a larger amount of calculation than the others outputs features extracted by the submodel to another computer each time the process is executed, and the other computer uses a buffer memory that updates the features each time the features are output, and executes recognition processing using the features extracted by the processes based on the other submodels and the features stored in the buffer memory.

[0026] In an information processing method according to an embodiment of the present disclosure, not only is a learning model divided into submodels for processing, but the hardware for executing processes based on the submodels may also be distributed to different computers. This allows for efficient use of the computing resources of edge devices with limited resources. Feature values ​​obtained as a result of processes executed by other computers are buffered, and the latest buffered extraction results are used to share the processing and execute recognition processing.

[0027] In an information processing method according to an embodiment of the present disclosure, processes based on the plurality of submodels are distributed to different computers, and a computer that executes a process based on a submodel requiring a larger amount of calculation than the others outputs, to the other computer, each time the process is executed, features extracted by the submodel in association with identification data that identifies the timing at which image data that is the subject of the features was captured, and the other computer uses a buffer memory that updates the features and identification data each time the features and the identification data of the image are output, compares the identification data that identifies the timing at which image data that is the subject of the features extracted by the process based on the other submodel with the identification data stored in the buffer memory, and determines whether the time difference between the image data is within a predetermined range. Only when it is determined that the time difference is within the predetermined range, may the other computer execute a recognition process using the features extracted by the process based on the other submodel and the features stored in the buffer memory.

[0028] In an information processing method according to an embodiment of the present disclosure, not only is a learning model divided into submodels for processing, but also, when the hardware that executes processes based on the submodels is distributed to different devices, feature values ​​obtained as a result of processes executed by other computers may be buffered and used. In one embodiment, if feature values ​​output and buffered from other computers are used for processing frame images with different timing, using those feature values ​​to execute recognition processing will result in a decrease in accuracy. This can be avoided by adding a determination as to whether the time difference is within a predetermined range.

[0029] In an information processing method according to an embodiment of the present disclosure, if the other computer determines that the time difference is outside a predetermined range, the other computer may reset the buffer memory and the process based on the submodel with a larger amount of calculation than the other submodels, and resume the processes based on the multiple submodels.

[0030] In the information processing method according to an embodiment of the present disclosure, when determining whether the time difference is within a predetermined range, if the time difference is outside the predetermined range, a process based on a submodel with a larger amount of calculation than the others is stuck or has stopped due to a fault, making it difficult to maintain high accuracy even if the process continues. Therefore, in such cases, a reset is performed to enable the process to be restarted, thereby maintaining high accuracy.

[0031] In an information processing method according to an embodiment of the present disclosure, when the computer uses a small amount of learning data to re-learn the plurality of sub-models to adapt them to the usage environment, the computer may re-learn only the sub-models that require a relatively small amount of calculations among the plurality of sub-models.

[0032] In an information processing method according to an embodiment of the present disclosure, submodels that require more computation than others have a correspondingly larger number of parameters, and therefore require a large amount of training before they can extract features with high accuracy. Therefore, when retraining multiple submodels to adapt them to a usage environment using a small amount of training data, only the submodels that require a relatively small amount of computation are retrained. This allows the submodels that require more computation than others to maintain their original trained state, thereby maintaining their generalization performance. Furthermore, the submodels that require a relatively small amount of computation can be completed, thereby improving their adaptability to the usage environment.

[0033] In an information processing method according to one embodiment of the present disclosure, the computer executes both a process based on a submodel of the plurality of submodels that has a greater amount of calculation than the others, and a process based on a submodel that has a relatively smaller amount of calculation, determines whether the difference between the output results of both executed processes is within a predetermined range, and if it determines that the difference is outside the predetermined range, stores the target image data as learning data, and performs learning of the submodel that has a relatively smaller amount of calculation based on the stored learning data.

[0034] In an information processing method according to an embodiment of the present disclosure, when only re-learning a submodel with a relatively low computational load using a small amount of training data is performed, it is more efficient to select data that is effective for training because storage resources for accumulating image data constructed for the analysis target and captured on-site are limited. In one embodiment, data in which the output results of a submodel with a lower computational load during training differ from the output results of an alternative submodel with a higher computational load are selected and used for training. This allows for more appropriate adaptation of detailed training using image data obtained from the analysis target, enabling high-speed, high-accuracy recognition processing to be performed on an edge device with limited resources.

[0035] An information processing device according to one embodiment of the present disclosure includes a processing unit that uses a trained model that is trained to output a recognition result of an object appearing in an image when image data is input, and divides the trained model into multiple sub-models based on each feature extracted within the model, performs a process of extracting features based on each of the multiple sub-models for image data that is input in a time series, changes the execution frequency of a process based on a sub-model among the multiple sub-models that requires more computation than the others, and performs recognition processing using each feature extracted by the process based on the multiple sub-models.

[0036] A computer program according to an embodiment of the present disclosure causes a computer to execute a process of extracting features based on each of the plurality of sub-models for image data input in a time series, using a trained model that is trained to output a recognition result of an object appearing in the image when the image data is input, by changing the execution frequency of a process based on a sub-model that requires more computation than the others, and by executing a process of performing recognition processing using each of the features extracted by the process based on the plurality of sub-models. [Effects of the Invention]

[0037] According to the present disclosure, when a learning model is divided into submodels for processing, the frequency of processes using submodels with a large amount of computation is reduced while maintaining the accuracy of the recognition process. This makes it possible to avoid stagnation of processes using submodels with a large amount of computation among the divided processes, and enables high-speed recognition processing using the learning model even on edge devices with limited resources. [Brief explanation of the drawings]

[0038] [Figure 1] FIG. 1 is a schematic diagram of an image analysis system. [Figure 2] FIG. 2 is a block diagram showing a configuration of an edge device. [Figure 3] FIG. 2 is a block diagram showing the configuration of a server. [Figure 4] FIG. 1 is a schematic diagram of a learning model. [Figure 5] FIG. 3 is an explanatory diagram of a process executed by the edge device of the first embodiment. [Figure 6] FIG. 10 is an explanatory diagram of processing executed by a plurality of edge devices in Modification 1. [Figure 7] 10 is a flowchart illustrating an example of a processing procedure executed by an edge device according to the second embodiment. [Figure 8] FIG. 10 is an explanatory diagram of a process executed by an edge device according to the second embodiment. [Figure 9] FIG. 11 is an explanatory diagram of a process executed by an edge device according to the third embodiment. [Figure 10] 13 is a flowchart illustrating an example of a processing procedure executed by an edge device according to the fourth embodiment. [Figure 11] FIG. 10 is an explanatory diagram of a process executed by an edge device according to the fourth embodiment. [Figure 12] 10 is a flowchart illustrating an example of a processing procedure executed by an edge device in Modification 2. [Figure 13] FIG. 13 is an explanatory diagram of processing executed by a plurality of edge devices according to the fifth embodiment. [Figure 14] FIG. 20 is an explanatory diagram of processing executed by a plurality of edge devices in the sixth embodiment. [Figure 15] FIG. 20 is an explanatory diagram of a process executed by an edge device according to the seventh embodiment. [Figure 16] 13 is a flowchart illustrating an example of a learning process procedure performed by an edge device according to the seventh embodiment. [Figure 17] 13 is a flowchart illustrating an example of a learning process procedure performed by an edge device according to the seventh embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0039] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present disclosure will be specifically described with reference to the drawings illustrating embodiments thereof. In the following embodiments, an image analysis system to which the information processing method of the present disclosure is applied will be described.

[0040] (First embodiment) 1 is a schematic diagram of an image analysis system 100. The image analysis system 100 in the first embodiment includes one or more cameras 4 installed in a space to be analyzed, such as a store, multiple edge devices 1 connected to the cameras 4, a server 2 that can be connected to the edge device 1 for communication, and a client 3 that can be connected to the server 2.

[0041] One or more cameras 4 and multiple edge devices 1 can be connected to each other via a wireless or wired local network LN. The local network LN may be a wired LAN or a wireless network such as WiFi or Bluetooth (registered trademark).

[0042] One or more cameras 4 use image elements that are responsive to visible light and / or near-infrared light and output image data. The cameras 4 output image data in time series at a rate of several fps to several tens of fps. The cameras 4 are installed so as to look down from the top of the space to be analyzed, such as on the ceiling or shelves. The cameras 4 may be ceiling-mounted cameras that can cover the entire space in a 360-degree field of view. The cameras 4 sequentially transmit the image data to the edge device 1 via the local network LN.

[0043] Some or all of the multiple edge devices 1 can be connected to a server 2 via a network N. The network N is a wired or wireless communication network that may include a public communication network, a dedicated line, or a carrier network. The client 3 can be connected to the server 2 via the network N.

[0044] In the first embodiment, multiple edge devices 1 extract features from image data acquired by cameras 4, recognize objects (specifically, people present in the space being analyzed) captured in the image data based on the features, and output the recognition results using a neural network (NN)-based learning model. A learning model using a neural network for image recognition can be separated into a "backbone" part that extracts features from input image data and a "head" part that performs recognition processing (recognition, discrimination, region segmentation, etc.) based on the extracted features. Furthermore, in the image analysis system 100 of the first embodiment, the "backbone" part of the learning model is divided into sub-models based on the extracted features. Processing based on the sub-models is performed in parallel to increase speed and distribute the load, achieving highly accurate recognition in the space being analyzed.

[0045] Hereinafter, in the first embodiment, the edge device 1 determines the attributes of the object shown in the image by providing the sub-model of the “Head” part with features derived from the image data based on the sub-model of the assigned “Backbone” part, stores the text indicating the attributes in association with identification data that enables the space to be analyzed to be identified, and transmits it to the server 2 as necessary.

[0046] The server 2 records data sent from the process in the "Head" part of the edge device 1 in a database 210 and can perform analysis processing such as statistical processing of the attributes of detected objects for each space being analyzed. The results of the analysis processing in the server 2 can be viewed from the client 3 for each space being analyzed. The manager of the space being analyzed can use the client 3 to visually check the attributes of objects detected within the space and the statistical results. More specifically, by installing the image analysis system 100 in a store, the store manager can check the aggregation results, such as the age group and gender of customers visiting the store, from the client 3.

[0047] When image data is input, each edge device 1 performs calculations on a sub-model that has been trained to output features for that image. The amount of calculations can be large depending on the features required. For image data output from the camera 4 at a rate of several fps to several tens of fps, if calculations are performed on all "Backbone" sub-models for all frame images, the "Head" process may not be able to complete in time. Therefore, in the image analysis system 100 of the first embodiment, calculations on some of the multiple "Backbone" sub-models do not need to be performed on all frame images.

[0048] The image analysis system 100 will be described in detail below.

[0049] 2 is a block diagram showing the configuration of the edge device 1. The edge device 1 is a box-shaped device that can be installed together with cameras 4 in a space to be analyzed, such as a store. The edge device 1 includes a processing unit 10, a storage unit 11, a first communication unit 12, and a second communication unit 13.

[0050] The processing unit 10 includes one or more processors such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), etc. The processing unit 10 includes a memory that is a temporary storage medium such as a static random access memory (SRAM) or a dynamic random access memory (DRAM). The processing unit 10 includes a timer and can acquire time information at each point in time from data from the timer. The processing unit 10 may be configured as a single piece of hardware (SoC: System On a Chip) that integrates a processor, a memory, a storage unit 11, a first communication unit 12, and a second communication unit 13. The specifications of the processing unit 10 may be the same or different between edge devices 1.

[0051] The processing unit 10 reads the information processing program P1 stored in the storage unit 11 into the memory and executes it, thereby causing the processor to execute various processes described below and function as the edge device 1 of the present disclosure.

[0052] The storage unit 11 is a relatively large-capacity non-transitory storage medium such as a hard disk, a flash memory, etc. A part of the storage unit 11 may be removable.

[0053] The storage unit 11 stores a program (program product) required for the processing unit 10 to execute processing, and reference setting data. The setting data includes an identifier for the edge device 1, identification data for the connected camera 4, etc. The program product includes an OS (Operating System) program, an information processing program P1 that runs on the OS, and a learning model M1 that corresponds to the submodel to be used. The learning model M1 will be described in detail later.

[0054] The information processing program P1 stored in the storage unit 11 may be an information processing program P9 stored in a computer-readable non-transitory storage medium 9 that is read by the processing unit 10 and stored in the storage unit 11, or may be a program that is pre-stored at the time of shipment. The information processing program P1 may be downloaded by the processing unit 10 from the server 2 or another download server via the second communication unit 13 and stored in the storage unit 11.

[0055] The learning model M1 stored in the memory unit 11 may be selected by the processing unit 10 from the sub-model group M9 stored in the non-temporary storage medium 9 and stored in the memory unit 11, or may be selected and stored in advance at the time of shipment. The learning model M1 stored in the memory unit 11 or a part thereof may be downloaded by the processing unit 10 from the server 2 or from another download server via the second communication unit 13 and stored in the memory unit 11.

[0056] The storage unit 11 may store the feature amounts extracted by the calculation of the "Backbone" portion by associating them with time information or data identifying the target image data. The storage unit 11 stores the recognition result (e.g., attribute) data obtained by the calculation of the "Head" portion by associating it with time information or data identifying the target image data.

[0057] The first communication unit 12 is a communication device that realizes communication via the local network LN. The first communication unit 12 may be a LAN network card or a CAN communication device. The first communication unit 12 may be a wireless communication device compatible with wireless networks such as Wi-Fi or Bluetooth (registered trademark). The first communication unit 12 may include multiple communication devices compatible with various types of cameras 4. The first communication unit 12 may include an interface such as a USB (Universal Serial Bus) connected to the camera 4. The first communication unit 12 can be replaced by an interface connected to the camera 4 via a coaxial cable or another serial bus. The processing unit 10 acquires image data from the camera 4 via the local network LN through the first communication unit 12. The processing unit 10 can transmit and receive data to and from other edge devices 1 through the first communication unit 12. The first communication unit 12 may be the same device as the second communication unit 13.

[0058] The second communication unit 13 is a communication device that realizes communication via an external network N. The second communication unit 13 may be a network card for a wired LAN, a communication device that realizes carrier communication via a carrier network, or a communication device that supports a wireless network such as WiFi or Bluetooth (registered trademark). The second communication unit 13 may support encrypted communication with the server 2 using SSL or the like. The second communication unit 13 may be an interface for realizing connection with the server 2 via a dedicated line. The second communication unit 13 may be provided only in the edge device 1H that outputs the recognition result to the server 2 based on the submodel of the "Head" part.

[0059] 3 is a block diagram showing the configuration of the server 2. The server 2 may be configured as a single server computer, or may be configured to distribute processing among multiple server computers. The server 2 includes a processing unit 20, a storage unit 21, and a communication unit 22.

[0060] The processing unit 20 includes one or more processors such as a CPU, an MPU, a GPU, etc. The processing unit 20 includes a memory that is a temporary storage medium such as an SRAM or a DRAM.

[0061] The storage unit 21 is a relatively large-capacity non-temporary storage medium such as a hard disk, a flash memory, etc. The storage unit 21 stores a program (program product) required for the processing unit 20 to execute processing.

[0062] The program product stored in the storage unit 21 includes a server program P2. The server program P2 includes a module that functions as a web server, and can output the results of processing in the server 2 to the client 3 via a web page. The storage unit 21 stores login account data for distinguishing users who use the client 3, making it possible to identify whether the logged-in user is an account with authority to operate the server 2, or an account of a customer who will view the processing results of the server 2. The program for outputting data to the client 3 included in the server program P2 is not limited to a web server program, and may be a program compatible with other protocols as long as it can output data to a screen displayed on the client 3.

[0063] The server program P2 stored in the storage unit 21 may be a server program P8 stored in a computer-readable non-transitory storage medium 8 that is read by the processing unit 20 and stored in the storage unit 21, or may be a program that is pre-stored at the time of shipment. The server program P2 stored in the storage unit 21 may be a program that the processing unit 20 downloads from a download server via the communication unit 22 and stores in the storage unit 21.

[0064] A database 210 is constructed in the storage unit 21, which stores space definition data and administrator data for the analysis target space in association with identification data of the analysis target space. The storage unit 21 may store the accounts of clients 3 in association with each analysis target space in the database 210, so that the analysis results that can be referenced by the clients 3 can be narrowed down. The storage unit 21 stores data such as the number of edge devices 1 and the number and specifications of cameras 4 arranged in the space in association with the identification data of the analysis target space. Specifically, the storage unit 21 stores layout data of the store's floor and shelves, installation positions and angle of view data of cameras 4 corresponding to the floor, and identification data of the cameras 4 in association with the store's identification data in the database 210.

[0065] The communication unit 22 is a communication device that realizes a communication connection with the client 3 via the network N. The communication unit 22 may be a communication device that realizes a communication connection with the client 3 via a dedicated line.

[0066] FIG. 4 is a schematic diagram of the learning model M1. FIG. 4 shows the function of the learning model M1 stored in the edge device 1. As shown in FIG. 4, the learning model M1 is a sub-model that each outputs a different feature value for input image data, or a sub-model that outputs a recognition result using the output feature value, and as a whole constitutes a learning model M0 that is trained to output a recognition result for an input image. In the first embodiment, as shown in FIG. 4, the learning model M0 includes a learning model M1 (sub-model) that extracts feature values ​​α, β, and γ, and a learning model M1 (sub-model) that uses the feature values ​​α, β, and γ in calculations to output a result of recognizing an object shown in the image. The learning models M1, which are sub-models, are combined and have been trained in advance by the server 2 or another device using training data, which is a set of image data and recognition results, so that the entire learning model M0 can perform accurate recognition.

[0067] In the first embodiment, a learning model M1 divided into multiple sub-models as shown in Fig. 4 is stored and used in one edge device 1. In the first embodiment, an example will be described in which processing based on multiple learning models (sub-models) M1 is performed in parallel in one edge device 1. However, multiple learning models (sub-models) M1 may be distributed to multiple edge devices 1 and executed (see Modification 1).

[0068] FIG. 5 is an explanatory diagram of the processing executed by the edge device 1 of the first embodiment. In the first embodiment, the edge device 1 executes three processes corresponding to the "Backbone" portion in parallel. The three "Backbone" processes each output feature quantities α, β, and γ. The edge device 1 also executes processing based on the learning model M1 of the "Head" portion using the output feature quantities α, β, and γ, recognizing objects appearing in the image of the input image data and outputting their attributes. In the image analysis system 100 of the first embodiment, as shown in FIG. 5, the processing procedures of the "Backbone" portion for one frame image are executed simultaneously in parallel. This enables the processing of recognizing objects appearing in the image to be accelerated.

[0069] In the example shown in FIG. 5, the processing unit 10 of the edge device 1 acquires image data sent from the camera 4 to the local network LN, and provides the image data to each of the three processes corresponding to the "Backbone" portion.

[0070] In a process based on a learning model M1 (submodel) that outputs a feature amount α, a processing unit 10 performs preprocessing such as filtering for the learning model M1 of the feature amount α. The processing unit 10 provides the preprocessed image data to the learning model M1, extracts the feature amount α, and provides the feature amount α to the learning model M1 of the "Head" portion. The process based on the learning model M1 that outputs the feature amount α repeats preprocessing and the process of extracting the feature amount α using the learning model M1 for all frames of input image data.

[0071] In a process based on a learning model M1 (submodel) that outputs a feature amount β, a processing unit 10 performs preprocessing such as filtering for the learning model M1 of the feature amount β. The processing unit 10 provides the preprocessed image data to the learning model M1, extracts the feature amount β, and provides the feature amount β to the learning model M1 of the "Head" portion. The process based on the learning model M1 that outputs the feature amount β repeats preprocessing and the process of extracting the feature amount β using the learning model M1 for all input image data.

[0072] The computational complexity of the process based on the learning model M1 (sub-model) that outputs the feature γ is greater than the computational complexity of the processes of other learning models M1. In this process based on the learning model M1 that outputs the feature γ, the processing unit 10 executes preprocessing, such as filtering, on the frame image once every three times when image data is input. That is, the processing unit 10 executes the process based on the learning model M1 that outputs the feature γ by thinning it out at a predetermined rate relative to the frame rate. The processing unit 10 provides the preprocessed image data to the learning model M1 stored in the storage unit 11 of the device, extracts the feature γ, and provides it to the learning model M1 in the "Head" section. The process based on the learning model M1 (sub-model) that outputs the feature γ is not executed two out of three times. Specifically, the processing unit 10 executes a process of incrementing the count each time image data is output from the camera 4, and each time the count reaches three, it resets the count to zero after performing preprocessing, a process of extracting the feature γ, and a process of outputting the feature γ. The frequency is not limited to once in three times, but may be once in two times, once in four times, or even less frequently than that. This makes it possible to avoid a backlog in the calculation for outputting the feature value γ.

[0073] In the process of the "Head" part, the processing unit 10 acquires the features α and β for all frame images from the "Backbone" process, and waits for the feature γ to be output by calculation once every three times before executing the object recognition process. When the feature γ is acquired, the processing unit 10 stores it. The processing unit 10 performs object recognition calculations using the features α, β, and γ using the learning model M1, and outputs the attributes of the recognized object to the server 2 via the second communication unit 13. Since the output of the feature γ is not executed two out of three times, the processing unit 10 performs calculations using the features α, β, and γ, including the stored feature γ. This prevents delays in processing by the edge device 1 due to a backlog in the calculation of the feature γ.

[0074] In this way, in the image analysis system 100 of the first embodiment, by omitting the process based on the "Backbone" part, which requires a large amount of calculation, at a preset frequency, such as executing it once every three times, the amount of calculation can be reduced, the processing load can be lightened, and high-speed processing can be maintained.

[0075] (Variation 1) In variant example 1, multiple sub-models corresponding to the "Backbone" portion that extract different features and a sub-model of the "Head" portion that integrates the features extracted from each sub-model are distributed to multiple edge devices 1, and processing is performed in parallel on the multiple edge devices 1 to increase speed and distribute the load, thereby achieving highly accurate recognition in the space to be analyzed.

[0076] FIG. 6 is an explanatory diagram of processing executed by multiple edge devices 1 in Modification 1. In Modification 1, the edge devices are divided into an edge device 1 that executes a process based on a submodel of the "Backbone" portion that outputs a feature amount α, an edge device 1 that executes a process based on a submodel of the "Backbone" portion that outputs a feature amount β, and an edge device 1 that executes a process based on a submodel of the "Backbone" portion that outputs a feature amount γ. Furthermore, processing may be distributed among four edge devices 1, including an edge device 1 that executes a process based on a submodel of the "Head" portion. By distributing the submodels to different edge devices 1, the processing load on each edge device 1 can be reduced, enabling high-speed processing.

[0077] In Modification 1, each of three edge devices 1 executing a process in the "Backbone" portion transmits feature quantities α, β, and γ obtained by calculation of the learning model M1 (sub-model) in the "Backbone" portion to an edge device 1 executing a process based on a sub-model in the "Head" portion via the first communication unit 12. The edge device 1 executing a process using the sub-model in the "Backbone" portion that outputs feature quantity γ executes the process at a predetermined frequency, for example, once every three times, as shown in the first embodiment.

[0078] When the processing unit 10 of the edge device 1 that executes a process based on the submodel of the "Head" portion is able to acquire the features α, β, and γ, it temporarily stores the feature γ and then provides it to the learning model M1 of the "Head" portion to perform object recognition calculations. In two out of three cases, when the process of the "Backbone" portion that outputs the feature γ is omitted, the processing unit 10 of this edge device 1 performs object recognition calculations using the stored feature γ. In this way, the amount of calculations can be reduced, the processing load can be lightened, and high-speed processing can be maintained, while also preventing delays in object recognition calculations due to backlogs in the processing of the edge device 1 that executes the process of the "Backbone" portion, which requires a large amount of calculations.

[0079] (Second embodiment) In the first embodiment, processing is omitted at a preset frequency, but in the second embodiment, the frequency is dynamically changed depending on the magnitude of the load. The configuration of image analysis system 100 in the second embodiment is similar to that of image analysis system 100 in the first embodiment, so the same reference numerals are used for common configurations and detailed description will be omitted.

[0080] 7 is a flowchart showing an example of a processing procedure executed by the edge device 1 of the second embodiment. The processing procedure shown in FIG. 7 explains the process of the "Backbone" portion, which has a larger amount of calculation than the other portions.

[0081] When image data is output from the camera 4, the processing unit 10 acquires it (step S101) and counts the number of times the image data has been output (step S102). The processing unit 10 determines whether or not to execute the following process based on a numerical value obtained by calculating the number of times and a load value that changes depending on the load (step S103). In step S103, the processing unit 10 determines whether or not the remainder when the number of times the image data has been output is divided by the load value is a specific value (for example, set to "1" to include the first time), and determines to execute the process if it is determined to be the specific value. Each time image data is output, the number of times is incremented, and if the number of times exceeds the load value, the number is reset. Alternatively, the processing unit 10 may determine whether the number of times is equal to or greater than the load value, and if it is determined to be equal to or greater than the load value, determine to execute the following process.

[0082] If it is determined in step S103 that the following processing is to be executed (S103: YES), the processing unit 10 performs preprocessing such as filtering on the output image data (step S104). The processing unit 10 provides the preprocessed image data to a process based on the learning model M1 stored in the storage unit 11 (step S105) and extracts features (step S106). The processing unit 10 outputs the features to the process of the "Head" part (step S107), which then executes object recognition processing (step S108), and transmits the recognition result to the server 2 via the second communication unit 13 (step S109).

[0083] The processing unit 10 calculates the processing load of the processor due to the calculations from steps S104 to S107 (step S110), determines a load value depending on whether the processing load is higher than a predetermined value (step S111), and ends the processing. In step S111, the processing unit 10 determines the load value from one of two numerical values. If the processor usage rate as the processing load is higher than a predetermined rate, the processing unit 10 determines the load value to be the higher numerical value, and if the usage rate is equal to or lower than the predetermined rate, the processing unit 10 determines the load value to be the lower numerical value.

[0084] In step S110, for example, the processing unit 10 sets the load value to either "3" or "4," and if the load on the processing unit 10 is heavy and the processor usage rate is higher than 50%, the load value is set to "4." Conversely, if the load on the processing unit 10 is light and the processor usage rate is 50% or less, the load value is set to "3." As a result, if the processing is heavy, the processing of steps S104-S109 is executed once every four times even if image data is output from the camera 4, and if the processing is light, the processing of steps S104-S109 is executed once every three times.

[0085] If it is determined in step S103 that the following processing is not to be executed (S103: NO), the processing unit 10 ends the processing.

[0086] 8 is an explanatory diagram of the processing executed by the edge device 1 of the second embodiment. In the second embodiment, as in the example shown in Fig. 5, the edge device 1 executes three processes corresponding to the "Backbone" portion in parallel, outputs the feature amounts α, β, and γ, and recognizes objects shown in the image of the image data using the feature amounts α, β, and γ, and outputs their attributes.

[0087] The process based on learning model M1 that outputs feature value α and the process based on learning model M1 that outputs feature value β are similar to the processing content described in the first embodiment and shown in Figure 5. In image analysis system 100 of the second embodiment, as shown in 8, the "backbone" process for one frame image is also distributed and processed simultaneously in parallel. This enables the speed of processing to recognize objects in images.

[0088] In the second embodiment, the edge device 1 executes the processing procedure shown in FIG. 7 for the "Backbone" process that outputs the feature quantity γ. As a result, the edge device 1 performs preprocessing, extracts the feature quantity γ, and outputs the feature quantity γ only when the remainder when the number of times is divided by the load value is a specific value (for example, zero, i.e., no remainder). If the load value is set to "4" when the load is heavy and "3" when the load is light, the edge device 1 executes the processing once every four times in the former case and once every three times in the latter case. As a result, even in the second embodiment, it is possible to avoid backlogs in the process that outputs the feature quantity γ.

[0089] In the process based on the learning model M1 in the "Head" part in the second embodiment, the processing unit 10 omits the process in the "Backbone" part that outputs the feature value γ when the load is heavy, so if the feature values ​​α, β, and γ can be acquired, the feature value γ is temporarily stored. If the processing unit 10 can acquire all of the feature values ​​α, β, and γ, it provides them to the learning model M1 in the "Head" part and executes the object recognition calculation. If the processing unit 10 of this edge device 1 omits the process in the "Backbone" part that outputs the feature value γ three times out of four, or two times out of three, it provides the features α, β, and γ, including the stored feature value γ, to the learning model M1 in the "Head" part and executes the object recognition calculation.

[0090] In this way, in the second embodiment, the amount of calculation can be reduced depending on the processing load at each point in time, thereby reducing the processing load and maintaining high-speed processing, and it is possible to avoid delays in object recognition calculations due to stagnation in processing of the edge device 1 that executes the ``Backbone'' part of the process, which requires a large amount of calculations.

[0091] (Third embodiment) In the third embodiment, the processing load of the "Backbone" process is reduced by changing the resolution of the image provided to the learning model M1 of the "Backbone" part. The configuration of the image analysis system 100 in the third embodiment is similar to that of the image analysis system 100 in the first embodiment, and therefore the same reference numerals are used for the common configuration, and detailed description thereof will be omitted.

[0092] 9 is an explanatory diagram of the processing executed by the edge device 1 of the third embodiment. In the third embodiment, as in the example shown in FIG. 5, the edge device 1 executes three processes corresponding to the "Backbone" portion in parallel, and outputs feature quantities α, β, and γ, respectively. Then, the edge device 1 recognizes objects shown in the image data using the feature quantities α, β, and γ through processing based on the learning model M1 of the "Head" portion, and outputs attributes.

[0093] In the third embodiment, the process using the learning model M1 in the third "Backbone" section that outputs the feature quantity γ also requires a large amount of calculation, so the frequency of the process is reduced, as in the first embodiment. In the example shown in FIG. 9, when image data is input, the processing unit 10 of this edge device 1 performs preprocessing such as filtering on the frame image once every three times, and provides the preprocessed image data to the learning model M1 in the "Backbone" section that outputs the feature quantity γ to extract the feature quantity γ. The process in the "Backbone" section that outputs the feature quantity γ is not executed on the image data output from the camera 4 two out of three times. The learning model M1 in the "Backbone" section that outputs the feature quantity γ is a type of (sub)model that makes it difficult to obtain accurate output results unless a high-resolution image is input.

[0094] Furthermore, in the third embodiment, in a process based on the learning model M1 (sub-model) that outputs a feature value α and a process based on the learning model M1 (sub-model) that outputs a feature value β, when the processing unit 10 acquires image data from the camera 4, it reduces the resolution in pre-processing such as filtering and then executes subsequent processing. At this time, the processing unit 10 reduces the resolution to an extent that does not reduce the accuracy of calculations using the learning model M1. The degree of resolution reduction is preferably set based on the learning model M1 or image data actually captured in the target space.

[0095] Furthermore, in the third embodiment, the processing unit 10 may set the resolution of the input image data to a low-resolution image for a process based on the learning model M1 (sub-model) that outputs the feature α and a process based on the learning model M1 (sub-model) that outputs the feature β, and may set the resolution of the input image data to a high-resolution image for a process based on the learning model M1 (sub-model) that outputs the feature γ.

[0096] In this way, in the third embodiment, it is possible to prevent the calculations of the edge device 1 from being delayed, and also to further reduce the processing load for outputting the feature amounts α and β, thereby maintaining high-speed processing.

[0097] (Fourth embodiment) In the fourth embodiment, when a time-series frame image contained in a video has a large difference from the previous frame image and the difference in feature amount is expected to be large, the system selects to execute a "Backbone" calculation, which requires a large amount of calculation. The configuration of the image analysis system 100 in the fourth embodiment is similar to that of the image analysis system 100 in the first embodiment, and therefore the same reference numerals are used for the common configuration and detailed description thereof will be omitted.

[0098] 10 is a flowchart showing an example of a processing procedure executed by the edge device 1 of the fourth embodiment. The processing procedure shown in FIG. 10 explains the process of the "Backbone" portion, which has a larger amount of calculation than the other portions.

[0099] The processing unit 10 acquires image data from the camera 4 (step S121), stores it in memory (step S122), and calculates the difference from the previous image data (step S123). The difference may be calculated by simply calculating the difference in pixel values ​​between the images, or by using a known algorithm that divides the area into background and foreground and then calculates the difference.

[0100] The processing unit 10 determines whether the difference is equal to or greater than a predetermined value (step S124). If it is determined that the difference is less than the predetermined value (S124: NO), the processing unit 10 selects a process with a small amount of calculation (step S125). The processing unit 10 performs preprocessing in the selected process on the image data acquired in step S121 (step S126). The processing unit 10 provides the preprocessed image data to the selected process (step S127) and extracts features (step S128). The processing unit 10 outputs the features to the process in the "Head" part (step S129), which then executes object recognition processing (step S130), and transmits the recognition result to the server 2 via the second communication unit 13 (step S131), thereby completing the processing for one piece of image data.

[0101] If it is determined in step S124 that the difference is equal to or greater than a predetermined value (S124: YES), the processing unit 10 selects a process for the learning model M1 that requires a large amount of calculation (step S132). The processing unit 10 performs preprocessing in the selected process on the image data acquired in step S121 (step S133). The processing unit 10 provides the preprocessed image data to the selected process (step S134) and extracts features (step S135). The processing unit 10 outputs the features to the process in the "Head" part (S129), executes steps S130-S131, and ends the process.

[0102] In step S125, the processing unit 10 reads out and executes one of the different learning models (sub-models) M1, M1, which has a small amount of calculation. When the sub-models are distributed to multiple edge devices 1 (variation 1), another edge device 1 that executes processing with a small amount of calculation may be selected. In step S132, the processing unit 10 may read out and execute a model with a large amount of calculation, or may select another edge device 1 that executes processing with a large amount of calculation.

[0103] FIG. 11 is an explanatory diagram of the processing executed by the edge device 1 of the fourth embodiment. In the fourth embodiment, the edge device 1 executes the process for outputting the feature amount α of the three processes corresponding to the “Backbone” portion for all frame images, but executes either one of the processes for outputting the feature amounts β and γ according to the processing procedure shown in FIG. 10 . Therefore, in the process corresponding to the “Head” portion of the fourth embodiment, when the time difference between the image data is large, the process for recognizing the object is executed using the feature amounts α and γ, and when the time difference between the image data is small, the process for recognizing the object is executed using the feature amounts α and β. Note that the feature amounts β and γ are the same type of feature amounts used in the same recognition process, but when an object is recognized using the feature amounts α and γ, the accuracy of the object recognition result by the “Head” is higher than when the object is recognized using the feature amounts α and β.

[0104] In the fourth embodiment, specifically, the processing unit 10 of the edge device 1 determines whether the difference (time difference) between the acquired image data and the previous image data is large (S124). If the processing unit 10 determines that the difference is less than a predetermined value, i.e., the difference is small (S124: NO), it provides the image data to a process using a learning model M1 that outputs a feature amount β (S127), and extracts the feature amount β (S128). The processing unit 10 outputs the feature amounts α and β obtained by the process to the process for the "Head" part.

[0105] If the difference is equal to or greater than a predetermined value, i.e., if the difference is determined to be large (S124: YES), the processing unit 10 provides the image data to a process that outputs the feature amount γ (S134) and extracts the feature amount γ (S135). The processing unit 10 outputs the feature amounts α and γ obtained by the process to the process for the "Head" portion.

[0106] The edge device 1 performs object recognition processing using the feature amount α, which can be acquired regardless of the magnitude of the difference, and either the feature amount β or the feature amount γ. In this way, in the fourth embodiment, the process of outputting the feature amount γ, which requires a large amount of calculation, is performed when the difference from the previous frame image is large and it is effective to output the feature amount γ. This makes it possible to avoid delays in the process of outputting the feature amount γ, which requires a large amount of calculation.

[0107] In the image analysis system 100 of the fourth embodiment, as shown in Modification 1, processing using sub-models of the "Backbone" part for one frame image may be distributed to multiple edge devices 1 and performed simultaneously in parallel. This further speeds up the processing for recognizing objects in the image.

[0108] (Variation 2) In the fourth embodiment, when the time difference between the image data is large, the process of recognizing an object is executed using the feature amounts α and γ, and when the time difference between the image data is small, the process of recognizing an object is executed using the feature amounts α and β. However, without being limited to this, when the time difference between the image data is large, the frequency of execution of the process of recognizing an object using the feature amounts α and γ may be increased, and when the time difference between the image data is small, the frequency of execution of the process of recognizing an object using the feature amounts α and β may be increased.

[0109] Fig. 12 is a flowchart showing an example of a processing procedure executed by the edge device 1 in Modification 2. Of the processing procedures shown in Fig. 12, steps common to the processing procedures shown in Fig. 10 are assigned the same step numbers, and detailed descriptions thereof will be omitted.

[0110] When the processing unit 10 acquires image data (S121), it counts the number of times the image data has been output (step S141). The processing unit 10 then stores the image data in memory (S122), calculates the difference from the previous image data (S123), and determines whether the difference is equal to or greater than a predetermined value (S124).

[0111] If it is determined that the difference is equal to or greater than the predetermined value (S124: YES), the processing unit 10 determines the frequency to be relatively large (step S142), and proceeds to step S144.

[0112] If it is determined that the difference is less than the predetermined value (S124: NO), the processing unit 10 determines the frequency to be relatively small (step S143), and proceeds to step S144.

[0113] In step S144, the processing unit 10 determines whether to execute a process of the learning model M1, which requires a large amount of calculation, based on a numerical value obtained by calculation using the number of times counted in step S141 and the frequency that changes depending on the difference (step S144). In step S144, if the frequency is set to once in two times in step S142 and once in four times in step S143, the processing unit 10 determines whether the counted number of times corresponds to the frequency.

[0114] If the processing unit 10 determines in step S144 to execute the process of the learning model M1, which requires a large amount of calculations (S144: YES), it executes the processes of steps S133-S135 and proceeds to step S129. If the processing unit 10 determines in step S144 not to execute the process of the learning model M1, which requires a large amount of calculations (S144: NO), it executes the processes of steps S126-S128 and proceeds to step S129.

[0115] As a result, referring to the explanatory diagram of the process shown in Figure 11, if it is determined that the difference is greater than or equal to a predetermined value, the processing unit 10 executes the process of outputting the feature value γ once every two times, and if it is determined that the difference is less than the predetermined value, it executes the process of outputting the feature value γ once every four times.

[0116] In image analysis system 100 of variation 2, in the time period after image data is captured that requires detailed analysis depending on the magnitude of the difference in the image data that is actually output, highly accurate processing is performed even if the amount of calculation is increased. Conversely, in image analysis system 100 of variation 2, in the time period when there is no change from the background, a process with a low amount of calculation that saves energy is performed, making it possible to maintain highly accurate recognition processing while reducing the processing load.

[0117] (Fifth embodiment) In the fifth embodiment, the processes of executing multiple learning models (sub-models) M1 corresponding to the "Backbone" portion are distributed to different edge devices 1, and the results of one of the processes are stored in a memory (buffer memory) that functions as a buffer and referenced. The configuration of the image analysis system 100 in the fifth embodiment is similar to that of the image analysis system 100 in the first embodiment, so the same reference numerals are used for the common configuration and detailed description will be omitted.

[0118] 13 is an explanatory diagram of processing executed by multiple edge devices 1 in the fifth embodiment. In the fifth embodiment, among the multiple sub-models M1 constituting the learning model M0 shown in FIG. 4, the process of the sub-model M1 in the "Backbone" part that outputs the feature value γ, which is likely to have a heavy processing load, is executed by different edge devices 1.

[0119] 13, the first edge device 1 executes processing based on a learning model M1 that outputs a feature amount α, processing based on a learning model M1 that outputs a feature amount β, and processing based on a learning model M1 of the "Head" portion that uses the feature amounts α and β as well as the feature amount γ. In the fifth embodiment, the second edge device 1 executes processing based on a learning model M1 that outputs the feature amount γ.

[0120] The processing unit 10 of the first edge device 1 acquires image data sent from the camera 4 to the local network LN and provides the image data to a process based on the learning model M1 (sub-model) that outputs a feature value α in the "Backbone" part, and a process based on the learning model M1 (sub-model) that outputs a feature value β. Each process provides the extracted features α and β to the process in the "Head" part. In the process in the "Head" part, the processing unit 10 acquires the features α and β for all frame images from the "Backbone" process, acquires the feature value γ received from the second edge device 1 and buffered in memory, and performs object recognition processing. The buffering of the feature value γ in memory is performed at any timing, receiving and storing the feature value γ each time it is sent from the second edge device 1.

[0121] The second edge device 1 also performs preprocessing such as filtering on the target image data each time image data is output from the camera 4. The processing unit 10 of this edge device 1 provides the preprocessed image data to a process based on a learning model M1 (sub-model) that outputs a feature value γ, and extracts the feature value γ. Each time the processing unit 10 obtains a feature value γ, it executes a process of writing the feature value γ to the memory of the processing unit 10 of the first edge device 1. As a result, the feature value γ extracted in the most recent process is stored in the memory of the first edge device 1, regardless of the timing of the processing in the first edge device 1 and the second edge device 1.

[0122] 13, the timing of the process based on the learning model M1 (sub-model) that outputs the feature amount α, the process based on the learning model M1 (sub-model) that outputs the feature amount β, and the process based on the learning model M1 (sub-model) that outputs the feature amount γ are not synchronized in different devices, and even if the timing is out of sync, the process in the "Head" part can be executed. The processing unit 10 of the first edge device 1 can continue the object recognition calculation in the process in the "Head" part without waiting for the output of the feature amount γ.

[0123] In the fifth embodiment, even if a process in the "Backbone" portion, which requires a large amount of calculation, is delayed, it is possible to avoid a situation in which a process in the "Head" portion cannot be executed.

[0124] (Sixth embodiment) In the sixth embodiment, similar to the fifth embodiment, the processes of executing multiple learning models (sub-models) M1 corresponding to the "Backbone" portion are distributed to different edge devices 1, and the results of one of the processes are stored in a memory (buffer memory) that functions as a buffer and referenced. The configuration of the image analysis system 100 in the sixth embodiment is similar to that of the image analysis system 100 in the first embodiment, and therefore the same reference numerals are used for the common configuration, and detailed description thereof will be omitted.

[0125] 14 is an explanatory diagram of processing executed by multiple edge devices 1 in the sixth embodiment. In the sixth embodiment, as in the fifth embodiment, among the multiple sub-models M1 constituting the learning model M0 shown in FIG. 4, the process of the sub-model M1 in the "Backbone" part that outputs the feature value γ, which is likely to have a heavy processing load, is executed by different edge devices 1.

[0126] In the sixth embodiment, the first edge device 1 also executes processing based on the learning model M1 that outputs a feature amount α, processing based on the learning model M1 that outputs a feature amount β, and processing based on the learning model M1 of the "Head" portion that uses the feature amounts α and β as well as the feature amount γ, as shown in Fig. 14. In the sixth embodiment, the edge device 1 executes the processing of the "Head" portion in different patterns, and is capable of switching between processing based on the learning model M1 that inputs two feature amounts α and β and executes image recognition processing, and processing based on the learning model M1 that inputs three feature amounts α, β, and γ.

[0127] In the sixth embodiment, the processing unit 10 of the first edge device 1 also acquires image data sent from the camera 4 to the local network LN, and provides the image data to a process based on a learning model M1 (sub-model) that outputs a feature value α of the “Backbone” part, and a process based on a learning model M1 (sub-model) that outputs a feature value β. Each process can provide the extracted features α and β to the “Head” part.

[0128] The second edge device 1 also acquires image data when it is output from the camera 4 and performs preprocessing such as filtering on the image data. The processing unit 10 of this edge device 1 also temporarily stores in memory time information or data identifying the image data of the frame image output from the camera 4. The processing unit 10 provides the preprocessed image data to a process based on a learning model M1 (sub-model) that outputs a feature value γ, and extracts the feature value γ. Each time the processing unit 10 obtains a feature value γ, it executes a process of writing the feature value γ and data identifying the time information or image data of the target frame image into the memory of the processing unit 10 of the first edge device 1. As a result, the feature value γ extracted in the most recent process and data identifying the time information or image data of the target frame image are stored in the memory of the first edge device 1.

[0129] In the sixth embodiment, the processing unit 10 of the first edge device 1 acquires feature quantities α and β for all frame images from the “Backbone” process and switches between two “Head” processes: a process for recognizing an object using the two feature quantities α and β, and a process for recognizing an object using the three feature quantities α, β, and γ. The processing unit 10 compares data identifying time information or image data for the frame images corresponding to the feature quantities α and β with data identifying time information or image data associated with the feature quantity γ received from the second edge device 1 and buffered in memory. If the comparison results in a time difference being within a predetermined range, the processing unit 10 executes a process for recognizing an object using the three feature quantities α, β, and γ, including the buffered feature quantity γ. If the comparison results in a time difference being outside the predetermined range, the processing unit 10 executes a process for recognizing an object using the two feature quantities α and β. Here, the predetermined range is set in advance in the storage unit 11 and is, for example, 2 to 3 seconds or more.

[0130] In the sixth embodiment, even if the computation load related to the extraction of the feature amount γ in the second edge device 1 is heavy, causing a delay in processing or a process stop, it is possible to perform computation for object recognition using the most recently derived feature amount γ as long as it is within an allowable range. If the process for outputting the feature amount γ is stopped and the time difference is outside the allowable range, the object recognition process can be continued using the two feature amounts α and β excluding the feature amount γ, thereby preventing the recognition process from stopping.

[0131] In this way, even in the sixth embodiment, even if a process in the "Backbone" part, which requires a large amount of calculation, is stalled or stopped, it is possible to avoid a situation in which a process in the "Head" part cannot be executed.

[0132] In the sixth embodiment, when a comparison is made between the data identifying the time information or image data of the frame image corresponding to the feature amounts α and β and the data identifying the time information or image data associated with the feature amount γ buffered in the memory, and the result shows that the time difference is outside a predetermined range, the stack in the processing of the second edge device 1 may be released. In this case, the processing of the first edge device 1 and the processing of the second edge device 1 may be reset and restarted.

[0133] (Seventh embodiment) In the seventh embodiment, the submodel is divided into a "Backbone" part that extracts features from image data and a "Head" part that integrates the extracted features and performs recognition processing, and the "Backbone" submodel, which requires less computation than the others, is tuned using image data obtained in the space to be analyzed. The configuration of image analysis system 100 in the seventh embodiment is similar to that of image analysis system 100 in the first embodiment, so the same reference numerals are used for common components and detailed description will be omitted.

[0134] FIG. 15 is an explanatory diagram of the processing executed by the edge device 1 of the seventh embodiment. In the seventh embodiment, the edge device 1 executes processes corresponding to the "Backbone" portion in parallel, as in the first to fourth embodiments. The three "Backbone" processes each output feature amounts α, β, and γ, and the process corresponding to the "Head" portion performs object recognition using the feature amounts α, β, and γ.

[0135] Here, the process for outputting the feature quantity β and the process for outputting the feature quantity γ output consistent feature quantities β and γ, respectively, when executed on the same object. The feature quantities β and γ do not need to be identical parameters, as long as they are at least comparable feature quantities.

[0136] In the image analysis system 100 of the seventh embodiment, the edge device 1 extracts three different feature quantities α, β, and γ simultaneously in parallel, as shown in Fig. 15, and the server 2 executes learning processing using video actually captured in the space to be analyzed. In particular, for the learning model M1 (sub-model) that outputs the feature quantity β, which requires a relatively small amount of calculation, the number of parameters has been reduced by distillation or the like, so on-site learning is performed to fine-tune the parameters of the learning model M1.

[0137] 16 and 17 are flowcharts showing an example of a learning process procedure by the edge device 1 of the seventh embodiment. The processing unit 10 of the edge device 1 or the processing unit 10 of another edge device 1 selects data to be learned and proceeds with learning as follows.

[0138] When image data is output from the camera 4, the processing unit 10 acquires it (step S171) and provides the image data to each process (step S172).

[0139] In each process, the processing unit 10 executes preprocessing such as filtering (step S173), provides the preprocessed image data to the learning model M1 of the selected process (step S174), and extracts features (step S175).

[0140] The processing unit 10 provides the acquired feature quantity to the process of the "Head" part (step S176). The processing unit 10 executes object recognition processing by the process of the "Head" part (step S177), and transmits the recognition result to the server 2 via the second communication unit 13 (step S178).

[0141] The processing unit 10 determines whether the learning stage of the learning model M1 to be learned is equal to or greater than a predetermined stage (step S179). In the example of the seventh embodiment, the learning model M1 to be learned in step S179 is the learning model M1 that outputs the feature β with a relatively small amount of calculation among the comparable feature β and feature γ. The processing unit 10 may determine whether the learning stage of step S179 is equal to or greater than a predetermined stage based on the number of times or a difference evaluation (S182) described later.

[0142] If it is determined that the learning stage is below the predetermined stage (S179: NO), the processing unit 10 selects comparable features from the features extracted in each process in step S175 (step S180).The processing unit 10 compares the selected features (step S181) and determines whether the difference between the features is within a predetermined range (step S182).

[0143] Whether the difference in feature amounts is within a predetermined range may be used as a criterion for determining whether the learning has progressed. Here, if the difference in feature amounts is within the predetermined range, it can be determined that even in the "Backbone" process, which requires a small amount of calculation, there is no discrepancy with the results of other "Backbone" processes, which have a large number of parameters, and that the learning has progressed to the point where sufficiently accurate recognition results can be obtained.

[0144] When the processing unit 10 determines that the difference in the feature amount is outside a predetermined range (S182: NO), it transmits the image data after the preprocessing in step S173 to the server 2 as a learning image and stores it in the storage unit 21 of the server 2 (step S183). When the processing unit 20 of the server 2 has accumulated enough learning data to perform (fine) tuning (relearning) of the learning model M1 with a small amount of computation, it uses the learning data (including the above image data) stored in the storage unit 21 to perform (fine) tuning of the learning model M1 with a small amount of computation.

[0145] In step S179, if it is determined that the learning stage is at or above a predetermined stage (S179: YES) and the learning has progressed sufficiently, the processing unit 10 omits the processing of steps S180-S183 and ends processing of one image data.

[0146] In step S182, if it is determined that the difference in the feature amount is within the predetermined range (S182: YES), the process of step S183 is also omitted, and the process for one image data is completed.

[0147] In the seventh embodiment, the selection of the three processes corresponding to the "Backbone" part that extracts features from image data may also be appropriately selected depending on the processing load and the specified frequency. Furthermore, in the seventh embodiment, when a sufficient amount of learning data has been accumulated in the storage unit 21 of the server 2, the learning model M1 used in the process that outputs the feature γ may be retrained.

[0148] As described above, in image analysis system 100 of the seventh embodiment, when re-learning is performed using a small amount of learning data to adapt the "Backbone" submodel to the usage environment, re-learning is performed only on a submodel among the multiple "Backbone" submodels that requires a relatively small amount of calculation (for example, the "Backbone" submodel that outputs the above feature value β). The reason for this is due to the following technical idea.

[0149] That is, when re-learning ((fine) tuning) the learning model used in the edge device 1 installed in the field (store, etc.) to adapt it to the field environment is performed, if the amount of training data (learning data) for re-learning, in which tag data is attached to images collected in the field (store, etc.), is small, it is difficult to complete the re-learning (tuning) of the large "Backbone" (submodel) with a large number of parameters, and even if tuning is performed using a small amount of training data for the large "Backbone," there is a high possibility that performance will degrade. Therefore, when tuning to adapt to the field environment is performed, if the amount of training data using images collected in the field (store, etc.) is small, tuning is not performed on the large "Backbone" (submodel) with a large number of parameters, and only fine tuning is performed on the small "Backbone" (submodel) with a small number of parameters.

[0150] This allows the high performance in general environments of a large "backbone" that has been trained using training data based on general images (such as images from multiple other stores) rather than images collected in an on-site environment to be maintained. On the other hand, for a small "backbone" that can be (fine-) tuned using a small amount of training data, it can be adapted to the on-site environment by (fine-) tuning using training data that uses images collected in the field (such as a store). Note that when there is a large amount of training data using images collected in the field (such as a store), the large "backbone" (submodel) will also be re-trained (tuned).

[0151] Furthermore, if the processing unit 10 of the edge device 1 determines in step S183 that the difference in the feature amounts is outside a predetermined range (S182: NO), the image data after the preprocessing in step S173 is transmitted to the server 2 as a learning image and stored in the storage unit 21 of the server 2 for the following reason: That is, when the feature amount β and the feature amount γ are comparable feature amounts (for example, feature amounts of the same type) as described above, if the difference between the feature amount β and the feature amount γ, which are feature amounts output when images collected on-site (such as a store) are input to a small (low-accuracy) "Backbone" and a large (high-accuracy) "Backbone," is large (S182: NO), this means that the processing of the small sub-model (the "Backbone" that outputs the feature amount β) is not working well in this on-site environment. Therefore, by utilizing this, we select (on-site) images with a large difference between feature β and feature γ, and use these images as images for (fine-)tuning (to adapt to the on-site environment) of a small sub-model (the "backbone" that outputs feature β).

[0152] The embodiments disclosed above are illustrative in all respects and are not restrictive. The scope of the present invention is defined by the claims, and includes all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0153] 100 Image Analysis System 1. Edge Devices 10 Processing section 12 First Communications Department 13 Second Communications Department 2 Server

Claims

1. The computer When image data is input, a trained model is trained to output a recognition result of the object appearing in the image. The trained model is divided into multiple sub-models based on the feature values ​​extracted within the model, and performing a process of extracting feature amounts based on each of the plurality of sub-models for image data input in a time series; changing the execution frequency of a process based on a submodel among the plurality of submodels that has a larger amount of calculation than the other submodels; Performing recognition processing using each feature extracted by the process based on the plurality of sub-models Information processing methods.

2. The trained model is divided into a plurality of sub-models corresponding to a backbone portion that extracts features from image data, and a sub-model corresponding to a head portion that is trained to output a recognition result of an object appearing in an image of the image data when the features of the image data extracted by each of the plurality of sub-models are input, The computer Using at least some of the feature amounts extracted by a plurality of sub-models corresponding to the backbone portion, including a sub-model with a larger amount of calculation than the others, a recognition process is performed based on the sub-model corresponding to the head portion. The information processing method according to claim 1 .

3. The trained model is divided into a plurality of sub-models corresponding to a backbone portion that extracts features from image data, and a sub-model corresponding to a head portion that is trained to output a recognition result of an object appearing in an image of the image data when the features of the image data extracted by each of the plurality of sub-models are input, The computer selecting, from among the sub-models corresponding to the head portion, a model that uses a feature extracted by the sub-model with a larger amount of calculation than others, or a model that does not use a feature extracted by the sub-model with a larger amount of calculation than others; Perform recognition processing based on the selected submodel The information processing method according to claim 1 .

4. The computer A process based on a sub-model that requires more computation than the others is executed at a predetermined rate relative to the frame rate of the image data. The information processing method according to any one of claims 1 to 3.

5. The computer changes the execution frequency of the process based on the sub-model having a larger amount of calculation than the others in accordance with the load on the computer. The information processing method according to any one of claims 1 to 3.

6. The computer uses different resolutions of image data input to the plurality of sub-models for each sub-model. The information processing method according to any one of claims 1 to 3.

7. The computer determining whether or not a time difference between the target image data and the image data immediately preceding the target image data among the image data input in time series is equal to or greater than a predetermined amount; If it is determined that the amount is equal to or greater than the predetermined amount, a process based on the sub-model having a larger amount of calculation is executed; If it is determined that the amount of calculation is less than the predetermined amount, a process based on another sub-model is executed in which the sub-model having a larger amount of calculation is substituted. The information processing method according to claim 3 .

8. The computer determining whether or not a time difference between the target image data and the image data immediately preceding the target image data among the image data input in time series is equal to or greater than a predetermined amount; If it is determined that the amount of calculation is equal to or greater than the predetermined amount, increasing the execution frequency of the process based on the sub-model having a larger amount of calculation than the others; If it is determined that the amount of calculation is less than the predetermined amount, the execution frequency of the process based on the sub-model having a larger amount of calculation than the others is decreased. The information processing method according to any one of claims 1 to 3.

9. Distributing processes based on the plurality of sub-models across different computers; a computer that executes a process based on the sub-model with a larger amount of calculation than the other computers outputs the feature values ​​extracted by the sub-model to the other computers each time the process is executed; The other computer a buffer memory for updating the feature amount each time the feature amount is output; Executes a recognition process using the feature values ​​extracted by the process based on the other submodel and the feature values ​​stored in the buffer memory.

3. The information processing method according to claim 1 or 2.

10. Distributing processes based on the plurality of sub-models across different computers; a computer that executes a process based on the sub-model having a larger amount of calculation than the other computers, the computer associates the feature extracted by the sub-model with identification data that identifies the timing at which the image data of the target of the feature was captured, and outputs the associated feature to another computer each time the computer executes the process; The other computer a buffer memory for updating the feature amount and the identification data of the image each time the feature amount and the identification data of the image are output; comparing identification data for identifying the timing at which image data that is the subject of feature quantities extracted by a process based on another sub-model was captured with the identification data stored in the buffer memory; determining whether the time difference of the image data is within a predetermined range; Only when it is determined that the feature is within a predetermined range, a recognition process is performed using the feature extracted by the process based on the other sub-model and the feature stored in the buffer memory.

3. The information processing method according to claim 1 or 2.

11. When it is determined that the time difference is outside a predetermined range, the other computer resets the buffer memory and the process based on the submodel with a larger amount of calculation than the other computers, and restarts the processes based on the multiple submodels. The information processing method according to claim 10.

12. When the computer performs re-learning for adapting the plurality of sub-models to a usage environment using a small amount of learning data, the computer performs re-learning only for sub-models with a relatively small amount of calculation among the plurality of sub-models. The information processing method according to any one of claims 1 to 3.

13. The computer Execute both a process based on a submodel having a larger amount of calculation than the others among the plurality of submodels and a process based on a submodel having a relatively smaller amount of calculation, Determine whether the difference between the output results of both executed processes is within a predetermined range; If it is determined that the image data is outside the predetermined range, the image data is stored as learning data. Based on the stored learning data, the sub-model with a relatively small amount of calculation is learned. The information processing method according to claim 12.

14. When image data is input, a trained model is trained to output a recognition result of an object appearing in the image. The trained model is divided into multiple sub-models based on the feature values ​​extracted within the model, and performing a process of extracting feature amounts based on each of the plurality of sub-models for image data input in a time series; changing the execution frequency of a process based on a submodel among the plurality of submodels that has a larger amount of calculation than the other submodels; Performing recognition processing using each feature extracted by the process based on the plurality of sub-models An information processing device including a processing unit.

15. On the computer, When image data is input, a trained model is trained to output a recognition result of the object appearing in the image. The trained model is divided into multiple sub-models based on the feature values ​​extracted within the model, and performing a process of extracting feature amounts based on each of the plurality of sub-models for image data input in a time series; changing the execution frequency of a process based on a submodel among the plurality of submodels that has a larger amount of calculation than the other submodels; Performing recognition processing using each feature extracted by the process based on the plurality of sub-models A computer program that executes a process.

Images