Information processing method, information processing device, and computer program

Abstract

To provide a method for processing information, an information processing apparatus, and a computer program that enable appropriate execution of processing using a learning model on an edge device with limited resources.SOLUTION: The method for processing information causes a computer to use a plurality of submodels obtained by dividing a trained model, which is trained to output a recognition result of an object appearing in an image when image data is input, for each feature extracted within the model. For image data input in time series, the computer performs, in parallel, processes for extracting features on the basis of the submodels. Among the plurality of submodels, the execution frequency of a process based on a submodel having a larger computational load than others is changed with respect to the frame rate of the image data, and recognition processing is executed using respective features extracted by the processes based on the plurality of submodels.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an information processing method, an information processing apparatus, and a computer program using a learned model (hereinafter referred to as a "learning model").

Background Art

[0002] An image recognition system is used that uses a camera that captures a store or a specific area as an angle, detects an object such as a person shown in an image captured by the camera, and outputs a recognition result of the detected object. In such an image recognition system, a learning model using a neural network (hereinafter referred to as NN: Neural Network) that is trained to output a detection result and a recognition result of an object shown in the target image data when the target image data is input is used.

[0003] Due to the improvement of computing power and hardware technology, instead of a configuration that collects data on a server with abundant computing resources and performs image processing, even on an edge device with relatively scarce computing resources, processing using a learning model has become possible. An AI (Artificial Intelligence) camera that mounts a learning model on the camera itself and outputs image data together with a recognition result is also used. By being able to perform processing using a learning model on an edge device, it is possible to maintain the function even if there is a communication failure with the server.

[0004] Further, Patent Document 1 discloses causing different processing to be performed on each edge device. For example, one edge device is caused to execute processing on an image obtained from a video camera, another edge device is caused to execute processing using a thermostat, another edge device is caused to control an automated teller machine, and another edge device is caused to execute processing for recognizing items placed in a cart.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-129390 [Overview of the project] [Problems that the invention aims to solve]

[0006] Image recognition systems that require real-time processing, such as displaying advertisements tailored to the attributes of customers captured by cameras installed in stores, demand faster and more accurate processing capabilities.

[0007] Even with improvements in edge device hardware performance, resources are limited, so performing computationally intensive processing on all frames of the target video may result in real-time processing not being able to keep up. Furthermore, in configurations that execute processing in parallel for each function to speed up processing, if the completion times of each distributed process are staggered, inconsistencies may occur when integrating the processing results.

[0008] The present invention aims to provide an information processing method, an information processing device, and a computer program that enable appropriate processing using a learning model on resource-limited edge devices. [Means for solving the problem]

[0009] An information processing method in one embodiment of the present disclosure is a computer that, when image data is input, outputs a trained model that is trained to output the recognition result of an object depicted in the image. Using , The trained model is divided into a number of submodels corresponding to a backbone portion that extracts features from image data, and a submodel corresponding to a head portion that is trained to output recognition results of objects in the image data when the features of the image data extracted by each of the submodels are input, and the computer, For image data input in time series, This corresponds to the backbone. A process is carried out to extract features based on each of the multiple submodels, This corresponds to the backbone. Among multiple submodels, the execution frequency of processes based on the submodel with a higher computational load than the others. Low death, From among the submodels corresponding to the head portion, one is selected that uses the features extracted by the submodel with a higher computational load than the others, and the other is selected that does not use the features extracted by the submodel with a higher computational load. Based on the selected submodel, the recognition process is executed. .

[0010] In one embodiment of the information processing method of this disclosure, the learning model is divided into submodels for processing. When performing this process, if you reduce the frequency of processes based on computationally intensive submodels, those processes may not be executed. Therefore, you may switch between performing recognition processing without using the features extracted by those processes and performing recognition processing using all features. .

[0011] An information processing method according to one embodiment of this disclosure is: The aforementioned The computer may use at least a portion of the feature quantities extracted by a plurality of submodels corresponding to the backbone portion, including a submodel with a higher computational load than others, and perform recognition processing based on the submodel corresponding to the head portion.

[0012] In one embodiment of the information processing method of this disclosure, when processing a learning model by dividing it into submodels, if the frequency of a process based on a submodel with a large computational cost is changed, that process may not be executed. Therefore, the results of that process are stored, and recognition processing is performed using all the features.

[0015] In one embodiment of the information processing method of this disclosure, the computer may execute a process based on a submodel that has a greater computational load than others at a predetermined rate relative to the frame rate of the image data.

[0016] In one embodiment of the information processing method described herein, processes based on submodels with a higher computational load than others are executed at a reduced rate, such as once every few frames. This prevents the processing of processes based on submodels with a higher computational load from becoming congested.

[0017] In one embodiment of the information processing method of this disclosure, the computer may change the execution frequency of the process based on the submodel with a larger computational load than others, according to the load on the computer.

[0018] In one embodiment of the information processing method described herein, the frequency of processes based on submodels with a higher computational load is changed depending on whether the actual load is heavy or not. By omitting such processes only when the load is heavy, the highest possible accuracy can be maintained.

[0019] In one embodiment of the information processing method of this disclosure, the computer may use different resolutions for the image data input to the plurality of submodels for each submodel.

[0020] In one embodiment of the information processing method described herein, the amount of computation can be further reduced and the processing load reduced by using different resolutions for the image data input to each of the multiple submodels.

[0021] In one embodiment of the information processing method of the present disclosure, the computer determines whether the time difference between the target image and the previous image data among the image data input in time series is greater than or equal to a predetermined amount. If it is determined to be greater than or equal to the predetermined amount, the computer may execute a process based on a submodel with a larger computational load than the others. If it is determined to be less than the predetermined amount, the computer may execute a process based on another submodel that replaces the submodel with the larger computational load.

[0022] In one embodiment of the information processing method of this disclosure, when there is a large difference between the frame image and the previous frame image in the time series, that is, when it is better to perform the process in the submodel which has a larger computational load than others, the process is selected and executed. This makes the calculations more efficient and balances reduced processing load with high accuracy.

[0023] In one embodiment of the information processing method of this disclosure, the computer may determine whether the time difference between the target image and the previous image data among the image data input in time series is greater than or equal to a predetermined amount. If it is determined to be greater than or equal to the predetermined amount, the computer may increase the execution frequency of the process based on the submodel with a larger computational load than the others. If it is determined to be less than the predetermined amount, the computer may decrease the execution frequency of the process based on the submodel with a larger computational load than the others.

[0024] In the information processing method of an embodiment of the present disclosure, not only when the difference from the previous frame image in time series is large, that is, when it is determined that it is better to perform the process in the sub-model with a larger amount of calculation than others, but also the frequency of that process is increased in the time zone with a large difference. Thereby, the calculation can be made efficient, and a balance can be achieved between reducing the processing load and improving the accuracy.

[0025] The information processing method of an embodiment of the present disclosure is the above-mentioned This corresponds to the backbone. Processes based on a plurality of sub-models are distributed to different computers, and the computer that executes the process based on the sub-model with a larger amount of calculation than others outputs the feature amount extracted by the sub-model to other computers each time the process is executed. Each time the feature amount is output, the other computers use a buffer memory that updates the feature amount, and perform recognition processing using the feature amount extracted by the process based on other sub-models and the feature amount stored in the buffer memory.

[0026] In the information processing method of an embodiment of the present disclosure, not only is the learning model divided into sub-models for processing, but also the hardware that executes the process based on the sub-model may be distributed to different computers. Thereby, it becomes possible to efficiently use the computing resources of edge devices with limited resources. The feature amounts obtained as a result of the processes executed by other computers are buffered, and the latest extraction results to be buffered are used to execute the recognition process by sharing the processing.

[0027] The information processing method of an embodiment of the present disclosure is In an information processing method in which a computer, when given image data as input, is trained to output recognition results of objects depicted in an image, and a pre-trained model is divided into multiple submodels based on the features extracted within the model, a process is performed to extract features from the image data input in a time series based on each of the multiple submodels, the execution frequency of the process based on the submodel that requires more computation than the others is reduced, and recognition processing is performed using each feature extracted by the processes based on the multiple submodels,The processes based on the multiple submodels are distributed across different computers, and the computer executing the process based on the submodel with a greater computational load than the others outputs the feature quantities extracted by the submodel to the other computers each time it executes the process, associating them with identification data that identifies the timing at which the image data targeting the feature quantities was taken. The other computers use a buffer memory to update the feature quantities and identification data each time the feature quantities and the image identification data are output, compare the identification data that identifies the timing at which the image data targeting the feature quantities extracted by the other submodel processes was taken with the identification data stored in the buffer memory, determine whether the time difference of the image data is within a predetermined range, and only if it is determined to be within the predetermined range, may perform recognition processing using the feature quantities extracted by the other submodel processes and the feature quantities stored in the buffer memory.

[0028] In one embodiment of the information processing method of this disclosure, not only is the learning model divided into submodels for processing, but the hardware that executes the processes based on the submodels may also be distributed to different devices. In such cases, the feature quantities obtained as a result of processes executed by other computers may be buffered and used. In one embodiment, if the feature quantities output from another computer and buffered are for processing frame images at different timings, using those feature quantities to perform recognition processing will reduce accuracy. This can be avoided by adding a determination of whether or not the time difference is within a predetermined range.

[0029] In one embodiment of the information processing method of this disclosure, if the other computer determines that the time difference is outside a predetermined range, it may reset the buffer memory and the process based on the submodel with a larger computation amount than the others, and restart the processes based on the multiple submodels.

[0030] In one embodiment of the information processing method described herein, when determining whether the time difference is within a predetermined range, if the time difference is outside the predetermined range, it means that a process based on a submodel with a larger computational load than others is either stalled or has stopped due to a failure, making it difficult to maintain high accuracy even if the process continues. Therefore, in such cases, a reset can be performed to allow the process to resume, thereby maintaining high accuracy.

[0031] In one embodiment of the information processing method of this disclosure, when the computer retrains the multiple submodels to adapt them to the operating environment using a small amount of training data, it may only retrain the submodels among the multiple submodels that require relatively less computation.

[0032] In one embodiment of the information processing method described herein, submodels with a higher computational load than others have more parameters, and therefore require a larger amount of training before they can perform accurate feature extraction. Consequently, when retraining multiple submodels to adapt them to a specific environment using a small amount of training data, only the submodels with relatively low computational loads are retrained. This allows the submodels with higher computational loads to maintain their original trained state, thereby preserving their generalization performance, while the submodels with relatively low computational loads can complete their training, thus improving their adaptability to the environment.

[0035] An information processing device in one embodiment of the present disclosure is a trained model that, when image data is input, outputs the recognition result of an object depicted in the image. Using , The trained model is divided into a number of submodels corresponding to a backbone portion that extracts features from image data, and a submodel corresponding to a head portion that is trained to output recognition results of objects in the image data when the features of the image data extracted by each of the submodels are input. For image data input in time series, This corresponds to the backbone. A process is carried out to extract features based on each of the multiple submodels, This corresponds to the backbone. Among multiple submodels, the execution frequency of processes based on the submodel with a higher computational load than the others. Low death, From among the submodels corresponding to the head portion, one is selected that uses the features extracted by the submodel with a higher computational load than the others, and the other is selected that does not use the features extracted by the submodel with a higher computational load. Based on the selected submodel, the recognition process is executed. It is equipped with a processing unit.

[0036] A computer program in one embodiment of the present disclosure is a trained model that is trained to output recognition results of objects in an image when image data is input to the computer. Using , The trained model is divided into a number of submodels corresponding to a backbone portion that extracts features from image data, and a submodel corresponding to a head portion that is trained to output recognition results of objects in the image data when the features of the image data extracted by each of the submodels are input. For image data input in time series, This corresponds to the backbone. A process is carried out to extract features based on each of the multiple submodels, This corresponds to the backbone. Among multiple submodels, the execution frequency of processes based on the submodel with a higher computational load than the others. Low death, From among the submodels corresponding to the head portion, one is selected that uses the features extracted by the submodel with a higher computational load than the others, and the other is selected that does not use the features extracted by the submodel with a higher computational load. Based on the selected submodel, the recognition process is executed. Execute the process. [Effects of the Invention]

[0037] According to this disclosure, when a learning model is divided into submodels for processing, the accuracy of the recognition process is maintained while reducing the frequency of processes using computationally intensive submodels. This avoids congestion in processes using computationally intensive submodels among the divided processes, and enables high-speed recognition processing using the learning model even on edge devices with limited resources. [Brief explanation of the drawing]

[0038] [Figure 1] This is an overview diagram of the image analysis system. [Figure 2] This is a block diagram showing the configuration of an edge device. [Figure 3] This is a block of the server configuration. [Figure 4] This is an overview diagram of the learning model. [Figure 5] This is a diagram illustrating the process performed on the edge device of the first embodiment. [Figure 6] This is a diagram illustrating the process performed on multiple edge devices in the modified example 1. [Figure 7] This flowchart shows an example of a processing procedure performed by the edge device of the second embodiment. [Figure 8]This is an explanatory diagram of the process performed on the edge device of the second embodiment. [Figure 9] This is a diagram illustrating the process performed on the edge device of the third embodiment. [Figure 10] This flowchart shows an example of a processing procedure performed by the edge device of the fourth embodiment. [Figure 11] This is a diagram illustrating the process performed on the edge device of the fourth embodiment. [Figure 12] This flowchart shows an example of the processing steps performed by the edge device in Modification Example 2. [Figure 13] This is a diagram illustrating the process performed by multiple edge devices in the fifth embodiment. [Figure 14] This is a diagram illustrating the process performed by multiple edge devices in the sixth embodiment. [Figure 15] This is an explanatory diagram of the processing performed on the edge device of the seventh embodiment. [Figure 16] This is a flowchart showing an example of a learning process procedure using an edge device according to the seventh embodiment. [Figure 17] This is a flowchart showing an example of a learning process procedure using an edge device according to the seventh embodiment. [Modes for carrying out the invention]

[0039] This disclosure will be described in detail with reference to the drawings illustrating its embodiments. The following embodiments describe an image analysis system to which the information processing method of this disclosure is applied.

[0040] (First Embodiment) Figure 1 is a schematic diagram of the image analysis system 100. The image analysis system 100 in the first embodiment includes one or more cameras 4 installed in the space to be analyzed, such as a store, multiple edge devices 1 connected to the cameras 4, a server 2 capable of communication with the edge devices 1, and a client 3 capable of connecting to the server 2.

[0041] One or more cameras 4 and multiple edge devices 1 can communicate with each other via a local network LN, which is either wireless or wired. The local network LN may be a wired LAN or a wireless network such as Wi-Fi or Bluetooth (registered trademark).

[0042] One or more cameras 4 output image data using image sensors that support visible light and / or near-infrared light. Cameras 4 output image data in time series at a rate of several fps to tens of fps. Cameras 4 are installed so as to look down from above the space, such as on the ceiling or shelves of the space to be analyzed. Cameras 4 may be ceiling-mounted and have a 360-degree field of view. Cameras 4 sequentially transmit image data to edge devices 1 via the local network LN.

[0043] Some or all of the multiple edge devices 1 can communicate with server 2 via network N. Network N is a wired or wireless communication network that may include a public communication network, a dedicated line, or a carrier network. Client 3 can communicate with server via network N.

[0044] In the first embodiment, multiple edge devices 1 extract features from image data acquired from a camera 4, recognize objects in the image data (specifically, people present in the analysis space) based on these features, and output recognition results using a neural network-based learning model. A learning model using a neural network in image recognition can be divided into a "Backbone" part that extracts features from the input image data and a "Head" part that performs recognition processing (recognition, discrimination, region segmentation, etc.) based on the extracted features. Furthermore, in the image analysis system 100 of the first embodiment, the part corresponding to the "Backbone" of the learning model is divided into submodels according to the features to be extracted. Processing based on the submodels is performed in parallel to achieve high speed and load balancing, thereby realizing high-precision recognition in the analysis space.

[0045] In the first embodiment, the edge device 1 provides the submodel of the "Head" portion with feature quantities derived from image data based on the assigned submodel of the "Backbone" portion to determine the attributes of objects in the image, stores the text indicating the attributes in association with identification data that allows identification of the space to be analyzed, and transmits it to the server 2 as needed.

[0046] Server 2 records data transmitted from the "Head" process of edge device 1 in database 210 and can perform analytical processing, such as statistical processing of the attributes of detected objects, for each space under analysis. The results of the analytical processing on Server 2 can be viewed by client 3 for each space under analysis. The manager of the space under analysis can use client 3 to view the attributes of objects detected in the space and their statistical results. More specifically, by building an image analysis system 100 in a store, the store manager can check aggregated results such as the age range and gender of customers visiting the store from client 3.

[0047] Each edge device 1, upon receiving image data, performs calculations on a submodel that has been trained to output features for that image. Depending on the required features, the amount of computation can be large. If the calculations for all "Backbone" submodels are performed for all frames of image data output from camera 4 at a rate of several fps to tens of fps, the completion of the "Head" part of the process may not keep up with the rate. Therefore, in the image analysis system 100 of the first embodiment, it is not necessary to perform calculations on some of the submodels of the multiple "Backbone" parts for all frames of image.

[0048] The following describes the details of this image analysis system 100.

[0049] Figure 2 is a block diagram showing the configuration of edge device 1. Edge device 1 is a box-type device that can be installed in multiple locations, such as stores, along with cameras 4. Edge device 1 comprises a processing unit 10, a storage unit 11, a first communication unit 12, and a second communication unit 13.

[0050] The processing unit 10 includes one or more processors such as CPUs (Central Processing Units), MPUs (Micro-Processing Units), and GPUs (Graphics Processing Units). The processing unit 10 includes memory, which is a temporary storage medium such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory). The processing unit 10 includes a timer and can obtain time information at each point in time from data from the timer. The processing unit 10 may be configured as a single hardware (SoC: System On a Chip) integrating the processor, memory, and furthermore, the storage unit 11, the first communication unit 12, and the second communication unit 13. The specifications of the processing unit 10 may be the same or different for each edge device 1.

[0051] The processing unit 10 reads the information processing program P1 stored in the storage unit 11 into memory and executes it, thereby causing the processor to perform various processes described later, and to function as the edge device 1 of this disclosure.

[0052] The storage unit 11 is a relatively large-capacity non-temporary storage medium such as a hard disk or flash memory. A portion of the storage unit 11 may be removable.

[0053] The memory unit 11 stores the program (program product) necessary for the processing unit 10 to execute processing, as well as reference configuration data. The configuration data includes the identifier of the edge device 1, identification data of the connected camera 4, etc. The program product includes the OS (Operating System) program, the information processing program P1 that runs on the OS, and the learning model M1 corresponding to the sub-model used. Details of the learning model M1 will be described later.

[0054] The information processing program P1 stored in the storage unit 11 may be one that the processing unit 10 reads from an information processing program P9 stored on a non-temporary storage medium 9 that is readable from a computer and stored in the storage unit 11, or it may be one that is pre-stored at the time of shipment. The information processing program P1 may also be one that the processing unit 10 downloads from the server 2 or another download server via the second communication unit 13 and stores in the storage unit 11.

[0055] The learning model M1 stored in the memory unit 11 may be selected by the processing unit 10 from a group of submodels M9 stored in the non-temporary storage medium 9 and stored in the memory unit 11, or it may be selected and stored in advance at the time of shipment. The learning model M1 or a part thereof stored in the memory unit 11 may be downloaded by the processing unit 10 from the server 2 or another download server via the second communication unit 13 and stored in the memory unit 11.

[0056] The memory unit 11 may store the feature quantities extracted by the calculations in the "Backbone" portion, associating them with time information or data that identifies the target image data, and storing them separately for each feature quantity. The memory unit 11 stores the recognition result data (e.g., attributes) obtained by the calculations in the "Head" portion, associating it with time information or data that identifies the target image data.

[0057] The first communication unit 12 is a communication device that enables communication via the local network LN. The first communication unit 12 may be a LAN network card or a CAN communication device. The first communication unit 12 may be a wireless communication device that supports wireless networks such as WiFi or Bluetooth (registered trademark). The first communication unit 12 may include multiple communication devices that support various types of cameras 4. The first communication unit 12 may include an interface such as USB (Universal Serial Bus) that connects to the camera 4. The first communication unit 12 can be replaced by an interface that connects to the camera 4 via a coaxial cable or other serial bus. The processing unit 10 acquires image data from the camera 4 via the local network LN using the first communication unit 12. The processing unit 10 can send and receive data with other edge devices 1 using the first communication unit 12. The first communication unit 12 may be the same device as the second communication unit 13.

[0058] The second communication unit 13 is a communication device that enables communication via an external network N. The second communication unit 13 may be a wired LAN network card, a communication device that enables carrier communication via a carrier network, or a communication device that supports wireless networks such as WiFi or Bluetooth (registered trademark). The second communication unit 13 may support encrypted communication such as SSL with the server 2. The second communication unit 13 may also be an interface for enabling connection with the server 2 via a dedicated line. The second communication unit 13 may only be provided by the edge device 1H that outputs recognition results to the server 2 based on the submodel of the “Head” part.

[0059] Figure 3 is a block diagram showing the configuration of Server 2. Server 2 may consist of a single server computer, or it may be configured to distribute processing across multiple server computers. Server 2 includes a processing unit 20, a storage unit 21, and a communication unit 22.

[0060] The processing unit 20 includes one or more processors such as CPUs, MPUs, and GPUs. The processing unit 20 also includes memory, which is a temporary storage medium such as SRAM or DRAM.

[0061] The storage unit 21 is a relatively large-capacity non-temporary storage medium such as a hard disk or flash memory. The storage unit 21 stores the program (program product) necessary for the processing unit 20 to execute processing.

[0062] The program product stored in the memory unit 21 includes the server program P2. The server program P2 includes a module that enables it to function as a web server and can output the results of processing on the server 2 to the client 3 via a web page. The memory unit 21 stores login account data to distinguish users using the client 3, and can identify whether the logged-in user is an account with operational privileges for the server 2 or a customer account that views the processing results of the server 2. The data output program to the client 3 included in the server program P2 may be a web server program, or a program that supports other protocols as long as it can output data to the screen displayed on the client 3.

[0063] The server program P2 stored in the storage unit 21 may be one that the processing unit 20 reads from the server program P8 stored in the non-temporary storage medium 8 readable from the computer and stores in the storage unit 21, or it may be one that is pre-stored at the time of shipment. The server program P2 stored in the storage unit 21 may also be one that the processing unit 20 downloads from the download server via the communication unit 22 and stores in the storage unit 21.

[0064] The storage unit 21 has a database 210 that stores spatial definition data and administrator data for the space to be analyzed, associated with the identification data of the space to be analyzed. The storage unit 21 should store the account of client 3 associated with each space to be analyzed in the database 210, so that the analysis results that can be accessed by client 3 can be narrowed down. The storage unit 21 stores data such as the number of edge devices 1, the number and specifications of cameras 4 installed in the space, associated with the identification data of the space to be analyzed. Specifically, the storage unit 21 stores in the database 210 the layout data of the store's floor and shelves, the installation position and field of view data of the cameras 4 corresponding to the floor, and the identification data of the cameras 4, associated with the identification data of the store.

[0065] The communication unit 22 is a communication device that enables communication with client 3 via network N. The communication unit 22 may also be a communication device that enables communication with client 3 via a dedicated line.

[0066] Figure 4 is a schematic diagram of the learning model M1. Figure 4 shows the functions performed by the learning model M1 stored on the edge device 1. As shown in Figure 4, the learning model M1 consists of submodels that output different feature quantities for the input image data, or submodels that output recognition results using the output feature quantities, and together they constitute a learning model M0 that is trained to output recognition results for the input image. In the first embodiment, as shown in Figure 4, the learning model M0 includes a learning model M1 (submodel) that extracts feature quantities α, β, and γ respectively, and a learning model M1 (submodel) that outputs the result of recognizing objects in the image using the feature quantities α, β, and γ in calculations. The learning models M1, which are submodels, are combined and have been trained in advance using training data, which is a set of image data and recognition results, so that they can be recognized accurately as a whole on the server 2 or other devices.

[0067] In the first embodiment, as shown in Figure 4, the learning model M1, which is divided into multiple submodels, is stored and used within a single edge device 1. In the first embodiment, an example is given in which processing based on multiple learning models (submodels) M1 is performed in parallel on a single edge device 1. However, multiple learning models (submodels) M1 may be distributed and executed on multiple edge devices 1 (see Modification 1).

[0068] Figure 5 is an explanatory diagram of the processing performed by the edge device 1 in the first embodiment. In the first embodiment, the edge device 1 performs three processes corresponding to the “Backbone” portion in parallel. The three “Backbone” processes output feature quantities α, β, and γ, respectively. The edge device 1 also performs processing based on the “Head” portion's learning model M1, which uses the output feature quantities α, β, and γ, to recognize objects depicted in the input image data and output their attributes. In the image analysis system 100 of the first embodiment, as shown in Figure 5, the processing steps of the “Backbone” portion for a single frame image are executed simultaneously in parallel. This speeds up the process of recognizing objects depicted in the image.

[0069] In the example shown in Figure 5, the processing unit 10 of the edge device 1 receives image data from the camera 4 when it is sent to the local network LN, and provides the image data to each of the three processes corresponding to the "Backbone" portion.

[0070] In a process based on a learning model M1 (submodel) that outputs feature α, the processing unit 10 performs preprocessing such as filtering of feature α for the learning model M1. The processing unit 10 provides the preprocessed image data to the learning model M1, extracts feature α, and provides feature α to the "Head" portion of the learning model M1. The process based on the learning model M1 that outputs feature α repeats preprocessing and the process of extracting feature α using the learning model M1 for all frames of the input image data.

[0071] In a process based on a learning model M1 (submodel) that outputs feature quantities β, the processing unit 10 performs preprocessing such as filtering of feature quantities β for the learning model M1. The processing unit 10 provides the preprocessed image data to the learning model M1, extracts feature quantities β, and provides the feature quantities β to the "Head" portion of the learning model M1. The process based on the learning model M1 that outputs feature quantities β repeats preprocessing and the process of extracting feature quantities β using the learning model M1 for all input image data.

[0072] The computational complexity of the process based on the learning model M1 (submodel) that outputs feature γ is greater than that of the processes of other learning models M1. In this process based on the learning model M1 that outputs feature γ, the processing unit 10 performs preprocessing such as filtering on the frame image once every three times when image data is input. In other words, the processing unit 10 executes the process based on the learning model M1 that outputs feature γ at a predetermined rate relative to the frame rate. The processing unit 10 provides the preprocessed image data to the learning model M1 stored in the device's memory unit 11, extracts feature γ, and provides it to the learning model M1 in the "Head" part. The process based on the learning model M1 (submodel) that outputs feature γ is not executed two out of three times. Specifically, the processing unit 10 increments the count each time image data is output from the camera 4, and each time the count reaches three times, it performs preprocessing, extracts feature γ, outputs feature γ, and then resets the count to zero. The frequency is not limited to once every three times; it could be once every two times, once every four times, or even less frequently. This avoids delays in the calculation required to output the feature γ.

[0073] In the "Head" part of the process, the processing unit 10 obtains feature quantities α and β from the "Backbone" process for all frame images, and once every three times, waits for feature quantity γ to be output by calculation before executing object recognition processing. The processing unit 10 stores feature quantity γ when it is obtained. The processing unit 10 performs object recognition calculations using feature quantities α, β, and γ with the learning model M1, and outputs the attributes of the recognized object to the server 2 via the second communication unit 13. Since the output of feature quantity γ is not performed two out of three times, the processing unit 10 performs calculations using feature quantities α, β, and γ, including the stored feature quantity γ. This prevents delays in processing on the edge device 1 due to delays in the calculation of feature quantity γ.

[0074] Thus, in the image analysis system 100 of the first embodiment, the computational load is reduced and the processing load is alleviated while maintaining high-speed processing by omitting processes based on the computationally intensive "Backbone" portion at a predetermined frequency, such as executing them once every three times.

[0075] (Variation 1) In Modification 1, submodels corresponding to multiple "Backbones" that extract different features, and a submodel corresponding to the "Head" that integrates the features extracted from each submodel, are distributed to multiple edge devices 1. Processing is performed in parallel on the multiple edge devices 1 to achieve high speed and load balancing, thereby realizing high-precision recognition in the analysis target space.

[0076] Figure 6 is an explanatory diagram of the processing performed by multiple edge devices 1 in Modification Example 1. In Modification Example 1, the process is divided into three edge devices 1: one that executes a process based on the submodel of the "Backbone" part that outputs feature quantity α, another that executes a process based on the submodel of the "Backbone" part that outputs feature quantity β, and yet another that executes a process based on the submodel of the "Backbone" part that outputs feature quantity γ. Furthermore, the processing may be distributed among four edge devices 1, including one that executes a process based on the submodel of the "Head" part. By distributing the submodels to different edge devices 1, the processing load on each edge device 1 can be reduced, enabling high-speed processing.

[0077] In Modification 1, the three edge devices 1 that execute the "Backbone" part of the process each transmit the feature quantities α, β, and γ obtained by the calculation of the learning model M1 (submodel) of the "Backbone" part to the edge device 1 that executes the process based on the submodel of the "Head" part via the first communication unit 12. The edge device 1 that executes the processing using the submodel of the "Backbone" part that outputs the feature quantity γ executes the processing at a predetermined frequency, for example, once every three times, as shown in the first embodiment.

[0078] The processing unit 10 of edge device 1, which executes a process based on the submodel of the "Head" part, temporarily stores feature quantities α, β, and γ if it can acquire them, and then provides them to the learning model M1 of the "Head" part to perform object recognition calculations. In two out of three cases, when the process of the "Backbone" part that outputs feature quantity γ is omitted, the processing unit 10 of edge device 1 uses the stored feature quantity γ to perform object recognition calculations. In this way, the amount of computation is reduced, the processing load is alleviated and high-speed processing is maintained, and delays in object recognition calculations due to congestion in the processing of edge device 1, which executes the computationally intensive "Backbone" part, can be avoided.

[0079] (Second Embodiment) In the first embodiment, processing is omitted at a preset frequency, but in the second embodiment, the frequency is dynamically changed according to the load. The configuration of the image analysis system 100 in the second embodiment is the same as that of the image analysis system 100 in the first embodiment, so the same reference numerals are used for common components and detailed explanations are omitted.

[0080] Figure 7 is a flowchart showing an example of a processing procedure performed by the edge device 1 of the second embodiment. The processing procedure shown in Figure 7 describes the process of the "Backbone" portion, which has a larger computational load than the others.

[0081] When image data is output from camera 4, processing unit 10 acquires it (step S101) and counts the number of times the image data has been output (step S102). Based on a numerical value obtained by a calculation using the count and a load value that changes according to the load, processing unit 10 decides whether or not to execute the following process (step S103). In step S103, processing unit 10 determines whether the remainder when the number of times the image data has been output is divided by the load value is a specific value (for example, set to "1" to include the first time), and if it is determined to be a specific value, it decides to execute the process. Each time image data is output, the count is incremented, and when the count exceeds the load value, the count is reset. Alternatively, it may determine whether the count is equal to or greater than the load value, and if it is determined that the count is equal to or greater than the load value, it may decide to execute the following process.

[0082] If it is determined in step S103 that the following processing should be performed (S103:YES), the processing unit 10 performs preprocessing such as filtering on the output image data (step S104). The processing unit 10 provides the preprocessed image data to the process based on the learning model M1 stored in the memory unit 11 (step S105) to extract features (step S106). The processing unit 10 outputs the features to the "Head" part process (step S107), the "Head" part process performs object recognition processing (step S108), and the recognition result is transmitted to the server 2 by the second communication unit 13 (step S109).

[0083] The processing unit 10 calculates the processing load from the calculations performed by the processor from steps S104 to S107 (step S110), determines the load value depending on whether the processing load is higher than a predetermined value (step S111), and terminates the process. In step S111, the processing unit 10 determines the load value from one of two numerical values. If the processor utilization rate is higher than a predetermined percentage as the processing load, the processing unit 10 determines the load value to the higher numerical value, and if the utilization rate is less than or equal to the predetermined percentage, it determines the load value to the lower numerical value.

[0084] In step S110, for example, the processing unit 10 sets the load value to either "3" or "4". If the processing unit 10 is under heavy load and the processor usage is higher than 50%, the load value is set to "4". Conversely, if the processing unit 10 is under light load and the processor usage is 50% or less, the load value is set to "3". As a result, when the processing is heavy, the process in steps S104-S109 is executed once every four times image data is output from the camera 4, and when the processing is light, the process in steps S104-S109 is executed once every three times.

[0085] If it is determined in step S103 that the following process should not be performed (S103:NO), the processing unit 10 terminates the process.

[0086] Figure 8 is an explanatory diagram of the processing performed by the edge device 1 in the second embodiment. In the second embodiment, as in the example shown in Figure 5, the edge device 1 performs three processes corresponding to the "Backbone" portion in parallel, outputting feature quantities α, β, and γ respectively, and using the feature quantities α, β, and γ, recognizes objects depicted in the image data and outputs their attributes.

[0087] The processes based on the learning model M1 that output feature quantity α and the processes based on the learning model M1 that output feature quantity β are the same as those described in the first embodiment shown in Figure 5. In the image analysis system 100 of the second embodiment, as shown in 8, the "Backbone" portion of the process for a single frame image is distributed and processed concurrently. This speeds up the process of recognizing objects in the image.

[0088] In the second embodiment, the edge device 1 executes the processing procedure shown in Figure 7 for the “Backbone” portion of the process that outputs feature quantity γ. As a result, the edge device 1 performs preprocessing, feature quantity γ extraction, and output of feature quantity γ only when the remainder when the number of iterations is divided by the load value is a specific value (for example, there is no remainder and it is zero). If the load value is set to “4” when the load is heavy and to “3” when the load is light, the process is executed once every four iterations in the former case and once every three iterations in the latter case. This prevents the process of outputting feature quantity γ from becoming congested, even in the second embodiment.

[0089] In the second embodiment, the processing based on the learning model M1 of the "Head" part also omits the "Backbone" part process that outputs feature quantity γ when the load is heavy. Therefore, if features α, β, and γ can be obtained, feature quantity γ is temporarily stored. If the processing unit 10 can obtain all of the features α, β, and γ, it provides them to the learning model M1 of the "Head" part and performs the object recognition calculation. In this edge device 1, if the processing unit 10 omits the "Backbone" part process that outputs feature quantity γ three times out of four, or two out of three times, it provides the features α, β, and γ, including the stored feature quantity γ, to the learning model M1 of the "Head" part and performs the object recognition calculation.

[0090] Thus, in the second embodiment, the amount of computation can be reduced according to the processing load at each point in time to alleviate the processing load and maintain high-speed processing, while also avoiding delays in object recognition calculations due to congestion in the processing of the edge device 1 that executes the computationally intensive "Backbone" portion.

[0091] (Third embodiment) In the third embodiment, the processing load of the "Backbone" portion is reduced by changing the resolution of the images provided to the learning model M1 in the "Backbone" portion. The configuration of the image analysis system 100 in the third embodiment is the same as that of the image analysis system 100 in the first embodiment, so the same reference numerals are used for common components and detailed descriptions are omitted.

[0092] Figure 9 is an explanatory diagram of the processing performed by the edge device 1 in the third embodiment. In the third embodiment, as in the example shown in Figure 5, the edge device 1 performs three processes corresponding to the "Backbone" portion in parallel, outputting feature quantities α, β, and γ respectively. Then, the edge device 1 uses the feature quantities α, β, and γ to recognize objects depicted in the image data and output their attributes through processing based on the learning model M1 in the "Head" portion.

[0093] In the third embodiment, the process using the third “Backbone” learning model M1 that outputs feature quantity γ is computationally intensive, so, as in the first embodiment, its frequency is reduced. In the example shown in Figure 9, when image data is input, the processing unit 10 of this edge device 1 performs preprocessing such as filtering on the frame image once every three times, and then provides the preprocessed image data to the “Backbone” learning model M1 that outputs feature quantity γ to extract feature quantity γ. The process of the “Backbone” part that outputs feature quantity γ is not performed on the image data output from the camera 4 two out of three times. The “Backbone” learning model M1 that outputs feature quantity γ is a type of (sub)model in which it is difficult to obtain accuracy in the output result unless high-resolution images are input.

[0094] In the third embodiment, in the process based on the learning model M1 (submodel) that outputs feature quantity α and the process based on the learning model M1 (submodel) that outputs feature quantity β, when the processing unit 10 acquires image data from the camera 4, it reduces the resolution in preprocessing such as filtering and then performs subsequent processing. At this time, the processing unit 10 reduces the resolution to the extent that the accuracy of calculations using the learning model M1 does not decrease. The degree of resolution reduction should be set based on the learning model M1 and the image data actually captured in the target space.

[0095] Furthermore, in the third embodiment, the processing unit 10 may use a low-resolution image for the input image data for the process based on the learning model M1 (submodel) that outputs feature quantity α and the process based on the learning model M1 (submodel) that outputs feature quantity β, while the processing unit 10 may use a high-resolution image for the input image data for the process based on the learning model M1 (submodel) that outputs feature quantity γ.

[0096] In this way, the third embodiment avoids congestion in the calculations of the edge device 1, further reduces the processing load for outputting the feature quantities α and β respectively, and maintains high-speed processing.

[0097] (Fourth Embodiment) In the fourth embodiment, if the difference between the time-series frame images included in the video and the previous frame image is large, and it is expected that the difference in feature quantities will be large, the computationally intensive "Backbone" operation is selected to be performed. The configuration of the image analysis system 100 in the fourth embodiment is the same as that of the image analysis system 100 in the first embodiment, so the same reference numerals are used for common components and detailed descriptions are omitted.

[0098] Figure 10 is a flowchart showing an example of a processing procedure performed by the edge device 1 of the fourth embodiment. The processing procedure shown in Figure 10 describes the process of the "Backbone" portion, which has a higher computational load than the others.

[0099] The processing unit 10 acquires image data from the camera 4 (step S121), stores it in memory (step S122), and calculates the difference with the previous image data (step S123). The difference can be calculated simply by calculating the difference in pixel values ​​between the images, or it can be calculated using a known algorithm that divides the region into background and foreground before calculating the difference.

[0100] The processing unit 10 determines whether the difference is greater than or equal to a predetermined value (step S124). If it is determined that the difference is less than the predetermined value (S124: NO), the processing unit 10 selects a process with less computation (step S125). The processing unit 10 performs preprocessing on the image data acquired in step S121 using the selected process (step S126). The processing unit 10 provides the preprocessed image data to the selected process (step S127) to extract features (step S128). The processing unit 10 outputs the features to the "Head" part process (step S129), the "Head" part process performs object recognition processing (step S130), the recognition result is sent to the server 2 by the second communication unit 13 (step S131), and processing for one image data is completed.

[0101] If, in step S124, it is determined that the difference is greater than or equal to a predetermined value (S124: YES), the processing unit 10 selects the process of the learning model M1 which requires a large amount of computation (step S132). The processing unit 10 performs preprocessing on the image data acquired in step S121 using the selected process (step S133). The processing unit 10 provides the preprocessed image data to the selected process (step S134) and extracts features (step S135). The processing unit 10 outputs the features to the "Head" part of the process (S129) and executes steps S130-S131 to terminate the process.

[0102] In step S125, the processing unit 10 reads and executes the model with the smaller computational load from among the different learning models (submodels) M1, M1. When distributing the submodels to multiple edge devices 1 (modification example 1), another edge device 1 may be selected to perform the computationally intensive processing. In step S132, the processing unit 10 may read and execute the model with the larger computational load, or it may select another edge device 1 to perform the computationally intensive processing.

[0103] Figure 11 is an explanatory diagram of the processing performed by the edge device 1 in the fourth embodiment. In the fourth embodiment, of the three processes corresponding to the “Backbone” portion, the process that outputs feature quantity α is performed for all frame images, but for the processes that output feature quantities β and γ, only one of them is performed according to the processing procedure shown in Figure 10. Therefore, in the process corresponding to the “Head” portion in the fourth embodiment, if the time difference of the image data is large, the process of recognizing an object using feature quantities α and γ is performed, and if the time difference of the image data is small, the process of recognizing an object using feature quantities α and β is performed. Note that feature quantities β and γ are the same type of feature quantities used in the same recognition process, but when an object is recognized using feature quantities α and γ, the accuracy of the object recognition result by the “Head” is higher than when an object is recognized using feature quantities α and β.

[0104] In the fourth embodiment, specifically, the processing unit 10 of the edge device 1 determines whether the difference (time difference) between the acquired image data and the previous image data is large or not (S124). If the processing unit 10 determines that the difference is less than a predetermined value, i.e., the difference is small (S124: NO), it provides the image data to a process using a learning model M1 that outputs feature quantities β (S127) and extracts feature quantities β (S128). The processing unit 10 outputs the feature quantities α and β obtained by the process to the process for the "Head" portion.

[0105] If the difference is greater than or equal to a predetermined value, i.e., if the difference is large (S124: YES), the processing unit 10 provides image data to the process that outputs feature quantity γ (S134) and extracts feature quantity γ (S135). The processing unit 10 outputs the feature quantity α and feature quantity γ obtained by the process to the process for the "Head" portion.

[0106] Edge device 1 performs object recognition processing using feature α, which can be acquired regardless of the size of the difference, and either feature β or feature γ. Thus, in this fourth embodiment, the process of outputting feature γ, which requires a large amount of computation, is executed only when the difference from the previous frame image is large and outputting feature γ is effective. This avoids the process of outputting feature γ, which requires a large amount of computation, from becoming congested.

[0107] In the image analysis system 100 of the fourth embodiment, as shown in Modification 1, the processing using the submodel of the “Backbone” portion for a single frame image may be distributed to multiple edge devices 1 and processed simultaneously. This further speeds up the processing of recognizing objects in the image.

[0108] (Modification 2) In the fourth embodiment, when the time difference of the image data is large, the process of recognizing an object using feature quantities α and γ is executed, and when the time difference of the image data is small, the process of recognizing an object using feature quantities α and β is executed. However, the process is not limited to this, and the execution frequency of the process of recognizing an object using feature quantities α and γ may be increased when the time difference of the image data is large, and the execution frequency of the process of recognizing an object using feature quantities α and β may be increased when the time difference of the image data is small.

[0109] Figure 12 is a flowchart showing an example of the processing procedure performed by edge device 1 in modified example 2. Of the processing procedures shown in Figure 12, steps that are common to the processing procedures shown in Figure 10 are given the same step numbers and detailed explanations are omitted.

[0110] When the processing unit 10 acquires image data (S121), it counts the number of times the image data has been output (step S141). The processing unit 10 then stores the image data in memory (S122), calculates the difference with the previous image data (S123), and determines whether the difference is greater than or equal to a predetermined value (S124).

[0111] If the processing unit 10 determines that the difference is greater than or equal to a predetermined value (S124: YES), it sets a relatively large frequency (step S142) and proceeds to step S144.

[0112] If the processing unit 10 determines that the difference is less than a predetermined value (S124: NO), it sets a relatively small frequency (step S143) and proceeds to step S144.

[0113] In step S144, the processing unit 10 determines whether or not to execute the process of the computationally intensive learning model M1 based on a numerical value obtained by a calculation using the number of times counted in step S141 and the frequency which changes according to the difference (step S144). In step S144, if the frequency was set to once every two times in step S142 and to once every four times in step S143, the processing unit 10 determines whether or not the counted number corresponds to the frequency.

[0114] If the processing unit 10 determines in step S144 to execute the computationally intensive process of learning model M1 (S144: YES), it executes the processes in steps S133-S135 and proceeds to step S129. If the processing unit 10 determines in step S144 not to execute the computationally intensive process of learning model M1 (S144: NO), it executes the processes in steps S126-S128 and proceeds to step S129.

[0115] As shown in the explanatory diagram of the process in Figure 11, if the difference is determined to be greater than or equal to a predetermined value, the processing unit 10 executes the process of outputting feature quantity γ once every two times, and if the difference is determined to be less than the predetermined value, the process of outputting feature quantity γ executes once every four times.

[0116] In the image analysis system 100 of Modified Example 2, depending on the magnitude of the difference in the actual output image data, high-precision processing is performed even if the computational load is large during the time period after the image data requiring detailed analysis has been captured. Conversely, in the image analysis system 100 of Modified Example 2, during the time period when there is no change from the background, a low-computational process that saves energy is executed, making it possible to reduce the processing load while maintaining high-precision recognition processing.

[0117] (Fifth embodiment) In the fifth embodiment, the process of executing multiple learning models (submodels) M1 corresponding to the "Backbone" portion is distributed across different edge devices 1, and the results of one of the processes are held and referenced in a memory functioning as a buffer (buffer memory). The configuration of the image analysis system 100 in the fifth embodiment is the same as that of the image analysis system 100 in the first embodiment, so the same reference numerals are used for common components and detailed descriptions are omitted.

[0118] Figure 13 is an explanatory diagram of the processing performed on multiple edge devices 1 in the fifth embodiment. In the fifth embodiment, among the multiple submodels M1 that constitute the learning model M0 shown in Figure 4, the process of the submodel M1 of the "Backbone" portion that outputs the feature quantity γ, which tends to have a heavy processing load, is executed on different edge devices 1.

[0119] As shown in Figure 13, the first edge device 1 performs processing based on a learning model M1 that outputs feature α, processing based on a learning model M1 that outputs feature β, and processing based on a learning model M1 for the "Head" portion that uses feature γ in addition to features α and β. In the fifth embodiment, the second edge device 1 performs processing based on a learning model M1 that outputs feature γ.

[0120] The processing unit 10 of the first edge device 1 receives image data from camera 4 when it is sent to the local network LN, and provides the image data to a process based on a learning model M1 (submodel) that outputs feature quantities α of the "Backbone" portion, and to a process based on a learning model M1 (submodel) that outputs feature quantities β. Each process provides the extracted feature quantities α and β to the process in the "Head" portion. In the process in the "Head" portion, the processing unit 10 obtains feature quantities α and β from the "Backbone" process for all frames of images, obtains feature quantity γ which is received from the second edge device 1 and buffered in memory, and performs object recognition processing. Feature quantity γ is buffered into memory at any time, by receiving and storing it each time it is transmitted from the second edge device 1.

[0121] The second edge device 1 also performs preprocessing, such as filtering, on the target image data each time image data is output from camera 4. The processing unit 10 of this edge device 1 feeds the preprocessed image data to a process based on the learning model M1 (submodel) that outputs feature quantities γ, and extracts the feature quantities γ. Each time the processing unit 10 obtains feature quantities γ, it executes a process to write them to the memory of the processing unit 10 of the first edge device 1. As a result, regardless of the processing timing of the first edge device 1 and the second edge device 1, the memory of the first edge device 1 stores the feature quantities γ extracted in the most recent processing.

[0122] In the example shown in Figure 13, the timing of the process based on the learning model M1 (submodel) that outputs feature α, the process based on the learning model M1 (submodel) that outputs feature β, and the process based on the learning model M1 (submodel) that outputs feature γ are not synchronized on a different device, and even if the timing is off, the "Head" part of the process can still be executed. The processing unit 10 of the first edge device 1 can continue the object recognition calculation in the "Head" part of the process without waiting for the output of feature γ.

[0123] In the fifth embodiment, even if there is a bottleneck in the computationally intensive "Backbone" portion of the process, it is possible to avoid a situation where the "Head" portion of the process cannot be executed.

[0124] (Sixth Embodiment) In the sixth embodiment, similar to the fifth embodiment, the process of executing multiple learning models (submodels) M1 corresponding to the "Backbone" portion is distributed to different edge devices 1, and the results of one of the processes are held and referenced in a memory functioning as a buffer (buffer memory). The configuration of the image analysis system 100 in the sixth embodiment is the same as that of the image analysis system 100 in the first embodiment, so the same reference numerals are used for common components and detailed descriptions are omitted.

[0125] Figure 14 is an explanatory diagram of the processing performed by multiple edge devices 1 in the sixth embodiment. In the sixth embodiment, similar to the fifth embodiment, the process of the submodel M1 of the “Backbone” portion that outputs the feature quantity γ, which tends to have a heavy processing load, is executed on different edge devices 1, among the multiple submodels M1 that constitute the learning model M0 shown in Figure 4.

[0126] In the sixth embodiment, as shown in Figure 14, the first edge device 1 executes processing based on a learning model M1 that outputs feature quantity α, processing based on a learning model M1 that outputs feature quantity β, and processing based on a learning model M1 for the "Head" portion that uses feature quantities α, β, and feature quantity γ. In the sixth embodiment, the edge device 1 can switch between executing processing based on a learning model M1 that performs image recognition processing with two feature quantities α and β as input, and processing based on a learning model M1 that performs image recognition processing with three feature quantities α, β, and γ as input.

[0127] In the sixth embodiment, the processing unit 10 of the first edge device 1 acquires image data when it is sent from the camera 4 to the local network LN, and provides the image data to a process based on a learning model M1 (submodel) that outputs feature quantities α of the "Backbone" portion, and to a process based on a learning model M1 (submodel) that outputs feature quantities β. Each process can then provide the extracted feature quantities α and β to the "Head" portion.

[0128] The second edge device 1 also acquires image data when it is output from camera 4 and performs preprocessing such as filtering on the image data. The processing unit 10 of this edge device 1 also temporarily stores in memory the time information of the frame image output from camera 4 or data that identifies the image data. The processing unit 10 feeds the preprocessed image data to a process based on the learning model M1 (submodel) that outputs feature quantities γ, and extracts the feature quantities γ. Each time a feature quantity γ is obtained, the processing unit 10 executes a process to write the feature quantity γ and the time information of the target frame image or data that identifies the image data to the memory of the processing unit 10 of the first edge device 1. As a result, the memory of the first edge device 1 stores the feature quantity γ extracted in the latest processing and the time information of the target frame image or data that identifies the image data.

[0129] In the sixth embodiment, the processing unit 10 of the first edge device 1 obtains feature quantities α and β from the “Backbone” process for all frame images and switches between two “Head” processes: one that recognizes an object using two feature quantities α and β, and another that recognizes an object using three feature quantities α, β, and γ. The processing unit 10 compares the time information or image data identifying the target frame image of feature quantities α and β with the time information or image data associated with feature quantity γ, which is received from the second edge device 1 and buffered in memory. If the time difference is within a predetermined range, the processing unit 10 executes the process of recognizing an object using three feature quantities α, β, and γ, including the buffered feature quantity γ. If the time difference is outside the predetermined range, the processing unit 10 executes the process of recognizing an object using two feature quantities α and β. Here, the predetermined range is set in advance in the storage unit 11, for example, 2 to 3 seconds or more.

[0130] In the sixth embodiment, even if the computational load for extracting feature quantity γ in the second edge device 1 is heavy, causing processing delays or process stoppages, object recognition calculations can still be performed using the most recently derived feature quantity γ, as long as it is within an acceptable range. If the process that outputs feature quantity γ has stopped and the time difference is outside the acceptable range, the process of recognizing the object can be continued using the two feature quantities α and β excluding feature quantity γ, thereby avoiding a halt in the recognition process.

[0131] Thus, even in the sixth embodiment, it is possible to avoid a situation where the processes in the "Head" part cannot be executed even if the processes in the "Backbone" part, which have a large amount of computation, become congested or stop.

[0132] In the sixth embodiment, if the time difference between the data identifying the time information or image data of the target frame image for features α and β and the data identifying the time information or image data associated with feature γ buffered in memory falls outside a predetermined range, the stack in the processing of the second edge device 1 may be released. In this case, the processing of the first edge device 1 and the processing of the second edge device 1 may be reset and restarted.

[0133] (Seventh Embodiment) In the seventh embodiment, the submodel is divided into a "Backbone" part that extracts features from image data and a "Head" part that integrates the extracted features and performs recognition processing. The "Backbone" submodel, which requires less computation than the others, is tuned using image data obtained in the analysis space. The configuration of the image analysis system 100 in the seventh embodiment is the same as that of the image analysis system 100 in the first embodiment, so common components are denoted by the same reference numerals and detailed explanations are omitted.

[0134] Figure 15 is an explanatory diagram of the processing performed by the edge device 1 in the seventh embodiment. In the seventh embodiment, the edge device 1 performs the processes corresponding to the “Backbone” portion in parallel, similar to the first to fourth embodiments. The three “Backbone” processes each output feature quantities α, β, and γ, and the process corresponding to the “Head” portion performs object recognition using the feature quantities α, β, and γ.

[0135] Here, the process that outputs feature β and the process that outputs feature γ will output consistent feature β and feature γ when performed on the same object. Feature β and feature γ do not need to have identical parameters; they just need to be comparable features.

[0136] In the image analysis system 100 of the seventh embodiment, the edge device 1 simultaneously extracts three different feature quantities α, β, and γ, as shown in Figure 15, and the server 2 performs learning processing using video actually captured in the space to be analyzed. In particular, for the learning model M1 (submodel) that outputs the feature quantity β, which requires relatively little computation, the number of parameters has been reduced by distillation, etc., so on-site learning is performed to fine-tune the parameters of the learning model M1.

[0137] Figures 16 and 17 are flowcharts showing an example of the learning process procedure by the edge device 1 of the seventh embodiment. The processing unit 10 of edge device 1, or the processing unit 10 of another edge device 1, selects the data to be learned and proceeds with learning, as shown below.

[0138] When image data is output from camera 4, processing unit 10 acquires it (step S171) and provides the image data to each process (step S172).

[0139] In each process, the processing unit 10 performs preprocessing such as filtering (step S173), provides the preprocessed image data to the learning model M1 of the selected process (step S174), and extracts features (step S175).

[0140] The processing unit 10 provides the acquired feature quantities to the "Head" part of the process (step S176). The processing unit 10 then uses the "Head" part of the process to perform object recognition (step S177), and the second communication unit 13 transmits the recognition result to the server 2 (step S178).

[0141] The processing unit 10 determines whether the learning progress of the learning model M1 to be learned is at or above a predetermined stage (step S179). In step S179, the learning model M1 to be learned is, in the example of the seventh embodiment, a learning model M1 that outputs feature β, which has a relatively small computational cost among the comparable feature quantities β and feature quantities γ. The processing unit 10 may perform the determination in step S179 whether the learning progress is at or above a predetermined stage based on the number of iterations or a difference evaluation (S182) described later.

[0142] If it is determined that the learning progress is below a predetermined stage (S179: NO), the processing unit 10 selects comparable features from the features extracted in each process in step S175 (step S180). The processing unit 10 compares the selected features (step S181) and determines whether the difference between the features is within a predetermined range (step S182).

[0143] Whether or not the differences in features are within a predetermined range can also be used as a criterion for determining whether the learning process described above has progressed. In this case, if the differences in features are within a predetermined range, it can be determined that the learning has progressed to the point where even the "Backbone" process, which has less computational power, does not contradict the results of the other "Backbone" processes, which have more parameters, and can obtain recognition results with sufficient accuracy.

[0144] If the processing unit 10 determines that the difference in feature quantities is outside a predetermined range (S182: NO), it sends the image data after preprocessing in step S173 to the server 2 as training images and stores them in the storage unit 21 of the server 2 (step S183). When the processing unit 20 of the server 2 has accumulated enough training data to perform (fine) tuning (retraining) of the low-computation-intensive training model M1, it uses the training data (including the above image data) stored in the storage unit 21 to perform (fine) tuning of the low-computation-intensive training model M1.

[0145] In step S179, if the learning progress stage is above a predetermined stage (S179: YES) and it is determined that learning has progressed sufficiently, the processing unit 10 omits the processing in steps S180-S183 and terminates the processing for one image data set.

[0146] If, in step S182, it is determined that the difference in features is within a predetermined range (S182: YES), the processing in step S183 is omitted, and the processing for one image data set is terminated.

[0147] In the seventh embodiment, the selection of the three processes corresponding to the “Backbone” portion that extracts features from image data may be appropriately selected according to the processing load and specified frequency. Also, in the seventh embodiment, if sufficient training data has been accumulated in the storage unit 21 of the server 2, the training model M1 used in the process that outputs the feature γ may be retrained.

[0148] As described above, in the image analysis system 100 of the seventh embodiment, when retraining a submodel of "Backbone" to adapt it to the operating environment using a small amount of training data, only the submodel with a relatively low computational load (for example, the submodel of "Backbone" that outputs the above-mentioned feature quantity β) is retrained. The reason for this is due to the following technical concept.

[0149] In other words, when retraining (fine-tuning) a learning model used on edge device 1 installed at a site (store, etc.) to adapt it to the field environment, if the amount of training data (training data) for retraining, which consists of images collected at the site (store, etc.) with tagged data, is small, it is difficult to complete the retraining (tuning) of a large "Backbone" (or its submodel) with a large number of parameters. Furthermore, if tuning is performed on a large "Backbone" using a small amount of training data, it is highly likely that the performance will actually decrease. Therefore, when tuning to adapt to the field environment, if the amount of training data using images collected at the site (store, etc.) is small, tuning is not performed on a large "Backbone" (or its submodel) with a large number of parameters, and only fine-tuning is performed on a small "Backbone" (or its submodel) with a small number of parameters.

[0150] This allows the large "Backbone," which is trained using training data based on general images (such as images from multiple other stores) rather than images collected in the field environment, to maintain its high performance in general environments. On the other hand, the smaller "Backbone," which can be fine-tuned using a small amount of training data, can be adapted to the field environment by fine-tuning it with training data using images collected in the field (stores, etc.). When there is a large amount of training data using images collected in the field (stores, etc.), the large "Backbone" (or its submodels) will also be retrained (tuned).

[0151] Furthermore, in step S183, if the processing unit 10 of the edge device 1 determines that the difference in features is outside a predetermined range (S182:NO), the reason why it sends the image data after preprocessing in step S173 to the server 2 as a training image and stores it in the storage unit 21 of the server 2 is as follows: In other words, as described above, when feature β and feature γ are comparable features (for example, features of the same type), if the difference between feature β and feature γ, which are the features output when images collected on-site (store, etc.) are input to a small (low-precision) "Backbone" and a large (high-precision) "Backbone," is large (S182:NO), it means that the processing of the small submodel (the "Backbone" that outputs feature β) is not working well in this on-site environment. Therefore, we utilize this to select images (from the field) where the difference between feature β and feature γ is large, and use these images as (fine) tuning images (to adapt to the field environment) for a smaller submodel ("Backbone" that outputs feature β).

[0152] The embodiments disclosed above are illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, and all modifications within the meaning and scope equivalent to the claims are included. [Explanation of symbols]

[0153] 100 Image Analysis Systems 1. Edge devices 10 Processing Unit 12. First Communications Department 13. Second Communications Department 2 servers

Claims

1. Computers Using a pre-trained model that is trained to output the recognition result of objects in an image when image data is input, The trained model is divided into a number of submodels corresponding to a backbone portion that extracts features from image data, and a submodel corresponding to a head portion that is trained to output recognition results of objects in the image data when the features of the image data extracted by each of the submodels are input. The aforementioned computer, A process is performed to extract features from the time-series input image data based on each of the multiple submodels corresponding to the backbone portion. Among the multiple submodels corresponding to the aforementioned backbone portion, the execution frequency of processes based on submodels with a higher computational load than others is reduced. Among the submodels corresponding to the head portion, one is selected that uses the features extracted by the submodel with a higher computational load than the others, and the other is selected that does not use the features extracted by the submodel with a higher computational load than the others. Perform recognition processing based on the selected submodel. Information processing methods.

2. The computer is The recognition process is performed using at least a portion of the features extracted by the multiple submodels corresponding to the backbone portion, including the submodel with a higher computational load than the others, and based on the submodel corresponding to the head portion. The information processing method according to claim 1.

3. The aforementioned computer, A process based on a submodel with a higher computational load than others is executed at a predetermined ratio relative to the frame rate of the image data. The information processing method according to claim 1 or 2.

4. The computer changes the execution frequency of processes based on submodels with a larger computational load than others, according to the load on the computer. The information processing method according to claim 1 or 2.

5. The computer uses different resolutions for the image data input to each of the multiple submodels corresponding to the backbone portion. The information processing method according to claim 1 or 2.

6. The aforementioned computer, Among the image data input in a time series, it is determined whether the time difference between the target image data and the previous image data is greater than or equal to a predetermined amount. If it is determined that the amount exceeds a predetermined amount, the process based on the submodel with a larger computational load than the others will be executed. If it is determined that the amount is less than a predetermined amount, the process based on another submodel will be executed, replacing the submodel with one that has a larger computational load than the others. The information processing method according to claim 1.

7. The aforementioned computer, Among the image data input in a time series, it is determined whether the time difference between the target image data and the previous image data is greater than or equal to a predetermined amount. If it is determined that the amount exceeds a predetermined amount, the execution frequency of the process based on the submodel with a larger computational load than others will be increased. If it is determined that the amount is below a predetermined level, the execution frequency of the process based on the submodel with a higher computational load than others will be reduced. The information processing method according to claim 1 or 2.

8. The processes based on multiple submodels corresponding to the aforementioned backbone portion are distributed across different computers. A computer that performs a process based on a submodel with a higher computational load than others outputs the features extracted by the submodel to other computers each time the process is executed. The aforementioned other computer, Each time the aforementioned feature quantity is output, a buffer memory is used to update the aforementioned feature quantity. The recognition process is performed using the features extracted by the process based on other submodels and the features stored in the buffer memory. The information processing method according to claim 1 or 2.

9. A computer, A pre-trained model, which is trained to output recognition results of objects in an image when image data is input, is divided into multiple submodels based on the features extracted within the model. A process is performed to extract features from the time-series input image data based on each of the multiple submodels. Among the aforementioned submodels, the execution frequency of processes based on the submodel with a higher computational load than the others is reduced. In an information processing method that performs recognition processing using each feature quantity extracted by the process based on the aforementioned multiple submodels, The processes based on the aforementioned multiple submodels are distributed across different computers. A computer that performs a process based on a submodel with a larger computational load than others outputs the features extracted by the submodel to other computers each time the process is performed, associating them with identification data that identifies the timing at which the image data corresponding to the features was captured. The aforementioned other computer, Each time the aforementioned feature quantities and the identification data of the image are output, a buffer memory is used to update the aforementioned feature quantities and identification data. The identification data that identifies the timing at which the image data, which is the target of the features extracted by a process based on other submodels, was taken, is compared with the identification data stored in the buffer memory. Determine whether the time difference in image data is within a predetermined range. Only if it is determined that the features fall within a predetermined range, the recognition process is executed using the features extracted by the process based on the other submodels and the features stored in the buffer memory. Information processing methods.

10. If the other computer determines that the time difference is outside a predetermined range, it resets the process based on the submodel with a larger buffer memory and computational load than the others, and restarts the processes based on multiple submodels. The information processing method according to claim 9.

11. When the computer retrains multiple submodels corresponding to the backbone portion to adapt them to the operating environment using a small amount of training data, it retrains only the submodels among the multiple submodels that require relatively less computation. The information processing method according to claim 1 or 2.

12. Using a pre-trained model that is trained to output the recognition result of objects in an image when image data is input, The trained model is divided into a number of submodels corresponding to a backbone portion that extracts features from image data, and a submodel corresponding to a head portion that is trained to output recognition results of objects in the image data when the features of the image data extracted by each of the submodels are input. A process is performed to extract features from the time-series input image data based on each of the multiple submodels corresponding to the backbone portion. Among the multiple submodels corresponding to the aforementioned backbone portion, the execution frequency of processes based on submodels with a higher computational load than others is reduced. Among the submodels corresponding to the head portion, one is selected that uses the features extracted by the submodel with a higher computational load than the others, and the other is selected that does not use the features extracted by the submodel with a higher computational load than the others. Perform recognition processing based on the selected submodel. An information processing device equipped with a processing unit.

13. On the computer, Using a pre-trained model that is trained to output the recognition result of objects in an image when image data is input, The trained model is divided into a number of submodels corresponding to a backbone portion that extracts features from image data, and a submodel corresponding to a head portion that is trained to output recognition results of objects in the image data when the features of the image data extracted by each of the submodels are input. A process is performed to extract features from the time-series input image data based on each of the multiple submodels corresponding to the backbone portion. Among the multiple submodels corresponding to the aforementioned backbone portion, the execution frequency of processes based on submodels with a higher computational load than others is reduced. Among the submodels corresponding to the head portion, one is selected that uses the features extracted by the submodel with a higher computational load than the others, and the other is selected that does not use the features extracted by the submodel with a higher computational load than the others. Perform recognition processing based on the selected submodel. A computer program that executes a process.

Images