Method for inspecting model, inspection apparatus, inspection system, and computer program

Abstract

To provide a method for inspecting a model, an inspection apparatus, an inspection system, and a computer program.SOLUTION: In the method for inspecting a model, when image data or data obtained from image data is input, a first trained neural network model, which has been trained to output a detection result or recognition result of an object appearing in the image, is deployed on a computer. A second trained neural network model, which outputs the same type of data as the first trained neural network model when the same input data is provided, is deployed on the same or a separate computer. When the same input data is input to both the first trained neural network model and the second trained neural network model, the data output from the first trained neural network model and the data output from the second trained neural network model are compared, and the comparison result is output.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a method, an apparatus, a system, and a computer program for inspecting a model. [Background technology]

[0002] Image recognition systems are used that use a camera that captures the inside of a store or a specific area within its field of view to detect people and other objects in the images captured by the camera and output recognition results for the detected objects. Such image recognition systems use a trained model (hereinafter referred to as a "trained model") that uses a neural network (hereinafter referred to as a "neural network") that is trained to input target image data and output detection and recognition results for objects in the image data.

[0003] Thanks to improvements in computing power and hardware technology, it is now possible to process images using learning models on edge devices with relatively limited computing resources, rather than on a server with abundant computing resources that collects data and processes the images. For example, AI (Artificial Intelligence) cameras are now being used that incorporate learning models into the camera itself and output image data along with the recognition results. Processing using learning models on edge devices makes it possible to maintain functionality even if there are problems with communication with the server.

[0004] Patent Document 1 discloses that an edge device with limited resources uses image data captured in a target space, such as a store, to which a learning model is applied, to carry out learning specific to the target space, thereby maintaining the accuracy of image analysis.Since there is a shortage of learning data to improve accuracy by carrying out learning using only image data captured in the target space, Patent Document 1 discloses that a learning server installed in the management department of the store in the target space uses image data captured in multiple stores to generate a learning dataset for inference that runs on an edge device installed in the store. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2024-031602 Summary of the Invention [Problem to be solved by the invention]

[0006] There are systems in stores that display advertisements on signage displays according to the attributes of customers visiting the store. Such systems identify the attributes of customers passing in front of the display. To identify these attributes, a learning model is used that outputs attributes when a photograph of a customer is input, so not only accuracy but also fast response is required.

[0007] To use a learning model on an edge device, it is first necessary to reduce the amount of calculation, for example by reducing the number of parameters. Reducing the amount of calculation tends to result in a decrease in accuracy. To maintain accuracy using a learning model with reduced calculations on an edge device, as disclosed in Patent Document 1, it is necessary to efficiently perform (fine) tuning (relearning) of the pre-trained learning model that is suited to the differences in the characteristics of each application location. Because edge devices have limited computing and storage resources, it is difficult to accumulate a large amount of image data or perform large-volume calculations. Therefore, various methods have been proposed, including Patent Document 1.

[0008] When tuning a learning model that has been provisionally deployed in a target space using actual image data captured in the target space, it is not easy to immediately evaluate the validity of the detection accuracy and recognition accuracy. It may be that, only after tuning, it is determined that the provisionally deployed learning model is not suited to the target space and that it will be difficult to achieve the target detection accuracy or recognition accuracy. Therefore, if it is determined that the learning model needs to be reselected and retuned, a loss of effort and cost will occur.

[0009] The present invention aims to provide a model inspection method, inspection device, inspection system, and computer program that can appropriately arrange and optimize a learning model. [Means for solving the problem]

[0010] A model inspection method according to one embodiment of the present disclosure deploys a second trained neural network model on the same or a different computer as a computer on which a first trained neural network model is deployed, the second trained neural network model outputting the same type of data as the first trained neural network model when the same input data as the first trained neural network model is input, the second trained neural network model comparing the data output from the first trained neural network model with the data output from the second trained neural network model when the same input data is input to the first trained neural network model and the second trained neural network model, and outputting the comparison result.

[0011] In a model testing method according to an embodiment of the present disclosure, the validity of the detection accuracy or recognition accuracy using a first trained neural network model (the model to be tested) can be quantified and confirmed by comparing it with the detection results or recognition results using a second trained neural network model (the model for testing). Based on the results of the comparison, further tuning or model reselection can be automatically performed via a server.

[0012] A model inspection method according to one embodiment of the present disclosure includes a second computer connected to a first computer using a first trained neural network model that has been trained to output detection results or object recognition results for objects appearing in an image when image data or data obtained from image data is input, the second computer acquires a second trained neural network model that outputs the same type of output data as the first trained neural network model when input with the same input data as the first trained neural network model, acquires input data to be input to the first trained neural network model, compares the data output when the input data is input to the second trained neural network model with the data output from the first trained neural network model, and outputs the comparison result.

[0013] In a model testing method according to an embodiment of the present disclosure, the validity of detection accuracy or recognition accuracy using a first trained neural network model (test target model) in a first computer can be quantified and confirmed by comparing it with processing in a second computer using a second trained neural network model (test model). Based on the results of the comparison, further tuning or model reselection can be automatically performed via a server.

[0014] In the model checking method according to the embodiment of the present disclosure, it is preferable that the second computer has computational resources with higher specifications than the first computer.

[0015] In a model inspection method according to an embodiment of the present disclosure, it is possible to evaluate the accuracy of a model to be inspected in an edge device with inferior computing resources by comparing it with the results of calculations using an inspection model on a high-spec computer with similar input and output.

[0016] In one embodiment of the model inspection method of the present disclosure, the second computer may determine whether the difference between the data output when the acquired input data is input to the second trained neural network model and the data output from the first trained neural network model is greater than or equal to a predetermined amount, and if it is determined that the difference is greater than or equal to the predetermined amount, store the input data and the data output from the second trained neural network model in a memory unit as training data.

[0017] In a model inspection method according to one embodiment of the present disclosure, if different results are output between the model to be inspected and a high-precision inspection model with a large number of parameters, the input data and the output from the inspection model are retained as learning data, making it possible to efficiently tune the model to be inspected after the fact.

[0018] In the model inspection method according to the embodiment of the present disclosure, the first trained neural network model may be retrained using the training data stored in the storage unit.

[0019] In a model inspection method according to an embodiment of the present disclosure, the first trained neural network model can be tuned and optimized using a data set of input data and output data that has been retained as training data.

[0020] A model inspection method according to an embodiment of the present disclosure may include selecting an alternative model of the first trained neural network model or the second trained neural network from the published trained neural networks based on the comparison result, and deploying the selected alternative model to the first computer or the second computer.

[0021] In the model inspection method of one embodiment of the present disclosure, if the comparison results show a large difference in results between the model to be inspected and a high-precision inspection model with a large number of parameters, the models (especially the model to be inspected) can be swapped early and the models can be appropriately positioned.

[0022] In one embodiment of the model inspection method of the present disclosure, the first trained neural network model or the second trained neural network is stored in a database of a server according to shooting conditions including the subject or shooting situation of the image data corresponding to the input data, and the server may select a model corresponding to the specified shooting conditions and transmit the selected model to the first computer or the second computer.

[0023] In the model inspection method according to an embodiment of the present disclosure, the server can appropriately select and arrange the model according to the shooting conditions.

[0024] In one embodiment of the model inspection method of the present disclosure, the server may store an inspection history using the second trained neural network, and select a second trained neural network for a new trained neural network to be inspected from the inspection history based on the shooting conditions in the environment in which the trained neural network of the new test object is used.

[0025] The model inspection method according to an embodiment of the present disclosure makes it possible to select an inspection model that has a proven track record in inspection.

[0026] An inspection device according to one embodiment of the present disclosure is connected to a computer that uses a first trained neural network model that has been trained to output detection results or object recognition results for objects appearing in an image when image data or data obtained from image data is input, and inspects the first trained neural network model.The inspection device includes a processing unit that acquires a second trained neural network model that outputs the same type of output data as the first trained neural network model when the same input data as the first trained neural network model is input, acquires input data to be input to the first trained neural network model, compares the data output when the input data is input to the second trained neural network model with the data output from the first trained neural network model, and outputs the comparison result.

[0027] An inspection system according to one embodiment of the present disclosure includes a second computer connected to a first computer that uses a first trained neural network model that has been trained to output detection results or object recognition results for objects appearing in an image when image data or data obtained from image data is input, and a server that transmits and receives data to the second computer via communication, wherein the second computer acquires a second trained neural network model that outputs the same type of output data as the first trained neural network model when input with the same input data as the first trained neural network model, acquires input data to be input to the first trained neural network model, compares the data output when the input data is input to the second trained neural network model with the data output from the first trained neural network model, and outputs the comparison result to the server.

[0028] In an inspection system according to one embodiment of the present disclosure, the server may store multiple trained models to be used as the first trained neural network model or the second trained neural network model, categorized by shooting conditions including the subject or shooting situation of the image data corresponding to the input data of the trained model, select a trained model corresponding to the specified shooting conditions, and transmit the selected model to the first computer or the second computer.

[0029] A computer program according to one embodiment of the present disclosure causes a second computer connected to a first computer using a first trained neural network model that has been trained to output detection results or object recognition results for objects appearing in an image when image data or data obtained from image data is input, to acquire a second trained neural network model that outputs the same type of output data as the first trained neural network model when the same input data as the first trained neural network model is input, acquire input data to be input to the first trained neural network model, compare the data output when the input data is input to the second trained neural network model with the data output from the first trained neural network model, and output the comparison result. [Effects of the Invention]

[0030] According to the present disclosure, the goodness of fit can be quantified by testing, which allows for appropriate deployment and optimization of trained neural network models on computers with relatively low computational resources. [Brief explanation of the drawings]

[0031] [Figure 1] FIG. 1 is a schematic diagram of an image analysis system. [Figure 2] FIG. 2 is a block diagram showing a configuration of an edge device. [Figure 3] FIG. 2 is a block diagram showing the configuration of a server. [Figure 4] FIG. 2 is a block diagram showing the configuration of a client; [Figure 5] FIG. 2 is a block diagram showing the configuration of an inspection device. [Figure 6] FIG. 2 is a block diagram showing the configuration of a server. [Figure 7] FIG. 2 is a functional block diagram showing functions in the inspection system. [Figure 8] 10 is a flowchart illustrating an example of a processing procedure performed by the inspection system. DETAILED DESCRIPTION OF THE INVENTION

[0032] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present disclosure will be specifically described with reference to the drawings showing embodiments thereof. In the following embodiments, an image analysis system to which the inspection system of the present disclosure is applied will be described.

[0033] 1 is a schematic diagram of an image analysis system 100. The image analysis system 100 includes one or more cameras 4 installed in a space to be analyzed, such as a store, multiple edge devices 1 connected to the cameras 4, a server 2 that can be connected to the edge device 1 for communication, and a client 3 that can be connected to the server.

[0034] One or more cameras 4 and multiple edge devices 1 can be connected to each other via a wireless or wired local network LN. The local network LN may be a wired LAN or a wireless network such as WiFi or Bluetooth (registered trademark).

[0035] One or more cameras 4 use image elements that are responsive to visible light and / or near-infrared light and output image data. The cameras 4 output image data in time series at a rate of several fps to several tens of fps. The cameras 4 are installed so as to look down from the top of the space to be analyzed, such as on the ceiling or shelves. The cameras 4 may be ceiling-mounted cameras that can cover the entire space in a 360-degree field of view. The cameras 4 sequentially transmit the image data to the edge device 1 via the local network LN.

[0036] Some or all of the multiple edge devices 1 can be connected to the server 2 via a local network LN and a network N. The network N is a wired or wireless communication network that may include a public communication network, a dedicated line, or a carrier network. The client 3 can be connected to the server 2 via the network N.

[0037] The edge device 1 performs image recognition processing on image data acquired from the camera 4. For example, the edge device 1 performs processing to detect objects within the field of view of the camera 4 by distinguishing them from the background, processing to detect the area of ​​the detected object, processing to determine the attributes of the detected object, and so on, and outputs text indicating the detection results and determination results. The edge device 1 may also identify the movement of objects within the field of view of the camera 4 and output text indicating the identification results. Based on the position of the camera 4 and the position of the object in the image, the edge device 1 may calculate data (time-series coordinate data within the space) indicating the path of the object detected in the space to be analyzed where the camera 4 is installed.

[0038] The edge device 1 stores text indicating the attributes of objects detected from the image data in association with identification data that enables identification of the space to be analyzed, and also transmits the text to the server 2 as necessary. The edge device 1 performs recognition processing on the image data obtained from the camera 4 to the extent possible within the device itself, so it does not necessarily need to transmit the image data to the server 2, thereby reducing the communication load and also reducing communication costs.

[0039] The server 2 records the data sent from the edge device 1 in a database 210 and can perform analytical processing such as tallying the attributes of detected objects and statistical processing for each space being analyzed. The results of the analytical processing by the server 2 can be viewed from the client 3 for each space being analyzed. The manager of the space being analyzed can use the client 3 to design the space according to the flow of objects detected within the space. More specifically, by installing the image analysis system 100 in a store, the store manager can check from the client 3 the tallying results of the age group, gender, etc. of customers visiting the store, as well as suggestions for product lineups and placement based on customers' movements within the store.

[0040] Because the edge device 1 has limited computing resources, it is necessary to implement a learning model that uses a so-called lightweight neural network with as little computational effort as possible. To perform accurate and immediate object detection and recognition on the edge device 1 using this lightweight learning model, tuning (fine tuning) is required to adapt it to the individual characteristics of the space to be analyzed.

[0041] Therefore, the image analysis system 100 of this embodiment uses the inspection system 500 to evaluate the accuracy of inference (object detection, object recognition, etc.) through calculations using a learning model applied to the edge device 1 during tuning or at the start of operation. The inspection system 500 is used temporarily in the image analysis system 100, for example, until operation using the learning model becomes stable. The inspection system 500 includes an inspection device 5 installed in the space to be analyzed (edge ​​side) of the image analysis system 100, and a server 6 that remotely (also referred to as the server side or cloud side) acquires the evaluation results from the inspection device 5. The server 6 may be part of the functions of the server 2.

[0042] In the inspection system 500, the inspection device 5 executes a process using a learning model similar to the process executed by the edge device 1, and evaluates the inference accuracy using the learning model of the target edge device 1. The inspection system 500 receives the evaluation results at the server 6 and makes them available for confirmation by the client 3. The inspection system 500 also has a function to retrain the learning model used in each edge device 1 or to select a different learning model depending on the evaluation results.

[0043] 2 is a block diagram showing the configuration of the edge device 1. The edge device 1 is a box-shaped device that can be installed together with cameras 4 in a space to be analyzed, such as a store. The edge device 1 includes a processing unit 10, a storage unit 11, a first communication unit 12, and a second communication unit 13.

[0044] The processing unit 10 includes one or more processors such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), etc. The processing unit 10 includes a memory that is a temporary storage medium such as a static random access memory (SRAM) or a dynamic random access memory (DRAM). The processing unit 10 includes a timer and can acquire time information at each point in time from data from the timer. The processing unit 10 may be configured as a single piece of hardware (SoC: System On a Chip) that integrates a processor, a memory, a storage unit 11, a first communication unit 12, and a second communication unit 13. The specifications of the processing unit 10 may be the same or different among the multiple edge devices 1.

[0045] The storage unit 11 is a relatively large-capacity non-transitory storage medium such as a hard disk, a flash memory, etc. A part of the storage unit 11 may be removable.

[0046] The storage unit 11 stores a program (program product) required for the processing unit 10 to execute processing, and setting data to be referenced during the processing. The setting data includes an identifier for the edge device, identification data for the connected camera 4, etc. The program product includes an OS (Operating System) program, a model calculation program that runs on the OS, and its (at least one) learning model M1. The model calculation program and learning model M1 stored in the storage unit 11 may be downloaded by the processing unit 10 from the server 2, the server 6, or another download server via the second communication unit 13 and stored in the storage unit 11.

[0047] The storage unit 11 stores image data acquired from the connected camera 4 in association with time information. The storage unit 11 preferably stores only image data in which the detection target has been detected by the processing unit 10. The storage unit 11 may continue to store data by overwriting it according to the FIFO (First In First Out) method within a range that corresponds to the size of its capacity. The storage unit 11 may store data of the detection results (such as text indicating attributes) in association with time information.

[0048] The first communication unit 12 is a communication device that realizes communication via the local network LN. The first communication unit 12 may be a LAN network card or a CAN communication device. The first communication unit 12 may be a communication device compatible with wireless networks such as Wi-Fi or Bluetooth (registered trademark). The first communication unit 12 may include multiple communication devices compatible with various types of cameras 4. The first communication unit 12 may include an interface such as a USB (Universal Serial Bus) connected to the camera 4. The first communication unit 12 can be replaced by an interface connected to the camera 4 via a coaxial cable or another serial bus. The processing unit 10 acquires image data from the camera 4 via the local network LN through the first communication unit 12. The processing unit 10 can transmit and receive data to and from other edge devices 1 through the first communication unit 12. The first communication unit 12 may be the same device as the second communication unit 13.

[0049] The second communication unit 13 is a communication device that realizes communication via an external network N. The second communication unit 13 may be a network card for a wired LAN, a communication device that realizes carrier communication via a carrier network, or a communication device that supports a wireless network such as WiFi or Bluetooth (registered trademark). The second communication unit 13 may support encrypted communication with the server 2 using SSL or the like. The second communication unit 13 may be an interface that realizes connection with the server 2 via a dedicated line.

[0050] The processing unit 10 acquires image data from the camera 4 via the first communication unit 12, and uses the model calculation program and the learning model M1 to perform calculations (inference processing) on ​​objects (e.g., people) captured in the image data. The processing unit 10 performs object detection and object recognition (e.g., determining the attributes of objects) based on the model calculation program and the learning model M1, either by itself or in collaboration with other edge devices 1. When sharing the work with other edge devices 1, different learning models M1 may be allocated to multiple edge devices 1, and each edge device 1 may perform high-speed processing by pipeline processing, passing extracted features and calculation results (object detection or object recognition results) to the next device.

[0051] The processing unit 10 of the edge device 1 associates text indicating the detection result or recognition result with the identification data of the edge device 1 or the identification data of the camera 4 and transmits the text from the second communication unit 13 to the server 2. The processing unit 10 may also transmit time information of the timing at which the image data was acquired together with the detection result or recognition result to the server 2. Here, the processing unit 10 does not need to transmit the image data, which reduces the communication load between the edge device 1 and the server 2 and suppresses communication costs.

[0052] 3 is a block diagram showing the configuration of the server 2. The server 2 may be configured as a single server computer, or may be configured to distribute processing among multiple server computers. The server 2 includes a processing unit 20, a storage unit 21, and a communication unit 22.

[0053] The processing unit 20 includes one or more processors such as a CPU, an MPU, a GPU, etc. The processing unit 20 includes a memory that is a temporary storage medium such as an SRAM or a DRAM.

[0054] The storage unit 21 is a relatively large-capacity non-temporary storage medium such as a hard disk, a flash memory, etc. The storage unit 21 stores a program (program product) required for the processing unit 20 to execute processing.

[0055] The program products stored in the storage unit 21 include a server program P2. The server program P2 includes a module that performs the function of a web server, and can output the results of processing in the server 2 to the client 3 via a web page.

[0056] The program product stored in the storage unit 21 includes a learning model group M2. The learning model group M2 is stored for distribution to the edge device 1, and does not need to be stored in the storage unit 21 if it is stored in another download server.

[0057] A database 210 is constructed in the storage unit 21 to store space definition data and administrator data for the analysis target space in association with the identification data of the analysis target space. The storage unit 21 may acquire and store calculation results (number of detected objects, object attributes, etc.) from the edge device 1 in association with the account of the client 3 for each analysis target space in the database 210, so that the analysis results that can be referenced by the client 3 can be narrowed down.

[0058] The communication unit 22 is a communication device that realizes a communication connection with the client 3 and the edge device 1 via the network N.

[0059] The processing unit 20 of the server 2 configured as described above stores the calculation results (object detection results and recognition results) transmitted from each edge device 1 via the communication unit 22 in the database 210 of the storage unit 21, in association with the identification data of the edge device 1 or the identification data of the camera 4 transmitted together. The processing unit 20 sequentially receives and stores the calculation results using the learning model M1 on the image data of the video data acquired in time series from the camera 4, thereby analyzing the trends and movements of objects in the analysis target space and performing statistical processing. This allows the processing unit 20 to tally, for example, the number of detected objects and the attributes of the objects for each analysis target space. The processing unit 20 can also identify the movement of the same object over time by analyzing the results of the time-series inference processing. Specifically, for each store in the analysis target space, the processing unit 20 can tally the number of customers by time period, tally the attributes of the customers, such as their age group, analyze the points of interest of the customers, and analyze the paths the customers take within the store.

[0060] 4 is a block diagram showing the configuration of the client 3. The client 3 is a personal computer, a smartphone, or a tablet terminal. The client 3 may be used by an administrator of the space to be analyzed, or by an operator of the server 2 or server 6.

[0061] The client 3 includes a processing unit 30, a storage unit 31, a communication unit 32, a display unit 33, and an operation unit 34. The processing unit 30 includes one or more processors such as a CPU, an MPU, a GPU, etc. The processing unit 30 includes a memory that is a temporary storage medium such as an SRAM or a DRAM.

[0062] The storage unit 31 is a memory of a non-temporary storage medium such as a hard disk or a flash memory. The storage unit 31 stores a client program corresponding to an information provision protocol from the server 2. The client program is, for example, a web browser program. The client program is not limited to a web browser program, but may be a special program that causes the processing unit 30 to execute a process of displaying data transmitted from the server 2 or 6 on a screen.

[0063] The communication unit 32 is a communication device that realizes a communication connection with the server 2 or the server 6 via the network N. The communication unit 32 may be a communication device that realizes a communication connection with the server 2 or the server 6 via a dedicated line.

[0064] The display unit 33 uses a display such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 33 displays a web page including text and images through processing based on the client program of the processing unit 30. The display unit 33 may use a display with a built-in touch panel.

[0065] The operation unit 34 is a user interface such as a keyboard or a pointing device that accepts operations from a user, an operator, or a serviceman. The operation unit 34 may be a touch panel built into the display of the display unit 33, or may be physical buttons. The operation unit 34 may be a voice input unit that accepts operations by voice using a voice recognition function. The operation unit 34 can notify the processing unit 30 of operation information by a user, an operator, or a serviceman.

[0066] 5 is a block diagram showing the configuration of the inspection device 5. The inspection device 5 is a device with more abundant computational resources than the edge device 1. The inspection device 5 may be configured, for example, as a box-type device similar to the edge device 1, or may be configured as a laptop or desktop personal computer. The inspection device 5 includes a processing unit 50, a storage unit 51, a first communication unit 52, and a second communication unit 53.

[0067] The processing unit 50 includes one or more processors such as a CPU, an MPU, a GPU, etc. The processing unit 50 also includes a memory that is a temporary storage medium such as an SRAM or a DRAM. The processing speed of the processor of the processing unit 50 and the storage capacity of the memory are higher than the processing speed of the processor of the processing unit 10 of the edge device 1 and the storage capacity of the memory.

[0068] The storage unit 51 is a relatively large-capacity non-temporary storage medium such as a hard disk or flash memory. The storage unit 51 stores a program (program product) required for the processing unit 50 to execute processing, setting data to be referenced during the processing, and data generated by the processing of the processing unit 50. The program product includes an OS program, a model calculation program P50 that runs on the OS, an inspection program P51, and a learning model M5. The storage unit 51 stores image data selected by the processing unit 50 and learning data.

[0069] The model calculation program P50, the inspection program P51, and the learning model M5 stored in the memory unit 51 may be read by the processing unit 50 from the model calculation program P90, the inspection program P91, and the learning model group M9 stored in the computer-readable non-transitory storage medium 9 and stored in the memory unit 51, or may be stored in advance at the time of shipment. The model calculation program P50, the inspection program P51, and the learning model M5 stored in the memory unit 51 may be downloaded by the processing unit 50 from the server 2, the server 6, or another download server via the second communication unit 53 and stored in the memory unit 51. The learning model M5 may be selected by the server 6 and transmitted by the server 6, as described below.

[0070] The first communication unit 52 is a communication device that realizes communication via the local network LN. The first communication unit 52 may be a LAN network card or a CAN communication device. The first communication unit 52 may be a wireless communication device. The processing unit 50 can acquire image data from the camera 4, as with the edge device 1, via the local network LN using the first communication unit 52. The processing unit 50 can transmit and receive data to and from the edge device 1 using the first communication unit 52. The first communication unit 52 may be the same device as the second communication unit 53.

[0071] The second communication unit 53 is a communication device that realizes communication via an external network N. The second communication unit 53 may be a network card for a wired LAN, a communication device that realizes carrier communication via a carrier network, or a communication device that supports a wireless network such as WiFi or Bluetooth (registered trademark). The second communication unit 53 may support encrypted communication with the server 6 using SSL or the like. The second communication unit 53 may be an interface that realizes connection with the server 6 via a dedicated line.

[0072] The processing unit 50 of the inspection device 5 acquires image data from the camera 4 via the first communication unit 52, and based on the model calculation program P50 and the learning model M5, is able to extract features from the image data, just as with the target edge device 1, and perform inference processing using the features. As will be described later, the learning model M5 is a model that uses a trained NN, which has a larger number of parameters and a heavier computational load than the learning model M1 used in the edge device 1. The processing unit 50 can acquire the inference results of the edge device 1, and evaluate the inference results of the edge device 1 to be inspected by comparing them with the inference results of its own device. The processing performed by the processing unit 50 will be described in detail later.

[0073] 6 is a block diagram showing the configuration of server 6. The hardware configuration of server 6 is similar to that of server 2, so corresponding reference numerals are used and detailed description will be omitted. Server 6 constituting inspection system 500 stores inspection server program P6 in storage unit 61, and executes the processing described below for learning models M1 and M5 used in edge device 1 and inspection apparatus 5 using this inspection server program P6.

[0074] The server 6 can transmit and receive data to and from the inspection device 5 via the communication unit 62. The processing unit 60 of the server 6 cooperates with the inspection device 5 by using the inspection server program P6, evaluates inference using the learning model M1 of the edge device 1 in the image analysis system 100 to be inspected, as will be described later, and executes processing according to the evaluation results.

[0075] The server 6 can also transmit and receive data to and from the server 2 via the communication unit 62. The processing unit 60 of the server 6 can transmit data proposing the learning model M1 to be deployed from the server 2 to the edge device 1.

[0076] The server 6 can send and receive data to and from the client 3 via the communication unit 62. The server 6 functions as a web server, and can output processing results to the client 3 via the communication unit 62.

[0077] The inspection process performed by inspection system 500 in image analysis system 100 including the devices configured as described above will be described in detail below. FIG.

[0078] The system to be inspected by the inspection system 500 uses one or more edge devices 1 for one or more cameras 4, and includes a model calculation unit 101 that, when image data or detection results for the image data are input, performs calculations (inference) such as detecting objects in the image and recognizing objects. Here, the system to be inspected performs image recognition based on one or more learning models M1 distributed to one or more edge devices 1. For example, if the system employs pipeline processing in which multiple edge devices 1 sequentially process a single image data using different learning models M1, the system deploys a learning model M1, which is a face detection model for detecting people who have visited a store from the image data, to a first edge device 1, and, if a face is detected, deploys a learning model M1, which is a model for extracting a face region for each face in the image data, to a second edge device 1. The system deploys a learning model M1, which is a model for determining the orientation of a face from an image extracted using a face detection model or a face region extraction model, and a learning model M1, which is a model for determining gender from a face, to a third edge device 1. Therefore, each learning model M1 is a model that outputs a detection result when image data is input, or a model that outputs a recognition result when small image data extracted from the detection result is input, and input data according to the processing is input to the learning model M1.

[0079] 7 shows an inspection system 500 for one learning model M1 and one model calculation unit 101 based on the corresponding model calculation program, for ease of explanation. The inspection system 500 sequentially executes the inspection process described below for each different learning model M1 via the local network LN, and evaluates the model calculation (inference process).

[0080] The inspection system 500 performs similar calculations for each distributed learning model M1 used in the system to be inspected using an inspection device 5 that has more abundant computing resources than the edge device 1, using a learning model M5 that is more accurate than the learning model M1, and compares the calculation results to evaluate them against the learning model M1 deployed on the edge device 1.

[0081] Here, the inspection device 5 of the inspection system 500 uses a learning model M5 corresponding to the learning model M1 used in the model calculation unit 101 of the inspection target. First, the learning model M1 used in the model calculation unit 101 is initially selected and deployed based on conditions set in the server 2 or server 6. The conditions include the detection target of the learning model M1, differences in situations, etc. The learning model M1 can also be selected via the network N from those published on the Internet and permitted for commercial use. For example, a trained NN model for detecting "people" "indoors," a trained NN model for detecting "people" "outdoors," a trained NN model for discriminating "gender," etc. can be acquired, and these models are stored in the database 210 (or database 610) of the storage unit 21 (or storage unit 61) of the server 2 (or server 6) in association with tags and identification data that identify each model.

[0082] In contrast, the learning model M5 used in the inspection device 5 is a model that has been trained to output the same output as the learning model M1 using different (multiple) parameters when the same input data is input. For example, if the learning model M1 to be inspected is a face detection model as described above, the learning model M5 used in the inspection device 5 is also a model that detects a face appearing in an image when image data is input. The learning model M5 may be selected from a group of learning models M6 stored in the server 6, or a high-precision (heavy) model may be preferentially selected from general-purpose trained NN models made public via the network N as shown in FIG. 7, and a high-precision trained NN model with a similar purpose (detection target and situation) may be selected for use in the inspection device 5.

[0083] The learning model M5 may be the same as the learning model M1 used by the model calculation unit 101 of the processing unit 10 of the edge device 1. In other words, the learning model M1 may be a model obtained by reducing the parameters of a learning model selected for the edge device 1 to be inspected from a publicly available general-purpose trained NN model through compression such as distillation and pruning for the edge device 1, and the original learning model may be the learning model used by the model calculation unit 501. The learning model used in the inspection device 5 is a model that achieves highly accurate detection and recognition on the premise of using abundant computing resources, and a model with relatively few parameters obtained by distilling and compressing this model for the edge device 1 may be the learning model M1 used by the model calculation unit 101.

[0084] In addition, the server 6 of the inspection system 500 stores identification data of the learning model used in the corresponding model calculation unit 501 for each condition for each learning model M1 deployed to each edge device 1, and when the inspection system 500 is connected to the system to be inspected, the server 6 may deploy a learning model corresponding to the learning model M1 to the inspection device 5.

[0085] In the inspection system 500, on the edge side, the processing unit 50 of the inspection apparatus 5 acquires the same input data as that of the model calculation unit 101 of the inspection target from the camera 4 or the edge device 1. Based on the model calculation program P50, the processing unit 50 provides the input data to the learning model M5 selected as described above for the learning model M1, and functions as a model calculation unit 501 that executes processes such as extraction and detection or recognition of feature quantities similar to those of the inspection target.

[0086] The processing unit 50 of the inspection device 5 functions as a comparison unit 502 that compares the calculation results of the model calculation unit 501 for input data with the results of processing by the model calculation unit 101 for the same input data acquired from the edge device 1, based on the inspection program P51. The comparison unit 502 calculates the difference between the results obtained by the model calculation unit 101 for the same input data and the results obtained by the model calculation unit 501. For example, the comparison unit 502 compares the number of objects and the number of people obtained by calculation in the model calculation unit 101 using the learning model M1 of the edge device 1 for input image data with the number of objects and the number of people obtained from the learning model M5 through calculation in the model calculation unit 501 of the inspection device 5. For example, the comparison unit 502 compares data such as the age or gender of a person obtained by the model calculation unit 101 using the learning model M1 of the edge device 1 for input detection results from other models and a rectangular image of a face region with data such as the age or gender of a person obtained by calculation in the model calculation unit 501 based on the learning model M5. The processing unit 50 can output the comparison result obtained by the function of the comparison unit 502 to the server 6 and display the result on the client 3 via the server 6. The inspection device 5 may have an output unit such as a display, and the processing unit 50 may be able to check the comparison result obtained by the comparison unit 502 on the edge side.

[0087] The processing unit 50 of the inspection device 5 functions as a determination unit 503 that determines, based on the comparison result by the comparison unit 502, whether the difference between the result of processing by the model calculation unit 501 for the same input data and the result of processing by the model calculation unit 101 for the same input data acquired from the edge device 1 exceeds a predetermined range. If the determination unit 503 determines that the difference exceeds the predetermined range, the processing unit 50 stores and saves the original target image data of the input data and the result of the inference processing by the model calculation unit 501 on its own device (detection result, recognition result, etc.) in the storage unit 51 as re-learning data. The result of the inference processing by the model calculation unit 501 on its own device becomes tag data in the re-learning data (tag data created by automatic annotation).

[0088] In the inspection system 500, the processing unit 60 of the server 6 on the server side functions as a model selection unit 601. Based on the comparison results by the inspection device 5, the model selection unit 601 selects a learning model M1 for the edge device 1 and a corresponding learning model M5 with many parameters and high accuracy, and deploys the learning model M1 to the edge device 1 and the learning model M5 to the inspection device 5. If the function of the determination unit 503 of the inspection device 5 determines that the difference is equal to or greater than a predetermined amount, the model selection unit 601 searches for other learning models according to the conditions from the database 610 or publicly available trained NN models. As a result of the search, the server 6 generates a learning model M1 for the edge device 1 by compressing the most appropriate trained NN model, for example by distilling it, thereby reducing the number of parameters. The server 6 deploys the learning model M1 to the edge device 1 via the server 2, and continues the inspection again.

[0089] The inspection procedure performed by these functions of the inspection system 500 and the processing procedure performed by each device will be described in more detail with reference to FIG. 7 or a flowchart.

[0090] First, a space manager or a serviceman from a system management company places cameras 4 and edge devices 1 in a store, which is the space to be analyzed, as shown in FIG. 1 , and then constructs an image analysis system 100 that is ready for communication with the server 2. A trained learning model M1 is provisionally deployed to each edge device 1 according to the configuration of the multiple edge devices 1, either through an operator configuring the server 2 via a client 3 or through automatic determination by the server 2. For example, from a group of learning models M2 stored in the server 2, a learning model M1 that matches the conditions of the detection target being "people" and the situation being "inside a store" is deployed. For example, the server 2 or server 6 selects a learning model M1 trained to detect customer faces based on images of customers shopping in a store taken from diagonally above from a trained NN model tagged with "store," "customer," etc. in the database 610 or from published models, and then deploys the model to the edge device 1 after reducing the weight by distillation or other methods. If the camera angle information is tagged, that trained NN model may be selected.

[0091] A service technician temporarily connects the inspection device 5 of the inspection system 500 to the system to be inspected (image analysis system 100) in which the learning model M1 is deployed. Then, the service technician operates the server 6 from the client 3 for the model to be inspected among the learning models M1 of each edge device 1, selects the learning model M5 from the learning model group M6 stored in the memory unit 61, and deploys it to the inspection device 5. After the inspection system 500 is constructed for the target system in this way, the inspection device 5 executes the following process.

[0092] 8 is a flowchart showing an example of a processing procedure by the inspection system 500. The processing unit 50 of the inspection device 5 acquires input data to be input to the model calculation unit 101 using the learning model M1 to be inspected (step S501). The input data acquired in step S501 may be image data or feature data extracted from image data by another learning model M1, or may be data of a detection result or a recognition result output by another learning model M1.

[0093] In step S501, the processing unit 50 may execute processing in response to an inspection request from the server 6, or may perform the following processing for the continued recognition processing of the detected object when it detects that an object has been detected in the edge device 1 to be inspected. If no object is detected and evaluation of the learning model M1 to be inspected is unnecessary, the processing of the processing unit 50 of the inspection device 5 may be omitted. This eliminates the need to constantly send input data to the inspection device 5, thereby reducing the communication load.

[0094] The processing unit 50 provides the acquired input data to the deployed learning model M5 using the function of the model calculation unit 501 (step S502). The processing unit 50 acquires the results output from the learning model M5 (step S503).

[0095] During this time, the processing unit 50 acquires output data from the learning model M1 obtained by calculation in the edge device 1 for the same input data (step S504).

[0096] The processing unit 50 compares the processing results using the learning model M5 in the inspection device 5 obtained in step S503 with the processing results using the learning model M1 in the edge device 1 obtained in step S504 using the function of the comparison unit 502 (step S505).

[0097] As a result of the comparison in step S505, the processing unit 50 determines whether the difference is equal to or greater than a predetermined amount (step S506). If it is determined that the difference is equal to or greater than the predetermined amount (S506: YES), the processing unit 50 stores the input data acquired in step S501 and the result acquired in step S503 as learning data in the storage unit 51 (step S507). In step S507, the processing result using the high-precision, heavy learning model M5 by the processing unit 50 of the inspection apparatus 5, which has more abundant computational resources than the processing unit 10 of the edge device 1, is stored as a correct answer for use in tuning the learning model M1 of the edge device 1. In step S507, the processing unit 50 may acquire image data of the frame that is the source of the input data and store it in association with the image data.

[0098] The processing unit 50 transmits the result of the comparison by the comparison unit 502 or the result of the determination by the determination unit 503 to the server 6 (step S508).

[0099] If the processing unit 50 determines in step S506 that the difference is less than the predetermined amount (S506: NO), the processing unit 50 proceeds to step S508.

[0100] The server 6 receives the comparison result or the judgment result from the inspection device 5 (step S601), and stores it in the database 610 of the storage unit 61 in association with the identification data of the learning model M5 used in the inspection device 5 (step S602). By storing the comparison result or the judgment result in the server 6, it can be output from the client 3.

[0101] The processing unit 60 of the server 6 determines whether reselection of a model is necessary based on the comparison result or the judgment result (step S603). In step S603, the processing unit 60 determines that reselection is necessary, for example, if the difference in the comparison result is equal to or greater than a predetermined amount and, based on previous storage in the storage unit 61, the difference has been determined to be equal to or greater than the predetermined amount multiple times. The processing unit 60 may also determine that reselection is necessary if the difference is determined to be equal to or greater than a second predetermined amount that is greater than the predetermined amount.

[0102] If it is determined that model reselection is necessary (S603: YES), the processing unit 60 reselects another trained NN model as an alternative model to the trained model M1 using the function of the model selection unit 601 according to the target of the model calculation unit 101 by the edge device 1 (step S604). The processing unit 60 notifies the server 2 of the selected other trained NN model (step S605) and ends the processing. In step S605, the processing unit 60 may simply notify the other trained NN model as a candidate, without actually deploying it.

[0103] If it is determined in step S603 that reselection of a model is not necessary (S603: NO), the processing unit 60 ends the process.

[0104] The inspection device 5 and server 6 of the inspection system 500 repeatedly perform the processing procedure shown in Fig. 8 for a certain period of time. The comparison results by the inspection device 5 of the inspection system 500 are stored in the server 6 (S602) and can be confirmed numerically by a serviceman via the client 3. For example, the difference in the number of faces (people) detected by the face detection model from the same frame image may be confirmed as the comparison result by the client 3. Furthermore, the difference in the age determined for each detected person may be confirmed as the comparison result by the client 3, or the difference in the aggregated values ​​of the gender of each detected person as a result of repeating the processing procedure shown in Fig. 8 for a certain period of time may be confirmed as the comparison result by the client 3.

[0105] After checking the comparison results, the service technician may tune the learning model M1 in the edge device 1 using the learning data stored in the inspection device 5 according to the quantified difference data, or may reselect and redeploy the model if it is determined in step S603 that reselection is necessary.

[0106] After the learning model M1 has been tuned, or after another alternative model has been reselected and redeployed, the service engineer performs the processing procedure shown in Fig. 8 again. When the execution of the processing procedure shown in Fig. 8 confirms that the difference between the processing result of the model calculation unit 501 using the learning model M5 in the inspection device 5 with relatively abundant computational resources and the processing result of the model calculation unit 101 using the learning model M1 in the edge device 1 is equal to or less than a predetermined amount, the service engineer can complete the inspection of the target learning model M1.

[0107] The service technician uses the inspection device 5 and the server 6 to switch the target learning model M1, repeatedly perform the processing procedure shown in Figure 8 for a certain period of time, perform tuning and reselection of other models, and complete the inspection of the learning model M1 used in the target image analysis system 100. The service technician can then remove the inspection system 500 from the system to be inspected and start operating the image analysis system 100.

[0108] In this way, the validity of the detection accuracy and recognition accuracy of the learning model M1 of the edge device 1 can be quantified and confirmed by comparing it with the processing in the inspection device 5, which has abundant computational resources and uses the learning model M5 with a large amount of computation (parameters). As a result of the comparison, it is also possible to further tune the learning model M1 in the edge device 1 or automatically reselect the learning model M1 via the server 2 and the server 6.

[0109] The method using the inspection system 500 of the present disclosure makes it possible to check the degree of conformance of the learning model M1 of the edge device 1 to the space to be analyzed in a short time with little effort. As described above, if the determining unit 503 determines that the difference is equal to or greater than a predetermined amount and the degree of conformance is considered low, the input data that caused the problem can be collected in the inspection device 5, allowing for focused tuning of the learning model M1 to be inspected and improving efficiency.

[0110] 8, if the processing unit 60 of the server 6 determines in step S603 that model reselection is unnecessary, it may determine that it is appropriate and store data identifying the learning model M5 used in the inspection device 5 in association with data identifying the analysis target space to which this learning model was applied as an inspection history in the database 610. The database 610 may also store the identification data of the learning model used in the inspection device 5 in association with data indicating the time of the inspection. This allows the processing unit 60 of the server 6 to refer to the stored inspection history when selecting the learning model M5 to be used in the inspection device 5 and select a proven learning model depending on the season, geographical conditions, and type of store (apparel, food, supermarket, pharmacy, etc.).

[0111] Furthermore, application of the inspection system 500 to the image analysis system 100 can be continued not only for a certain period after the system starts operating, but also during operation. Even after starting to use the tuned learning model M1 through processing by the inspection system 500, it is possible to re-verify it when the seasons change or when the store layout is changed. Inspection by the inspection system 500 is performed according to seasonal changes, making it possible to tune or reselect the model taking into account the impact on accuracy due to factors such as clothing that may change with the seasons.

[0112] Furthermore, after being removed, inspection system 500 can be connected to a local network LN in a completely different space to be analyzed and used to inspect learning model M1 in image analysis system 100 in that new space to be analyzed. In other words, inspection system 500 can be deployed to a different system simply by replacing the inspection learning model.

[0113] In the above embodiment, an example is shown in which the learning model M1, which is the model to be inspected, and the learning model M5, which is a high-precision (heavy) model for inspection, are deployed on separate computers (the edge device 1 and the inspection device 5), but the present invention is not limited to this example, and the model to be inspected and the high-precision (heavy) model for inspection may be deployed on the same computer. For example, the high-precision (heavy) model for inspection (e.g., the above learning model M5) may be deployed on the edge device on which the model to be inspected (e.g., the above learning model M1) is deployed.

[0114] The embodiments disclosed above are illustrative in all respects and are not restrictive. The scope of the present invention is defined by the claims, and includes all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0115] 100 Image Analysis System 1. Edge Devices 10 Processing section 11 Storage section 2 Server 20 Processing section 21 Memory section 210 databases 3 Client 33 Display section

Claims

1. Deploying a second trained neural network model, which outputs the same type of data as the first trained neural network model when inputted with the same input data as the first trained neural network model, on a computer that is the same as or different from a computer on which a first trained neural network model is deployed, the second trained neural network model having been trained to output a detection result or an object recognition result of an object appearing in an image when image data or data obtained from image data is inputted; comparing data output from the first trained neural network model with data output from the second trained neural network model when the same input data is input to the first trained neural network model and the second trained neural network model; Output the comparison results How to inspect the model.

2. a second computer connected to a first computer using a first trained neural network model that has been trained to output a detection result or an object recognition result of an object appearing in an image when image data or data obtained from the image data is input; acquiring a second trained neural network model that outputs the same type of output data as the first trained neural network model when the same input data as the first trained neural network model is input; obtaining input data to be input to the first trained neural network model; comparing data output when the input data is input to the second trained neural network model with data output from the first trained neural network model; Output the comparison results How to inspect the model.

3. The second computer has computing resources with higher specifications than the first computer. The method of claim 2 .

4. The second computer determining whether or not a difference between data output when the acquired input data is input to the second trained neural network model and data output from the first trained neural network model is equal to or greater than a predetermined amount; If it is determined that the difference is equal to or greater than a predetermined amount, the input data and the data output from the second trained neural network model are stored in a storage unit as training data. A method for inspecting a model according to claim 2 or 3.

5. Retraining the first trained neural network model using the training data stored in the storage unit. The method of claim 4 .

6. selecting an alternative model of the first trained neural network model or the second trained neural network model from the published trained neural networks based on the comparison; Deploying the selected alternative model to the first computer or the second computer. The method of claim 2 .

7. the first trained neural network model or the second trained neural network is stored in a database of a server according to shooting conditions including a shooting subject or a shooting situation of image data corresponding to input data; The server Select the model that corresponds to the specified shooting conditions, Sending the selected model to the first computer or the second computer. The method of claim 6 .

8. The server storing an inspection history using the second trained neural network; A second trained neural network for the trained neural network of the new test subject is selected from the test history based on imaging conditions in an environment in which the trained neural network of the new test subject is used. The method of claim 7 .

9. An inspection device connected to a computer using a first trained neural network model that has been trained to output a detection result or an object recognition result of an object appearing in an image when image data or data obtained from the image data is input, and that inspects the first trained neural network model, The inspection device includes: acquiring a second trained neural network model that outputs the same type of output data as the first trained neural network model when the same input data as the first trained neural network model is input; obtaining input data to be input to the first trained neural network model; comparing data output when the input data is input to the second trained neural network model with data output from the first trained neural network model; Output the comparison results An inspection device equipped with a processing unit.

10. a second computer connected to the first computer using a first trained neural network model that has been trained to output a detection result or an object recognition result of an object appearing in an image when image data or data obtained from the image data is input; a server that transmits and receives data to and from the second computer through communication; Including, The second computer acquiring a second trained neural network model that outputs the same type of output data as the first trained neural network model when the same input data as the first trained neural network model is input; obtaining input data to be input to the first trained neural network model; comparing data output when the input data is input to the second trained neural network model with data output from the first trained neural network model; The comparison result is output to the server. Inspection system.

11. The server A plurality of trained models to be used as the first trained neural network model or the second trained neural network model are stored according to shooting conditions including a shooting subject or a shooting situation of image data corresponding to input data of the trained model; Select a trained model that corresponds to the specified shooting conditions, Sending the selected model to the first computer or the second computer. The inspection system of claim 10.

12. a second computer connected to a first computer using a first trained neural network model that has been trained to output a detection result or an object recognition result of an object appearing in an image when image data or data obtained from the image data is input; acquiring a second trained neural network model that outputs the same type of output data as the first trained neural network model when the same input data as the first trained neural network model is input; obtaining input data to be input to the first trained neural network model; comparing data output when the input data is input to the second trained neural network model with data output from the first trained neural network model; Output the comparison results A computer program that executes a process.

Images