Information processing device and information processing program

Abstract

This invention provides an information processing device and an information processing program that collect appropriate training data for an image processing model in an environment where a camera is installed. [Solution] In an information processing system in which a server device and multiple cameras are connected by a predetermined network, the camera 50, which acts as an information processing device, comprises a setting unit 541 and a collection unit 543. The setting unit 541 sets the conditions for collecting images that will be used as training data for an image processing model. The collection unit 543 collects images from among those captured by the camera that satisfy the collection conditions set by the setting unit 541 as training data.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and an information processing program.

Background Art

[0002] Conventionally, there is an image processing system that analyzes an image captured by a camera. For example, a technique for determining whether a detection target shown in an image is in a specific state by using an AI technique called VQA (Visual Question Answering) is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, there is room for improvement in optimizing the image processing model according to the actual environment where the camera is installed. Specifically, there is room for improvement in collecting appropriate learning data in the environment where the camera is installed.

[0005] The present invention has been made in view of the above, and an object thereof is to provide an information processing apparatus and an information processing program capable of collecting appropriate learning data in an environment where a camera is installed.

Means for Solving the Problems

[0006] To solve the above-mentioned problems and achieve the objective, the information processing device according to the present invention comprises a setting unit and a collection unit. The setting unit sets the conditions for collecting images to be used as training data. The collection unit collects images from among those captured by the camera that satisfy the collection conditions set by the setting unit as training data. [Effects of the Invention]

[0007] According to the present invention, appropriate training data can be collected in the environment in which the camera is installed. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of the configuration of an information processing system according to an embodiment. [Figure 2] Figure 2 is a block diagram of the camera according to the embodiment. [Figure 3] Figure 3 shows an example of information stored in the image data storage unit according to this embodiment. [Figure 4] Figure 4 is an explanatory diagram of the collection conditions according to the embodiment. [Figure 5] Figure 5 is an explanatory diagram of the process for determining the collection conditions according to the embodiment. [Figure 6] Figure 6 is a flowchart showing the processing procedure performed by the camera according to the embodiment. [Modes for carrying out the invention]

[0009] The camera 50 (an example of an information processing device) according to the embodiment described below comprises a setting unit 541 that sets the conditions for collecting images to be used as training data, and a collection unit 543 that collects images from among those captured by the camera 50 that satisfy the collection conditions set by the setting unit 541 as training data.

[0010] In the embodiment described below, the setting unit 541 sets the coordinate conditions of the target to be detected as collection conditions, and the collection unit 543 collects images in which the coordinates of the target to be detected satisfy the coordinate conditions as training data.

[0011] In the embodiment described below, the setting unit 541 sets coordinate conditions for a specific location to be detected as collection conditions, and the collection unit 543 collects images in which the coordinates of the specific location to be detected satisfy the coordinate conditions as training data.

[0012] In the embodiment described below, the setting unit 541 sets orientation conditions related to the orientation of the object to be detected as collection conditions, and the collection unit 543 collects images in which the orientation of the object to be detected satisfies the orientation conditions as training data.

[0013] The camera 50 (an example of an information processing device) according to the embodiment described below includes a learning unit 544 that performs training on an image processing model that analyzes the state of the object to be detected from an image, based on the training data collected by the collection unit 543.

[0014] The information processing program according to the embodiment described below causes the computer to execute a setting procedure for setting conditions for collecting images to be used as training data, and a collection procedure for collecting images from among the images captured by the camera 50 that satisfy the collection conditions set in the setting procedure as training data.

[0015] (Embodiment) Embodiments of the present invention will be described below with reference to the drawings. Note that the embodiments described below are not intended to limit the technology disclosed by the present invention. Furthermore, the same reference numerals are used for the same parts in each embodiment, and redundant descriptions are omitted.

[0016] First, the information processing system will be explained using Figure 1. Figure 1 is a diagram showing an example configuration of the information processing system according to the embodiment. The information processing system S according to the embodiment can be introduced into various facilities, such as factories.

[0017] For example, the information processing system S provides various services based on the analysis results of the images captured by each camera 50. For example, when the information processing system S is introduced into a factory, it determines whether a worker is in a dangerous state (e.g., an equipment state that violates the safety manual), and provides services such as notifying the determination result to the factory manager or the like. Further, the information processing system S can also provide various services such as crowd flow analysis based on the images captured by each camera 50.

[0018] As shown in FIG. 1, the information processing system S includes a server device 1 and a plurality of cameras 50. Also, as shown in FIG. 1, the server device 1 and the plurality of cameras 50 are connected by a predetermined network N.

[0019] The server device 1 controls the entire information processing system S. Also, the server device 1 provides various services based on the images captured by each camera 50.

[0020] The camera 50 is installed in various facilities and captures images. Also, the camera 50 has an image processing model and performs image processing using the image processing model. For example, the image processing model is a model (AI) that is machine-learned to detect detection targets such as people, and for example, it is VQA (Visual Question Answering).

[0021] Note that the camera 50 corresponds to an example of the information processing device according to the embodiment. Also, the camera 50 may be configured integrally with, for example, a lighting device installed on the ceiling. For example, in such a case, the camera 50 can capture an image overlooking from the ceiling.

[0022] By the way, the image processing model is required to be optimized for each actual environment where the camera 50 is installed. As a method for optimizing the image processing model, there is a method of re-learning the image processing model using the images captured in the actual environment as learning data.

[0023] When retraining an image processing model, it is necessary to prepare appropriate training data in advance. Against this backdrop, this disclosure explains the process for appropriately generating training data using the figures from Figure 2 onward.

[0024] Next, an example of the configuration of the camera 50 according to the embodiment will be described using Figure 2. Figure 2 is a block diagram of the camera 50 according to the embodiment. As shown in Figure 2, the camera 50 has a communication unit 51, an imaging unit 52, a storage unit 53, and a control unit 54.

[0025] The communication unit 51 is implemented, for example, by a predetermined communication circuit such as a NIC (Network Interface Card), and performs data communication with the server device 1, etc., via a communication network such as Ethernet (registered trademark) or LAN.

[0026] The imaging unit 52 is implemented using a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor. The imaging unit 52 captures an image area, generates an image, and outputs it to the control unit 54.

[0027] The memory unit 53 is implemented using, for example, semiconductor memory elements such as RAM (Random Access Memory) or flash memory, or storage devices such as hard disks or optical discs. In the example shown in Figure 2, the memory unit 53 includes an image data storage unit 531, an acquisition condition storage unit 532, a training data storage unit 533, and an image processing model storage unit 534.

[0028] The image data storage unit 531 stores image data. The image data is data of images captured by the imaging unit 52. Figure 3 is a diagram showing an example of the information stored in the image data storage unit 531 according to this embodiment.

[0029] As shown in Figure 3, the image data storage unit 531 stores information such as "imaging date and time," "image data," and "analysis results" in a manner that associates them with each other. The "imaging date and time" item stores the imaging date and time of the corresponding image data.

[0030] The "Image Data" field stores the data of the corresponding image. The "Analysis Results" field stores the analysis results of the image processing applied to the corresponding image.

[0031] Returning to the explanation of Figure 2, let's describe the collection condition storage unit 532. The collection condition storage unit 532 stores the collection conditions. The collection conditions are the collection conditions set for the images that will be used as training data.

[0032] For example, the collection conditions are the coordinate conditions of the object to be detected. Here, an example of collection conditions according to the embodiment will be explained using Figure 4. Figure 4 is an explanatory diagram of the collection conditions according to the embodiment.

[0033] Figure 4 shows an elevation view of the space where the camera 50 is installed. As shown in Figure 4, the camera 50 is installed on the ceiling. In this case, when analyzing the facial features of person T as the detection target using image processing, it is preferable to collect images showing person T's face as training data.

[0034] Therefore, as a condition for collecting training data, for example, coordinate information of the region in which person T's face is captured is set. Specifically, as shown in Figure 4, a first angle α and a second angle β are set as collection conditions to define the region in which person T's face is captured. Then, when the detection angle of person T's head is γ, images that satisfy "α < γ < β" are collected as training data. For example, the installation height of camera 50 from the floor is known, and the height of person T is assumed to be the average height of a Japanese person.

[0035] The first angle α and the second angle β represent the upper and lower limits of the region in which the face of person T is captured, with camera 50 as the reference point. That is, the region satisfying "α < γ < β" is the target region A1 in which the face of person T is captured, and the region not satisfying "α < γ < β" is the non-target region A2 in which the face of person T is not captured.

[0036] Thus, the data collection conditions for training include information about the coordinates of the target region A1. This allows for the efficient collection of images showing the face of person T.

[0037] Returning to the explanation of Figure 2, let's describe the training data storage unit 533. The training data storage unit 533 stores training data. Training data is a group of images that satisfy the collection conditions described above.

[0038] The image processing model storage unit 534 stores image processing models. The image processing model is a so-called VQA, which is a model that generates natural language answers to natural language questions about images. However, the image processing model storage unit 534 is not limited to VQA, and may be a model (AI) trained to detect any target.

[0039] Next, the control unit 54 will be described. The control unit 54 is a controller that controls the entire camera 50. For example, the control unit 54 can be implemented by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Alternatively, the control unit 54 may be implemented by an integrated circuit such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0040] In the example shown in Figure 2, the control unit 54 includes a setting unit 541, an analysis unit 542, a data collection unit 543, and a learning unit 544. The setting unit 541 sets the data collection conditions for the images that will be used as learning data.

[0041] Specifically, the setting unit 541 sets the coordinate conditions of the detection target as a collection condition. More specifically, the setting unit 541 sets the coordinate conditions for a specific location of the detection target as a collection condition. For example, the setting unit 541 sets the height at which the camera 50 is installed and the imaging range of the camera 50 as initial parameters, and sets the coordinate conditions for the face of the person to be detected.

[0042] Furthermore, the setting unit 541 sets orientation conditions related to the orientation of the object to be detected as collection conditions. Specifically, when detecting the condition of a person's face by image processing, the person's face must be facing the camera 50. In other words, in such cases, even if the above coordinate conditions are met, if the person is not facing the camera 50 (for example, with their back to the camera 50), it will be difficult to analyze the condition of the person's face by analyzing the image.

[0043] Therefore, for example, the setting unit 541 can collect appropriate images as training data by setting the orientation condition for the orientation of the object to be detected to the condition that the person is facing the camera 50.

[0044] The analysis unit 542 performs various analysis processes using an image processing model on the image captured by the imaging unit 52.

[0045] For example, the analysis unit 542 analyzes whether the detection target (e.g., a person) in the image captured by the imaging unit 52 is in a pre-set state. More specifically, the analysis unit 542 inputs the image and question text to an image processing model, which is a VQA, and obtains the answer from the image processing model.

[0046] For example, the question text might be something like, "Is the face visible in the picture?" or "Is that person wearing glasses?", which are questions about the state of the object being detected (for example, a person such as a worker).

[0047] For such a question, the VQA image processing model will output a "Yes" or "No" response. The analysis unit 542 performs this process for each image, thereby analyzing the state of each detection target in each image.

[0048] The collection unit 543 collects images from the camera 50 that satisfy the collection conditions set by the setting unit 541 as training data. For example, first, the collection unit 543 detects the skeletons of people in the images using a skeleton detection model (AI) that has been trained to detect the skeletons of people in the images.

[0049] At this point, the data collection unit 543 determines from the skeletal detection results whether the person is facing the camera 50, and performs the following processing on images in which the person is facing the camera 50. In other words, at this stage, images in which the person's skeleton is not detected, or images in which the person's skeleton is detected but the person is not facing the camera 50, are excluded from the training data.

[0050] The collection unit 543 calculates the coordinate information of the person's head from the detected person's skeleton and determines whether the calculated coordinate information satisfies the coordinate conditions set as collection conditions. Here, the determination of collection conditions according to the embodiment will be explained using Figure 5.

[0051] Figure 5 is an explanatory diagram of the collection condition determination process according to the embodiment. Figure 5 schematically shows an image captured by camera 50. In Figure 5, reference point P0 is shown to be directly below camera 50, and detection point Pt is shown to be the coordinates where a person's head was detected.

[0052] For example, in such a case, as shown in Figure 4, the data collection unit 543 determines whether the detection angle γ of the person T's head at the detection point Pt falls within the range of a first angle α to a second angle β. At this time, the pixel difference from the image center (reference point P0) to the detection point Pt is calculated, and since the height of the camera 50 is known, the detection angle γ of the person T's head can be calculated using trigonometric functions based on the pixel difference and the height of the camera 50.

[0053] In this case, to establish a correspondence between pixels and the actual scale, a known point P1 whose distance from the reference point P0 is known is set in advance. Note that the known point P1 can also be set by placing an arbitrary object.

[0054] The collection unit 543 then determines whether the detected angle γ of the person T's head, calculated based on the detection point Pt, falls within the range of the first angle α to the second angle β set as collection conditions. The collection unit 543 then collects images for which it has determined the detected angle γ satisfies the collection conditions as training data.

[0055] Returning to the explanation of Figure 2, let's describe the learning unit 544. The learning unit 544 performs training of the image processing model using the training data collected by the collection unit 543. For example, the learning unit 544 retrains the image processing model when the amount of training data reaches a predetermined value. Various known methods can be used to retrain the image processing model.

[0056] Thus, in this disclosure, images that meet the collection conditions are collected as training data in the environment in which the camera 50 is actually installed, so that appropriate training data can be collected in the environment in which the camera 50 is installed.

[0057] Next, the processing procedure performed by the camera 50 according to the embodiment will be described using Figure 6. Figure 6 is a flowchart showing the processing procedure performed by the camera 50 according to the embodiment.

[0058] As shown in Figure 6, the camera 50 first acquires an image captured by the imaging unit 52 (step S101). Next, the camera 50 performs skeleton detection using a skeleton detection model (step S102).

[0059] Next, camera 50 calculates the coordinates of the head detected by skeletal detection (step S103). Then, camera 50 determines whether the detected angle γ of person T's head, calculated based on the head coordinates, satisfies the collection conditions (step S104).

[0060] If camera 50 determines that the detection angle γ of person T's head satisfies the collection conditions (step S104; Yes), it adds the corresponding image to the training data (step S105). If, in the determination in step S104, camera 50 determines that the detection angle γ of person T's head does not satisfy the collection conditions (step S104; No), it proceeds to the process in step S101.

[0061] Next, camera 50 determines whether or not the administrator has performed an operation to end the learning process (step S106). If the administrator has performed an operation to end the process (step S106; Yes), camera 50 terminates the process. If, in the determination in step S106, the administrator has not performed an operation to end the process (step S106; No), camera 50 proceeds to the process in step S101.

[0062] As described above, the camera 50 (an example of an information processing device) according to the embodiment comprises a setting unit 541 that sets the conditions for collecting images to be used as training data, and a collection unit 543 that collects images from among the images captured by the camera 50 that satisfy the collection conditions set by the setting unit 541 as training data.

[0063] Therefore, according to the camera 50 of this embodiment, appropriate training data can be collected in the environment in which the camera 50 is installed.

[0064] While embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0065] 1 Server device 50 Cameras 51 Communications Department 52 Imaging Department 53 Storage section 54 Control Unit 531 Image data storage unit 532 Collection Condition Storage Unit 533 Learning Data Storage Unit 534 Image Processing Model Storage Unit 541 Settings Section 542 Analysis Department 543 Collection Department 544 Learning Department S Information Processing System

Claims

1. A setting unit for setting the conditions for collecting images to be used as training data; A collection unit collects images from the images captured by the camera that satisfy the collection conditions set by the setting unit as training data; An information processing device equipped with the following:

2. The aforementioned setting unit is, As the aforementioned collection conditions, set the coordinate conditions of the target to be detected. The aforementioned collection unit is The images whose coordinates satisfy the coordinate conditions are collected as training data. The information processing apparatus according to claim 1.

3. The setting unit is, As the collection conditions, coordinate conditions are set for the specific location of the detection target, The aforementioned collection unit is The image in which the coordinates of the specific location to be detected satisfy the coordinate conditions is collected as training data. The information processing apparatus according to claim 2.

4. The aforementioned setting unit is, As the collection conditions, a directional condition is set regarding the orientation of the object to be detected. The aforementioned collection unit is The images whose orientation satisfies the orientation condition are collected as training data. The information processing apparatus according to claim 1.

5. A learning unit that performs training on an image processing model that analyzes the state of the target to be detected from the image, based on the training data collected by the collection unit; The information processing apparatus according to claim 1, comprising the following:

6. The procedure for setting the conditions for collecting images to be used as training data; A collection procedure for collecting images from among those captured by the camera that satisfy the collection conditions set by the setting procedure as training data; An information processing program that causes a computer to execute an action.

Images