Information processing device, information processing method, and program

Abstract

[Problem] To meet the need for the provision of a technology with which it is possible to suppress the cost of collecting training data required for the generation of an inference model. [Solution] Provided is an information processing device comprising: a training data generation unit for generating second training data on the basis of failure of detection information to satisfy a first condition, the detection information having been obtained on the basis of sensor data detected by a sensor and a first inference model generated through training based on first training data, and being associated with detection of a prescribed event; and a retraining unit for performing retraining on the basis of the second training data and obtaining a second inference model.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, a technique for detecting a predetermined event based on data obtained by a sensor and a learned inference model has been known. In such a technique, the appropriate inference model may differ depending on the environment where the sensor is provided. Therefore, a technique has been disclosed in which information such as the environment where the sensor is provided is transmitted from the sensor to the server, the server selects an inference model corresponding to the information, and returns the selected inference model to the sensor (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, it is desired to provide a technique capable of suppressing the cost of collecting learning data required for generating an inference model.

Means for Solving the Problems

[0005] According to an aspect of the present disclosure, a learning data generation unit that generates second learning data based on the fact that detection information related to the detection of a predetermined event, obtained based on a first inference model generated by learning based on first learning data and sensor data detected by a sensor, does not satisfy a first condition, and a relearning unit that performs relearning based on the second learning data to obtain a second inference model are provided.

[0006] Furthermore, from another perspective of this disclosure, an information processing method is provided, which includes generating second training data based on the fact that detection information related to the detection of a predetermined event, obtained based on a first inference model generated by learning based on first training data and sensor data detected by a sensor, does not satisfy a first condition, and a processor retraining based on the second training data to obtain a second inference model.

[0007] Furthermore, from another perspective of this disclosure, a program is provided that causes a computer to function as an information processing device comprising: a learning data generation unit that generates second learning data based on detection information related to the detection of a predetermined event obtained based on a first inference model generated by learning based on first learning data and sensor data detected by a sensor, and that does not satisfy a first condition; and a retraining unit that performs retraining based on the second learning data to obtain a second inference model. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram illustrating an example configuration of an information processing system related to a comparative example. [Figure 2] This is a diagram illustrating an example configuration of an information processing system according to an embodiment of this disclosure. [Figure 3] This flowchart shows an example of the operation of an information processing system according to the embodiment of this disclosure. [Figure 4] This figure illustrates a first example of processing performed by an information processing system according to an embodiment of the present disclosure. [Figure 5] This figure illustrates a second example of processing performed by an information processing system according to an embodiment of the present disclosure. [Figure 6] This figure illustrates a third example of processing performed by an information processing system according to an embodiment of the present disclosure. [Figure 7] This figure illustrates a fourth example of processing performed by an information processing system according to an embodiment of the present disclosure. [Figure 8] This is a block diagram showing an example of the hardware configuration of an information processing device. [Modes for carrying out the invention]

[0009] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0010] Furthermore, in this specification and drawings, multiple components having substantially identical or similar functional configurations may be distinguished by adding different numbers after the same reference numeral. However, if there is no need to particularly distinguish each of multiple components having substantially identical or similar functional configurations, only the same reference numeral will be used. Also, similar components of different embodiments may be distinguished by adding different letters after the same reference numeral. However, if there is no need to particularly distinguish each of similar components, only the same reference numeral will be used.

[0011] The explanation will be given in the following order. 0. Overview 1. Details of the Embodiment 1.1. Example System Configuration 1.2. Example of Operation 1.3. Feature Details 2. Hardware Configuration Example 3. Summary

[0012] <0. Overview> First, an overview of the embodiments of this disclosure will be described. In recent years, a technology has been known that detects a predetermined event based on data obtained by a sensor and a trained inference model. In such a technology, it is common for an operator to manually collect the training data necessary for generating the inference model. It is also common for an operator to manually label the training data with correct answers. Such a general technology will be described as a comparative example.

[0013] FIG. 1 is a diagram for explaining a configuration example of an information processing system according to a comparative example. As shown in FIG. 1, an information processing system 8 according to the comparative example includes a server 80, a network N1, and image sensors 21 to 23.

[0014] The image sensors 21 to 23 are configured to be communicable with the server 80 via the network N1. Further, an inference model M1 is stored in the memory mounted on the image sensor 21, an inference model M⏟2 is stored in the memory mounted on the image sensor 22, and an inference model M3 is stored in the memory mounted on the image sensor 23. The server 80 includes a meta information acquisition unit 81 and a re - learning unit 82.

[0015] The meta information acquisition unit 81 acquires the meta information of each of the image sensors 21 to 23. For example, the meta information may include environmental information related to the environment in which each of the image sensors 21 to 23 exists.

[0016] In the comparative example, an operator performs data collection B1 including acquisition of learning images according to the meta information acquired by the meta information acquisition unit 81 and correct label assignment to the learning images. The re - learning unit 82 performs re - learning based on the learning images and the correct labels. Thereby, an inference model corresponding to the environment in which each of the image sensors 21 to 23 exists can be generated.

[0017] However, there is a situation that the data collection cost including acquisition of learning images by an operator and correct label assignment is likely to increase. In particular, in order to collect real - life images as learning images, it is necessary to change the environment where the sensor exists, etc., and the learning images and correct labels, acquisition so the data collection cost can become enormous.

[0018] Therefore, in this specification, a technique capable of suppressing the data collection cost of learning data required for generating an inference model is mainly proposed.

[0019] The embodiments of this disclosure have been described above.

[0020] <1. Details of the Embodiment> Next, embodiments of this disclosure will be described in detail.

[0021] (1.1. Example System Configuration) First, an example of the configuration of an information processing system according to the embodiment of this disclosure will be described.

[0022] Figure 2 is a diagram illustrating an example configuration of an information processing system according to an embodiment of the present disclosure. As shown in Figure 2, the information processing system 1 according to an embodiment of the present disclosure comprises a server 10, a network N1, and image sensors 21 to 23. The server 10 may be implemented by a computer. The server 10 is connected to the network N1 and can communicate with each of the image sensors 21 to 23 via the network N1.

[0023] (Image sensors 21-23) Image sensors 21-23 are located in different places from each other. In this embodiment of the disclosure, we mainly assume that the information processing system 1 is equipped with three image sensors. However, the type of sensor equipped in the information processing system 1 is not limited to image sensors, as will be explained later. Also, the number of image sensors is not limited to three; there may be multiple sensors.

[0024] Each of the image sensors 21 to 23 obtains sensor data by performing sensing. More specifically, each of the image sensors 21 to 23 detects an image as an example of sensor data by taking images of the surroundings. Image sensors 21 to 23 are equipped with AI chips to realize AI (Artificial Intelligence) processing functions. More specifically, the AI ​​chips include a computing unit and memory. The memory installed in image sensor 21 stores inference model M1, the memory installed in image sensor 22 stores inference model M2, and the memory installed in image sensor 23 stores inference model M3.

[0025] Each of the inference models M1 to M3 is a pre-trained model generated by training based on training images (the first training data). Here, the machine learning algorithm used for training may typically be a neural network (e.g., deep learning), but it is not limited to this example. For example, the machine learning algorithm used for training may be an SVM (Support Vector Machine) or any other machine learning algorithm.

[0026] Note that inference models M1 to M3 may be different from each other, or at least two of the inference models M1 to M3 may be the same.

[0027] The computing device mounted on the image sensor 21 inputs the image obtained by imaging into the inference model M1, and obtains the output from the inference model M1 corresponding to the image input to the inference model M1 as the inference result.

[0028] In embodiments of the present invention, it is mainly assumed that the computing device mounted on the image sensor 21 obtains as inference results the number of times it detects whether or not a predetermined event has occurred in the image (hereinafter also referred to as the "number of detections") and the number of times it determines that a predetermined event has occurred in the image out of the number of detections (hereinafter also referred to as the "number of event determinations"). However, the inference results output from the inference model M1 are not limited to this example.

[0029] Furthermore, an example of a specified event is a suspicious person appearing in an image. However, the specified event is not limited to such examples. For example, the specified event may be an object other than a suspicious person appearing in an image, or it may be an abnormal condition occurring within the image (for example, a fire breaking out).

[0030] The computing device mounted on the image sensor 21 obtains detection information related to the detection of a predetermined event. Here, we mainly assume that the computing device mounted on the image sensor 21 calculates a detection rate related to the detection of a predetermined event as an example of detection information. In particular, we mainly assume that the detection rate related to the detection of a predetermined event is calculated as the ratio of the number of times the predetermined event occurred to the number of images (i.e., the number of frames) for which inference was performed. However, the detection information obtained by the computing device mounted on the image sensor 21 is not limited to the detection rate related to the detection of a predetermined event.

[0031] Furthermore, the computing unit mounted on image sensor 22, similar to the computing unit mounted on image sensor 21, inputs the image obtained through imaging into inference model M2, obtains the output from inference model M2 corresponding to the image input to inference model M2 as an inference result, and calculates the detection rate as an example of detection information. In addition, the computing unit mounted on image sensor 23, similar to the computing unit mounted on image sensor 21, inputs the image obtained through imaging into inference model M3, obtains the output from inference model M3 corresponding to the image input to inference model M3 as an inference result, and calculates the detection rate as an example of detection information.

[0032] Furthermore, the image sensor 21 has a timing function, which allows it to obtain the current time. Additionally, the image sensor 21 has a temperature sensor, which measures the temperature of the environment in which the image sensor 21 is located, thereby obtaining the ambient temperature. Furthermore, the image sensor 21 obtains various camera parameters. These camera parameters may include gain settings, aperture values, focal length, and sensor identification numbers.

[0033] Furthermore, the image sensor 22, like the image sensor 21, obtains the current time, ambient temperature, and various camera parameters. In addition, the image sensor 23, like the image sensor 21, obtains the current time, ambient temperature, and various camera parameters.

[0034] Image sensor 21 transmits metadata including the inference result, detection rate related to the detection of a predetermined event, current time, ambient temperature, and camera parameters, along with the image obtained through imaging, to server 10 via network N1. Image sensors 22 and 23 also transmit metadata and images to server 10 via network N1, similar to image sensor 21.

[0035] (Server 10) As shown in Figure 2, the server 10 is implemented by a computer and includes a metadata acquisition unit 11, a retraining necessity determination unit 12, a training data generation unit 13, and a retraining unit 14. Since the server 10 performs similar processing on image sensors 21 to 23, the following description will mainly focus on the processing performed on image sensor 21, and details of the processing performed on image sensors 22 and 23 will be omitted.

[0036] The metadata acquisition unit 11, the retraining necessity determination unit 12, the training data generation unit 13, and the retraining unit 14 are provided in a control unit (not shown). For example, the control unit (not shown) may be composed of one or more CPUs (Central Processing Units). When the control unit (not shown) is composed of a processing unit such as a CPU, such a processing unit may be composed of electronic circuits. The control unit (not shown) can be realized by the execution of a program by such a processing unit.

[0037] (Metadata acquisition unit 11) The metadata acquisition unit 11 acquires metadata and images from the image sensor 21 via the network N1. Furthermore, the metadata acquisition unit 11 generates other metadata based on the images acquired from the image sensor 21. This other metadata may include image brightness, light source bias, subject bias, noise characteristics (white spots, random noise), lens contamination, lens misalignment, labels attached to the object (subject), and the orientation of the object (subject). Similarly, the metadata acquisition unit 11 generates other metadata based on the images acquired from the image sensor 22 and the image sensor 23, respectively.

[0038] (Retraining necessity determination unit 12) The retraining necessity determination unit 12 determines whether the detection rate included in the metadata acquired from the image sensor 21 via the network N1 satisfies a first condition. Here, the first condition mainly assumes that the detection rate related to the detection of a predetermined event is within an assumed range (first range).

[0039] Furthermore, if the detection rate related to the detection of a predetermined event does not satisfy the first condition, the retraining necessity determination unit 12 may determine whether the environmental information related to the environment in which the image sensor 21 exists satisfies the second condition. For example, the environmental information may include the brightness of the environment in which the image sensor 21 exists. In this case, the second condition may include the condition that the brightness of the environment is within the second range. The brightness of the environment can be determined from the current time or the brightness of the image, etc., which are included in the metadata.

[0040] Alternatively, the retraining requirement determination unit 12 may determine whether the sensor information related to the image sensor 21 satisfies a third condition if the detection rate related to the detection of a predetermined event does not satisfy the first condition. For example, the sensor information may include the amount of noise present in the image. In this case, the third condition may include the condition that the amount of noise is within a third range. The amount of noise can be determined from the gain setting, aperture value, ambient temperature, noise characteristics, lens contamination, or lens misalignment, which are included in the metadata.

[0041] Alternatively, the retraining necessity determination unit 12 may determine whether the object information related to objects surrounding the image sensor 21 satisfies the fourth condition if the detection rate related to the detection of a predetermined event does not satisfy the first condition. For example, the object information may include the type of object recognized from the image. In this case, the fourth condition may include the condition that the type of object recognized from the image is present in the training image used to train the inference model M1. The type of object can be determined from the labels attached to the subject, the bias of the light source, the bias of the subject, etc., which are included in the metadata.

[0042] Alternatively, the object information may include the pose of the object recognized from the image. In this case, the fourth condition may include the condition that the pose of the object recognized from the image is a predetermined pose. The object's pose can be determined from the subject's pose, which is included in the metadata.

[0043] (Training data generation unit 13) The learning data generation unit 13 generates a training image (second learning data) based on the fact that the detection rate related to the detection of a predetermined event does not satisfy the first condition. More specifically, in the embodiments of this disclosure, the learning data generation unit 13 generates a training image using simulation technology based on the fact that the detection rate related to the detection of a predetermined event does not satisfy the first condition.

[0044] The simulation technique may include a technique for generating training images based on environmental information related to the environment in which the image sensor 21 exists. That is, the training data generation unit 13 may generate training images based on the condition that the detection rate related to the detection of a predetermined event is not within the expected range, and that the environmental information does not satisfy the second condition.

[0045] Alternatively, the simulation technique may include a technique for generating training images based on sensor information related to the image sensor 21. That is, the training data generation unit 13 may generate training images based on the condition that the detection rate related to the detection of a predetermined event is not within the expected range, and that the sensor information does not satisfy a third condition.

[0046] Alternatively, the simulation technique may include a technique for generating training images based on object information related to objects present around the image sensor 21. That is, the training data generation unit 13 may generate training images based on the condition that the detection rate related to the detection of a predetermined event is not within the expected range, and that the object information does not satisfy a fourth condition.

[0047] Furthermore, simulation techniques include methods for placing objects in a virtual space, adjusting light sources in that virtual space (e.g., position, direction, and brightness of light sources), and generating CG images based on the virtual space after the light source adjustments and the placement of the objects. For example, CG images may be generated by ray tracing. In addition, simulation techniques may include sensor simulation techniques that adjust noise parameters in the generated CG images according to sensor information.

[0048] In other words, the learning data generation unit 13 can adjust the light source based on environmental information when generating CG images. Alternatively, the learning data generation unit 13 can control objects placed in the virtual space based on object information when generating CG images. Alternatively, the learning data generation unit 13 can adjust the noise parameters added to the CG image based on sensor information in sensor simulations.

[0049] The learning data generation unit 13 generates correct answer data corresponding to the generated training images and attaches the generated correct answer data to the training images.

[0050] (Relearning Section 14) The retraining unit 14 performs retraining based on the training images generated by the training data generation unit 13 to obtain an inference model (second inference model). More specifically, the retraining unit 14 generates an inference model by performing retraining based on the training images and the ground truth data attached to the training images.

[0051] For example, the retraining unit 14 may perform transfer learning based on training images generated by the training data generation unit 13, using the same inference model M1 stored in the memory mounted on the image sensor 21. Domain adaptation techniques may also be used for such transfer learning.

[0052] Alternatively, the retraining unit 14 may retrain the model from scratch based on training images generated by the training data generation unit 13, rather than using the same inference model M1 stored in the memory mounted on the image sensor 21.

[0053] The retraining unit 14 outputs the generated inference model to the image sensor 21. As a result, the inference model M1 stored in the memory mounted on the image sensor 21 is updated with the inference model output by the retraining unit 14.

[0054] As described above, according to the embodiments of the present disclosure, the training images necessary for generating the inference model are automatically generated, thereby reducing the cost of collecting training images. Furthermore, according to the embodiments of the present disclosure, training images can be generated according to the environment in which the image sensor exists, the camera parameters of the image sensor, or objects present around the image sensor.

[0055] The above describes an example of the configuration of the information processing system 1 according to the embodiment of this disclosure.

[0056] (1.2. Example of operation) Next, an example of the operation of the information processing system 1 according to the embodiment of this disclosure will be described.

[0057] Figure 3 is a flowchart showing an example of the operation of the information processing system 1 according to the present disclosure. The computing device mounted on the image sensor 21 inputs the image obtained by imaging to the inference model M1 and obtains the output from the inference model M1 corresponding to the image input to the inference model M1 as the inference result. In the embodiment of the present invention, the computing device mounted on the image sensor 21 obtains as the inference result the number of times it has detected whether or not a predetermined event has occurred in the image (number of detections) and the number of times it has determined that a predetermined event has occurred in the image out of the number of detections (number of event determinations).

[0058] The computing unit mounted on the image sensor 21 calculates the detection rate related to the detection of a predetermined event. The image sensor 21 transmits the inference result and the detection rate related to the detection of the predetermined event to the server 10 via the network N1.

[0059] In the server 10, the metadata acquisition unit 11 acquires the detection rate from the image sensor 21 via the network N1 (S11). Furthermore, the metadata acquisition unit 11 acquires environmental information related to the environment in which the image sensor 21 exists, sensor information related to the image sensor 21, or object information related to objects present around the image sensor 21 (S12). The environmental information, sensor information, or object information may be acquired from the image sensor 21, or may be acquired based on the image acquired from the image sensor 21.

[0060] The retraining necessity determination unit 12 determines whether the condition that the detection rate related to the detection of a predetermined event is within the expected range is met (S13). If the condition that the detection rate is within the expected range is met (YES in S13), the operation proceeds to S11. On the other hand, if the retraining necessity determination unit 12 does not meet the condition that the detection rate is within the expected range (NO in S13), it determines whether the environmental information, sensor information, or object information meets the condition (S14).

[0061] If the environmental information, sensor information, or object information satisfies the conditions (YES in S14), the operation proceeds to S11. On the other hand, if the environmental information, sensor information, or object information does not satisfy the conditions (NO in S14), the learning data generation unit 13 generates images for training using simulation technology (S15). The learning data generation unit 13 also adds correct answer data to the training images.

[0062] The retraining unit 14 generates an inference model by retraining based on the training images generated by the training data generation unit 13 and the correct data attached to the training images (S16). The retraining unit 14 transmits the generated inference model to the image sensor 21 (S17). The computing unit mounted on the image sensor 21 updates the inference model M1 stored in memory with the inference model transmitted from the retraining unit 14 (S18).

[0063] The above describes an example of the operation of the information processing system 1 according to the embodiment of this disclosure.

[0064] (1.3. Feature Details) Next, the functional details of the information processing system 1 according to the embodiment of this disclosure will be described. More specifically, the first to fourth examples of processing performed by the information processing system 1 according to the embodiment of this disclosure will be described in order.

[0065] (Example 1) First, a first example of processing performed by the information processing system 1 according to the embodiment of this disclosure will be described. More specifically, the first example may correspond to an example of generating a training image according to the brightness of the environment in which the image sensor 21 is present.

[0066] Figure 4 is a diagram illustrating a first example of processing performed by the information processing system 1 according to the embodiment of this disclosure. Referring to Figure 4, an image sensor 21 is shown. Here, we mainly assume that the image sensor 21 is installed in an urban area (for example, a shopping mall) and that it detects suspicious persons as a predetermined event. However, the location where the image sensor 21 is installed and the targets detected by the computing device mounted on the image sensor 21 are not limited.

[0067] Furthermore, referring to Figure 4, we see image A11 captured by the image sensor 21. In the example shown in Figure 4, because the environment in which the image sensor 21 is located is dark, image A11 captured by the image sensor 21 is also dark, which hinders the normal detection of the predetermined event. The computing device mounted on the image sensor 21 inputs the image obtained by imaging into an inference model and obtains the output from the inference model corresponding to the image input to the inference model as the inference result.

[0068] The computing unit mounted on the image sensor 21 obtains as inference results the number of times it detected whether or not a suspicious person was present in the image (detection count) and the number of times it determined that a suspicious person was present in the image out of the detection count (event determination count). Furthermore, the computing unit mounted on the image sensor 21 calculates a detection rate related to the detection of suspicious persons, which is the ratio of the number of times it detected whether or not a suspicious person was present to the number of images (i.e., the number of frames) for which inference was performed.

[0069] The image sensor 21 transmits the inference results and images obtained in this manner to the server 10 via the network N1. The metadata acquisition unit 11 acquires the image from the image sensor 21 via the network N1. Furthermore, based on the image acquired from the image sensor 21, the metadata acquisition unit 11 calculates the brightness of the image as an example of the brightness of the environment in which the image sensor 21 exists.

[0070] More specifically, the metadata acquisition unit 11 calculates the brightness of an image based on the average pixel values ​​of the image acquired from the image sensor 21. Graph G11 shows the brightness distribution for each image. Note that ambient brightness may be acquired by other methods. For example, if the current time is transmitted as metadata from the image sensor 21, the metadata acquisition unit 11 may estimate the ambient brightness based on that current time.

[0071] The retraining necessity determination unit 12 determines whether the condition that the detection rate is within the expected range is met. Furthermore, the retraining necessity determination unit 12 determines whether the condition that the brightness of the image is within the expected range is met. Graph G12 shows the expected brightness R1. For example, if the brightness of the image is 10% or less of the minimum value of this expected brightness R1, it may be determined that the brightness of the image is outside the expected range.

[0072] The training data generation unit 13 generates training images according to the brightness of the image, based on the condition that the detection rate is not within the expected range and the condition that the brightness of the image is not within the expected range. Referring to Figure 4, training images B11 to B13 are shown as examples of training images generated by the training data generation unit 13.

[0073] More specifically, the learning data generation unit 13 adjusts the brightness of the light source based on the brightness of the image and generates a CG image based on the brightness-adjusted light source. Then, the learning data generation unit 13 applies sensor simulation technology to the generated CG image to generate a training image. The learning data generation unit 13 generates ground truth data (for example, whether or not there is a suspicious person) corresponding to the generated training image and attaches the generated ground truth data to the training image.

[0074] The relearning unit 14 is , studies The retraining unit 14 generates an inference model L1 by retraining based on the training images and the correct data attached to the training images. As already explained, the retraining unit 14 may perform transfer learning based on the training images using the same inference model as the inference model stored in the memory mounted on the image sensor 21. Alternatively, the retraining unit 14 may retrain from the beginning based on the training images without using the same inference model as the inference model stored in the memory mounted on the image sensor 21.

[0075] The retraining unit 14 outputs the generated inference model to the image sensor 21. As a result, the inference model stored in the memory mounted on the image sensor 21 is updated with the inference model L1 output by the retraining unit 14.

[0076] The above describes a first example of processing performed by the information processing system 1 according to the embodiment of this disclosure.

[0077] (Second example) Next, a second example of processing performed by the information processing system 1 according to the embodiment of this disclosure will be described. More specifically, the second example may correspond to an example of generating training images according to the amount of noise present in an image.

[0078] Figure 5 is a diagram illustrating a second example of processing performed by the information processing system 1 according to the embodiment of this disclosure. Referring to Figure 5, an image sensor 21 is shown. Here, we mainly assume that the image sensor 21 is installed in an urban area (for example, a shopping mall) and that it detects suspicious persons as a predetermined event. However, the location where the image sensor 21 is installed and the targets detected by the computing device mounted on the image sensor 21 are not limited.

[0079] Furthermore, referring to Figure 5, we see image A21 captured by the image sensor 21. In the example shown in Figure 5, the amount of noise in image A21 is high, which hinders the normal detection of the predetermined event. The computing device mounted on the image sensor 21 inputs the image obtained by imaging into an inference model and obtains the output from the inference model corresponding to the image input to the inference model as the inference result.

[0080] The computing unit mounted on the image sensor 21 obtains as inference results the number of times it detected whether or not a suspicious person was present in the image (detection count) and the number of times it determined that a suspicious person was present in the image out of the detection count (event determination count). Furthermore, the computing unit mounted on the image sensor 21 calculates a detection rate related to the detection of suspicious persons, which is the ratio of the number of times it detected whether or not a suspicious person was present to the number of images (i.e., the number of frames) for which inference was performed.

[0081] The image sensor 21 transmits the inference results and images obtained in this manner to the server 10 via the network N1. The metadata acquisition unit 11 acquires the image from the image sensor 21 via the network N1. Furthermore, the metadata acquisition unit 11 calculates the amount of noise in the image based on the image acquired from the image sensor 21.

[0082] More specifically, the metadata acquisition unit 11 calculates the noise level for each of the multiple frames acquired from the image sensor 21. Graph G21 shows the distribution of the number of images (frames) for each noise level. Note that the noise level of an image may be acquired by other methods. For example, if the image sensor 21 transmits metadata such as gain setting, aperture value, ambient temperature, noise characteristics, lens dirt, or lens misalignment, the metadata acquisition unit 11 may calculate the noise level of an image based on the gain setting, aperture value, ambient temperature, noise characteristics, lens dirt, or lens misalignment.

[0083] The retraining necessity determination unit 12 determines whether the condition that the detection rate is within the expected range is met. Furthermore, the retraining necessity determination unit 12 determines whether the condition that the amount of noise in the image is within the expected range is met. Graph G22 shows the expected noise amount R22. For example, if the standard deviation of the amount of noise in the image is 30% or more greater than the expected value, it may be determined that the amount of noise in the image is outside the expected range.

[0084] The training data generation unit 13 generates training images corresponding to the noise level of the image based on the condition that the detection rate is not within the expected range and the condition that the amount of noise in the image is not within the expected range. Referring to Figure 5, training images B21 to B23 are shown as examples of training images generated by the training data generation unit 13.

[0085] More specifically, the training data generation unit 13 generates a CG image. Then, the training data generation unit 13 applies a sensor simulation technique to the CG image, which adjusts the noise parameters according to the amount of noise in the image, to generate a training image.

[0086] The training images may be generated based on images obtained by an image sensor belonging to the same lot as the image sensor 21, or they may be generated based on images obtained by an image sensor belonging to a different lot than the image sensor 21.

[0087] The learning data generation unit 13 generates correct data (for example, whether or not there is a suspicious person) corresponding to the generated learning image, and attaches the generated correct data to the learning image.

[0088] The relearning unit 14 is , studies The retraining unit 14 generates an inference model L1 by retraining based on the training images and the correct data attached to the training images. As already explained, the retraining unit 14 may perform transfer learning based on the training images using the same inference model as the inference model stored in the memory mounted on the image sensor 21. Alternatively, the retraining unit 14 may retrain from the beginning based on the training images without using the same inference model as the inference model stored in the memory mounted on the image sensor 21.

[0089] The retraining unit 14 outputs the generated inference model to the image sensor 21. As a result, the inference model stored in the memory mounted on the image sensor 21 is updated with the inference model L1 output by the retraining unit 14.

[0090] The above describes a second example of processing performed by the information processing system 1 according to the embodiment of this disclosure.

[0091] (Third example) Next, a third example of processing performed by the information processing system 1 according to the embodiment of this disclosure will be described. More specifically, the third example may correspond to an example of generating training images according to the type of object to be recognized from an image.

[0092] Figure 6 is a diagram illustrating a third example of processing performed by the information processing system 1 according to the embodiment of this disclosure. Referring to Figure 6, an image sensor 21 is shown. Here, we mainly assume that the image sensor 21 is installed in an urban area (for example, a shopping mall) and that it detects suspicious persons as a predetermined event. However, the location where the image sensor 21 is installed and the targets detected by the computing device mounted on the image sensor 21 are not limited.

[0093] Furthermore, referring to Figure 6, we see image A31 captured by the image sensor 21. In the example shown in Figure 6, the presence of a person E31 wearing a helmet in image A31 hinders the normal detection of the predetermined event. The computing device mounted on the image sensor 21 inputs the image obtained through imaging into an inference model and obtains the output from the inference model corresponding to the image input as the inference result.

[0094] The computing unit mounted on the image sensor 21 obtains as inference results the number of times it detected whether or not a suspicious person was present in the image (detection count) and the number of times it determined that a suspicious person was present in the image out of the detection count (event determination count). Furthermore, the computing unit mounted on the image sensor 21 calculates a detection rate related to the detection of suspicious persons, which is the ratio of the number of times it detected whether or not a suspicious person was present to the number of images (i.e., the number of frames) for which inference was performed.

[0095] The image sensor 21 transmits the inference results and images obtained in this manner to the server 10 via the network N1. The metadata acquisition unit 11 acquires the image from the image sensor 21 via the network N1. Furthermore, the metadata acquisition unit 11 recognizes the type of object present in the image based on the image acquired from the image sensor 21. Here, it is assumed that the object type recognized is a person E31 wearing a helmet. Note that the recognition of the type of object may also be performed by the processing unit mounted on the image sensor 21.

[0096] The retraining necessity determination unit 12 determines whether the condition that the detection rate is within the expected range is met. Furthermore, the retraining necessity determination unit 12 determines whether the condition that the same type of object as the object recognized from the image is present in the training image used to train the inference model is met. In the example shown in Figure 6, the training image G31 used to train the inference model does not contain a person wearing a helmet.

[0097] The training data generation unit 13 generates training images containing objects of the same type as those recognized, based on the condition that the detection rate is not within the expected range and that the training images used to train the inference model do not contain objects of the same type as those recognized from the image. Referring to Figure 6, training images B31 to B33 are shown as examples of training images generated by the training data generation unit 13, and they contain objects of the same type as those recognized, such as a person wearing a helmet or hat.

[0098] More specifically, the learning data generation unit 13 places a person wearing a helmet or hat, which is the same type of object as the recognized object, as an object in the virtual space, and generates a CG image based on the virtual space in which the object is placed and the light source in the virtual space. Then, the learning data generation unit 13 applies sensor simulation technology to the generated CG image to generate a training image. The learning data generation unit 13 generates ground truth data (for example, whether or not there is a suspicious person) corresponding to the generated training image, and attaches the generated ground truth data to the training image.

[0099] The relearning unit 14 is , studies The retraining unit 14 generates an inference model L1 by retraining based on the training images and the correct data attached to the training images. As already explained, the retraining unit 14 may perform transfer learning based on the training images using the same inference model as the inference model stored in the memory mounted on the image sensor 21. Alternatively, the retraining unit 14 may retrain from the beginning based on the training images without using the same inference model as the inference model stored in the memory mounted on the image sensor 21.

[0100] The retraining unit 14 outputs the generated inference model to the image sensor 21. As a result, the inference model stored in the memory mounted on the image sensor 21 is updated with the inference model L1 output by the retraining unit 14.

[0101] The above describes a third example of processing performed by the information processing system 1 according to the embodiment of this disclosure.

[0102] (Fourth example) Next, a fourth example of processing performed by the information processing system 1 according to the embodiment of this disclosure will be described. More specifically, the fourth example may correspond to an example of generating a training image corresponding to the pose of an object recognized from an image.

[0103] Figure 7 is a diagram illustrating a fourth example of processing performed by the information processing system 1 according to the embodiment of this disclosure. Referring to Figure 7, an image sensor 21 is shown. Here, we mainly assume that the image sensor 21 is installed in a factory and is used to detect a suspicious person as a predetermined event. However, the location where the image sensor 21 is installed and the target of detection by the computing device mounted on the image sensor 21 are not limited.

[0104] Furthermore, referring to Figure 7, we see image A41 captured by the image sensor 21. In the example shown in Figure 7, there is a person E41 in image A41 whose lower body is hidden by an obstruction, which hinders the normal detection of the predetermined event. The computing device mounted on the image sensor 21 inputs the image obtained by imaging into an inference model and obtains the output from the inference model corresponding to the image input to the inference model as the inference result.

[0105] The computing unit mounted on the image sensor 21 obtains as inference results the number of times it detected whether or not a suspicious person was present in the image (detection count) and the number of times it determined that a suspicious person was present in the image out of the detection count (event determination count). Furthermore, the computing unit mounted on the image sensor 21 calculates a detection rate related to the detection of suspicious persons, which is the ratio of the number of times it detected whether or not a suspicious person was present to the number of images (i.e., the number of frames) for which inference was performed.

[0106] The computing unit mounted on the image sensor 21 transmits the inference results and images obtained in this manner to the server 10 via the network N1. The metadata acquisition unit 11 acquires images from the image sensor 21 via the network N1. Furthermore, the metadata acquisition unit 11 recognizes the posture of objects present in the image based on the images acquired from the image sensor 21. Here, it is assumed that the posture of the object recognized is that of a person E41 whose lower body is hidden by an obstruction. Note that the recognition of the object's posture may also be performed by the image sensor 21.

[0107] The retraining necessity determination unit 12 determines whether the condition that the detection rate is within the expected range is met. Furthermore, the retraining necessity determination unit 12 determines whether the condition that the posture of the object recognized from the image is a predetermined posture is met. In the example shown in Figure 7, the condition that the entire body of person E41 is recognized from the image is not met.

[0108] The training data generation unit 13 generates training images containing an object with the same posture as the recognized object, based on the condition that the detection rate is not within the expected range and the condition that the posture of the object recognized from the image is a predetermined posture. Referring to Figure 7, training images B41 to B43 are shown as examples of training images generated by the training data generation unit 13, and depict a person whose lower body is hidden by an obstruction, which is an object with the same posture as the recognized object.

[0109] More specifically, the learning data generation unit 13 places an object in the virtual space that has the same posture as the recognized posture, specifically a person whose lower body is hidden by an obstacle. Based on the virtual space where the object is placed and the light source in the virtual space, it generates a CG image. The learning data generation unit 13 then applies sensor simulation technology to the generated CG image to generate a training image. The learning data generation unit 13 generates ground truth data (for example, whether or not there is a suspicious person) corresponding to the generated training image, and attaches the generated ground truth data to the training image.

[0110] The relearning unit 14 is , studies The retraining unit 14 generates an inference model L1 by retraining based on the training images and the correct data attached to the training images. As already explained, the retraining unit 14 may perform transfer learning based on the training images using the same inference model as the inference model stored in the memory mounted on the image sensor 21. Alternatively, the retraining unit 14 may retrain from the beginning based on the training images without using the same inference model as the inference model stored in the memory mounted on the image sensor 21.

[0111] The retraining unit 14 outputs the generated inference model to the image sensor 21. As a result, the inference model stored in the memory mounted on the image sensor 21 is updated with the inference model L1 output by the retraining unit 14.

[0112] The above describes a fourth example of processing performed by the information processing system 1 according to the embodiment of this disclosure.

[0113] The functional details of the information processing system 1 according to the embodiment of this disclosure have been described above.

[0114] <2. Hardware Configuration Example> Next, with reference to Figure 8, an example of the hardware configuration of an information processing device 900 as an example of a server 10 according to the present disclosure will be described. Figure 8 is a block diagram showing an example of the hardware configuration of an information processing device 900. Note that the server 10 does not necessarily have to have all of the hardware configuration shown in Figure 8, and some of the hardware configuration shown in Figure 8 may not be present in the server 10.

[0115] As shown in Figure 8, the information processing device 900 includes a CPU (Central Processing unit) 901, a ROM (Read Only Memory) 903, and a RAM (Random Access Memory) 905. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925. The information processing device 900 may have a processing circuit such as a DSP (Digital Signal Processor) or an ASIC (Application Specific Integrated Circuit) instead of, or together with, the CPU 901.

[0116] The CPU 901 functions as an arithmetic processing unit and control unit, controlling all or part of the operation within the information processing unit 900 according to various programs recorded in the ROM 903, RAM 905, storage device 919, or removable recording medium 927. The ROM 903 stores programs and arithmetic parameters used by the CPU 901. The RAM 905 temporarily stores programs used in the execution of the CPU 901 and parameters that change as appropriate during its execution. The CPU 901, ROM 903, and RAM 905 are interconnected by a host bus 907, which is composed of internal buses such as the CPU bus. Furthermore, the host bus 907 is connected to an external bus 911, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 909.

[0117] The input device 915 is a device operated by the user, such as a button. The input device 915 may include a mouse, keyboard, touch panel, switch, and lever. The input device 915 may also include a microphone that detects the user's voice. The input device 915 may be a remote control device using infrared or other radio waves, or an external connection device 929 such as a mobile phone that is compatible with the operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs it to the CPU 901. By operating this input device 915, the user inputs various data to the information processing device 900 or instructs it to perform processing operations. The imaging device 933, which will be described later, can also function as an input device by imaging the user's hand movements, fingers, etc. In this case, the pointing position may be determined according to the hand movements and finger orientation.

[0118] The output device 917 is comprised of a device capable of visually or audibly notifying the user of the acquired information. The output device 917 may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker or headphones. The output device 917 may also include a PDP (Plasma Display Panel), a projector, a hologram, a printer, etc. The output device 917 outputs the results obtained from the processing of the information processing device 900 as images such as text or pictures, or as sound such as voice or acoustics. The output device 917 may also include a light to illuminate the surroundings.

[0119] The storage device 919 is a data storage device configured as an example of the storage unit of the information processing device 900. The storage device 919 is composed of, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. This storage device 919 stores programs executed by the CPU 901, various data, and various data acquired from external sources.

[0120] The drive 921 is a reader / writer for removable recording media 927, such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory, and is either built into or external to the information processing device 900. The drive 921 reads information recorded on the installed removable recording media 927 and outputs it to the RAM 905. The drive 921 also writes data to the installed removable recording media 927.

[0121] The connection port 923 is a port for directly connecting equipment to the information processing device 900. The connection port 923 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. Alternatively, the connection port 923 may be an RS-232C port, an optical audio terminal, or an HDMI (High-Definition Multimedia Interface) port. By connecting an external device 929 to the connection port 923, various types of data can be exchanged between the information processing device 900 and the external device 929.

[0122] The communication device 925 is a communication interface, for example, consisting of a communication device for connecting to the network 931. The communication device 925 may be, for example, a communication card for wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB). Alternatively, the communication device 925 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication. The communication device 925 sends and receives signals, for example, to the Internet or other communication devices using a predetermined protocol such as TCP / IP. The network 931 connected to the communication device 925 is a network connected by wire or wireless, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0123] <3. Summary> According to an embodiment of the present disclosure, an information processing device is provided, comprising: a learning data generation unit that generates second learning data based on the fact that detection information related to the detection of a predetermined event, obtained based on a first inference model generated by learning based on first learning data and sensor data detected by a sensor, does not satisfy a first condition; and a retraining unit that performs retraining based on the second learning data to obtain a second inference model. With such a configuration, it is possible to suppress the cost of collecting learning data necessary for generating an inference model.

[0124] While preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear to any person with ordinary skill in the art of the present disclosure that various modifications or alterations may be conceived within the scope of the technical ideas described in the claims, and these will naturally also fall within the technical scope of the present disclosure.

[0125] For example, the above mainly described the case where the sensor provided by the information processing system 1 is an image sensor. However, the sensor provided by the information processing system 1 may be something other than an image sensor. For example, the sensor provided by the information processing system 1 may be a microphone, a distance sensor, a vibration sensor, a gyroscope, an acceleration sensor, a pressure sensor, an odor sensor, a thermal sensor, an infrared sensor, or an ultraviolet sensor. In such cases as well, the same processing as the processing described above for images should be performed on the sensor data obtained by the sensor.

[0126] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that are obvious to those skilled in the art from the description herein, in addition to or instead of the effects described herein.

[0127] Furthermore, the following configurations also fall within the technical scope of this disclosure. (1) A learning data generation unit generates second learning data based on the fact that detection information related to the detection of a predetermined event, obtained based on a first inference model generated by learning based on first learning data and sensor data detected by a sensor, does not satisfy a first condition. A retraining unit that performs retraining based on the second training data to obtain a second inference model, An information processing device equipped with the following features. (2) The training data generation unit generates the second training data using simulation technology based on the fact that the detection information does not satisfy the first condition. The information processing device described in (1) above. (3) The simulation technique includes a technique for generating the second training data based on environmental information related to the environment in which the sensor exists, sensor information related to the sensor, or object information related to objects present around the sensor. The information processing device described in (2) above. (4) The training data generation unit generates the second training data based on the fact that the detection information does not satisfy the first condition and the environmental information does not satisfy the second condition. The information processing device described in (3) above. (5) The aforementioned environmental information includes the brightness of the environment in which the sensor is located. The second condition includes the condition that the brightness is within the second range, The information processing device described in (4) above. (6) The learning data generation unit generates second learning data corresponding to the brightness based on the fact that the detection information does not satisfy the first condition and the condition that the brightness is within the second range. The information processing device described in (5) above. (7) The learning data generation unit generates the second learning data based on the fact that the detection information does not satisfy the first condition and the sensor information does not satisfy the third condition. The information processing device described in (3) above. (8) The sensor information includes the amount of noise present in the sensor data. The third condition includes the condition that the noise amount is within the third range. The information processing device described in (7) above. (9) The training data generation unit generates second training data corresponding to the noise amount based on the fact that the detection information does not satisfy the first condition and the condition that the noise amount is within the third range. The information processing device described in (8) above. (10) The training data generation unit generates the second training data based on the fact that the detection information does not satisfy the first condition and the object information does not satisfy the fourth condition. The information processing device described in (3) above. (11) The object information includes the type of object recognized from the sensor data, The fourth condition includes the condition that objects of the aforementioned type exist in the first training data. The information processing device described in (10) above. (12) The learning data generation unit determines that the detection information does not satisfy the first condition, and Types of objects Based on the fact that the condition that the object of the aforementioned type is not present in the first training data is not met, a second training data is generated in which the aforementioned type of object exists. The information processing device described in (11) above. (13) The object information includes the orientation of the object recognized from the sensor data, The fourth condition includes the condition that the orientation of the object is a predetermined orientation. The information processing device described in (10) above. (14) The learning data generation unit generates second learning data in which an object in the aforementioned posture exists, based on the fact that the detection information does not satisfy the first condition and the condition that the posture of the object does not satisfy the predetermined posture. The information processing device described in (13) above. (15) The detection information includes a detection rate related to the detection of the predetermined event, The first condition includes the condition that the detection rate is within a first range, An information processing device as described in any one of the above items (1) to (14). (16) The retraining unit outputs the second inference model to the sensor. An information processing device as described in any one of the above items (1) to (15). (17) The first inference model is stored in the memory mounted on the sensor, The first inference model stored in the memory mounted on the sensor is updated with the second inference model output by the retraining unit. The information processing device described in (16) above. (18) The sensor includes an image sensor that detects an image as sensor data. An information processing device as described in any one of the above items (1) to (17). (19) Based on the fact that the detection information related to the detection of a predetermined event, obtained from the first inference model generated by learning based on the first training data and sensor data detected by the sensor, does not satisfy the first condition, second training data is generated. The processor retrains based on the second training data to obtain a second inference model, An information processing method comprising: (20) Computers, A learning data generation unit generates second learning data based on the fact that detection information related to the detection of a predetermined event, obtained based on a first inference model generated by learning based on first learning data and sensor data detected by a sensor, does not satisfy a first condition. A retraining unit that performs retraining based on the second training data to obtain a second inference model, A program that enables the device to function as an information processing device equipped with the necessary components. [Explanation of symbols]

[0128] 1. Information Processing System 10 servers 11 Metadata Acquisition Unit 12. Retraining Requirement Determination Unit 13. Training Data Generation Unit 14 Relearning Section 21-23 Image Sensors M1-M3 Inference Models

Claims

1. A learning data generation unit generates second learning data based on the first inference model generated by learning based on first learning data and sensor data detected by the sensor, and the detection information related to the detection of a predetermined event obtained based on the first condition and the condition that the amount of noise present in the sensor data is within a predetermined range, A retraining unit that performs retraining based on the second training data to obtain a second inference model, An information processing device equipped with the following features.

2. The learning data generation unit generates second learning data corresponding to the noise amount based on the fact that the detection information does not satisfy the first condition and the condition that the noise amount is within the predetermined range. The information processing apparatus according to claim 1.

3. The detection information includes a detection rate related to the detection of the predetermined event, The first condition includes the condition that the detection rate is within a first range, The information processing apparatus according to claim 1.

4. The retraining unit outputs the second inference model to the sensor. The information processing apparatus according to claim 1.

5. The first inference model is stored in the memory mounted on the sensor. The first inference model stored in the memory mounted on the sensor is updated with the second inference model output by the retraining unit. The information processing apparatus according to claim 4.

6. The sensor includes an image sensor that detects an image as sensor data. The information processing apparatus according to claim 1.

7. Based on the first inference model generated by learning based on the first training data and the sensor data detected by the sensor, detection information related to the detection of a predetermined event does not satisfy the first condition and does not satisfy the condition that the amount of noise present in the sensor data is within a predetermined range, second training data is generated. The processor retrains based on the second training data to obtain a second inference model, An information processing method comprising:

8. Computers, A learning data generation unit generates second learning data based on the first inference model generated by learning based on first learning data and sensor data detected by the sensor, and the detection information related to the detection of a predetermined event obtained based on the first condition and the condition that the amount of noise present in the sensor data is within a predetermined range, A retraining unit that performs retraining based on the second training data to obtain a second inference model, A program that enables the device to function as an information processing device equipped with the necessary components.

Images