Information processing method, information processing system, program product, and recording medium

Abstract

An information processing method, in which a computer performs the following operations: obtaining a first image and object data of an object reflected in the first image (S10), extracting a portion of the first image corresponding to a difference between an object detection result and the object data, wherein the object detection result is obtained by inputting the first image into a learned model, the learned model being a model that takes an image as input and outputs an object detection result (S60), obtaining a second image, the second image including a portion corresponding to the following object data, the object data being the same object data as the object data corresponding to the extracted portion of the first image (S70), and generating training data for the learned model (S90) by reflecting an image based on the extracted portion of the first image to a portion of the obtained second image corresponding to the same object data (S80).

Description

Technical Field

[0001] The present disclosure relates to an information processing method and an information processing system for generating a training data set for machine learning processing. More specifically, it relates to an information processing method and an information processing system for generating a training data set used in machine learning processing for coping with false detection in an object detection device for detecting general objects from images. Background Art

[0002] In recent years, as a general object detection device, due to the use of deep learning technology, the performance has been greatly improved, and various research or commercialization has been carried out. In addition, even in an autonomous driving car (robot car) operated by a machine instead of a driver, a general object detection device for images transmitted from a camera device or a ranging sensor device such as a radar or a stereo camera is one of the essential functions in an autonomous driving car. General objects refer to pedestrians, passenger cars, road signs, buildings, and road areas. The autonomous driving car controls the main body of the autonomous driving car based on the detection information from the general object detection device, and it is a necessary condition to move safely to the destination.

[0003] Therefore, although a general object detection device needs to correctly detect an object, it cannot deny the occurrence of erroneous detection (hereinafter referred to as false detection). As a method of dealing with this situation, for example, after the erroneously detected image data and detection data are correctly corrected, a re-learning process is performed on the object detection device as training data. This process has the effect that the same false detection is less likely to occur in the object detection device. In a neural network detector, the re-learning process includes transfer learning or fine-tuning.

[0004] Therefore, in the relearning process, a large amount of training data is required for countermeasures against false detection. The method disclosed in Patent Document 1 is to generate training image data including the recognition object by geometrically transforming the area where the recognition object is mapped, generate a synthetic image of the same background part (just the background), a synthetic image of different background parts (unnatural), and a synthetic image of different object parts (unnatural), and generate training image data that does not include the recognition object passed by the identifier.

[0005] (Prior art literature)

[0006] (Patent Document)

[0007] Patent Document 1: Japanese Patent Application Publication No. 2012-88787

[0008] However, in the invention disclosed in Patent Document 1, the recognition performance of the recognizer (object detection unit) is not necessarily improved in re-learning. For example, in Patent Document 1, since the region where the recognition object is reflected is arbitrarily offset or rotated as a geometric transformation, the region where the recognition object is reflected after the geometric transformation is unnatural in the original image, which has a negative impact on the learning effect. Summary of the invention

[0009] Therefore, an object of the present disclosure is to provide an information processing method and an information processing system that can reliably improve the object detection performance of a learned model for object detection in re-learning.

[0010] In an information processing method involved in one form of the present disclosure, a computer performs the following operations: obtain a first image and object data of an object reflected in the first image, extract a portion of the first image corresponding to a difference between an object detection result and the object data, wherein the object detection result is obtained by inputting the first image into a learned model, wherein the learned model is a model that takes an image as input and outputs an object detection result, obtain a second image, the second image including a portion corresponding to the following object data, the object data being the same object data as the object data corresponding to the extracted portion of the first image, and generate training data for the learned model by reflecting an image based on the extracted portion of the first image to a portion of the obtained second image corresponding to the same object data.

[0011] Furthermore, an information processing system involved in one form of the present disclosure includes an acquisition unit and a generation unit, wherein the acquisition unit acquires a first image and object data of an object reflected in the first image, and the generation unit extracts a portion of the first image corresponding to a difference between an object detection result and the object data, wherein the object detection result is obtained by inputting the first image into a learned model, which is a model that takes an image as input and outputs an object detection result, and obtains a second image, wherein the second image includes a portion corresponding to the following object data, which is the same object data as the object data corresponding to the extracted portion of the first image, and training data for the learned model is generated by reflecting an image based on the extracted portion of the first image on the portion of the obtained second image corresponding to the same object data.

[0012] In addition, these general or specific forms may be implemented by systems, methods, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or by any combination of systems, methods, integrated circuits, computer programs, and recording media.

[0013] According to the information processing method and the information processing system according to one aspect of the present disclosure, the object detection performance of a learned model for object detection can be reliably improved during re-learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a block diagram showing the functional configuration of the information processing system according to the first embodiment.

[0015] Figure 2 This is a flowchart showing the operation of the server device according to the first embodiment.

[0016] Figure 3 This is a diagram showing an example of log information obtained by the server device according to the first embodiment.

[0017] Figure 4 An example of an image input to the determination detection unit according to the first embodiment is shown.

[0018] Figure 5 The detection results of the determination detection unit according to the first embodiment are shown.

[0019] Figure 6 An example of a clipped image according to the first embodiment is shown.

[0020] Figure 7 An example of the image group stored in the storage unit is shown.

[0021] Figure 8 The detection results of the determination detection unit for the images included in the image group are shown.

[0022] Fig.9A An example of an image group included in the training data is shown.

[0023] Fig. 9B An example of annotation information included in training data is shown.

[0024] Fig.10 An example of the category structure of object detection is shown.

[0025] Fig.11A This is a flowchart showing a first example of the operation of the generation unit according to the modification of the first embodiment.

[0026] Fig. 11B This is a flowchart showing a second example of the operation of the generation unit according to the modification of the first embodiment.

[0027] Fig. 11CThis is a flowchart showing a third example of the operation of the generation unit according to the modification of the first embodiment.

[0028] Fig.11D This is a flowchart showing a fourth example of the operation of the generation unit according to the modification of the first embodiment.

[0029] Fig.11E This is a flowchart showing a fifth example of the operation of the generation unit according to the modification of the first embodiment.

[0030] Fig.11F This is a flowchart showing a sixth example of the operation of the generation unit according to the modification of the first embodiment.

[0031] Fig.12 This is a flowchart showing the operation of the server device involved in the second embodiment.

[0032] Fig.13 This is a diagram showing an example of log information obtained by the server device according to the second embodiment.

[0033] Fig.14 An example of an image input to the determination detection unit according to the second embodiment is shown.

[0034] Fig.15 An example of an image group included in training data is shown. DETAILED DESCRIPTION

[0035] (Insights that form the basis of this disclosure)

[0036] In the field of various industrial products including self-driving cars, products equipped with neural network-type object detection devices that automatically brake based on the detection results have been launched on the market.

[0037] However, the learning process of the neural network type object detection device requires the collection and annotation of a huge amount of image data. Furthermore, the annotation work is generally done manually and is very costly.

[0038] Therefore, various automated methods for producing training data have been proposed. However, there has been no study on the design and technical solutions for automatically producing training data that effectively counteracts false detection by an object detection device.

[0039] Therefore, the inventors of the present application have conducted intensive studies on the generation of training data for effectively preventing erroneous detection in an object detection device, and have proposed an information processing method and an information processing system to be described below.

[0040] In an information processing method involved in one form of the present disclosure, a computer performs the following operations: obtain a first image and object data of an object reflected in the first image, extract a portion of the first image corresponding to a difference between an object detection result and the object data, wherein the object detection result is obtained by inputting the first image into a learned model, wherein the learned model is a model that takes an image as input and outputs an object detection result, obtain a second image, the second image including a portion corresponding to the following object data, the object data being the same object data as the object data corresponding to the extracted portion of the first image, and generate training data for the learned model by reflecting an image based on the extracted portion of the first image to a portion of the obtained second image corresponding to the same object data.

[0041] Based on this, training data (training data) can be automatically generated. Moreover, the second image is data including a portion corresponding to object data, which is the same object data (for example, object category) as the object data corresponding to the portion of the first image (the misdetected object). The image generated by reflecting (for example, overlapping or replacing) the misdetected object on such a second image becomes a natural image with less sense of incongruity. That is, it is possible to suppress the adverse effects caused by the learning effect due to the training data becoming an unnatural image.

[0042] Therefore, according to the information processing method according to one aspect of the present disclosure, the object detection performance of the learned model for object detection can be reliably improved during re-learning.

[0043] Furthermore, for example, the object detection result is further obtained, and the obtained object detection result is compared with the object data, and then the part of the first image is extracted based on the comparison result.

[0044] This allows all processing to be performed on a computer (eg, a server device). In other words, all processing for generating training data can be performed automatically, so training data can be generated at a lower cost.

[0045] Furthermore, for example, the image based on the portion of the first image is an image obtained by processing the portion of the first image.

[0046] Thus, an image based on the portion of the first image can be obtained from the first image. In other words, since training data can be generated based on the erroneously detected object (the portion of the first image) itself, effective training data can be generated for the erroneously detected object.

[0047] Furthermore, for example, the image based on the portion of the first image is an image similar to the portion of the first image selected from images prepared in advance.

[0048] Accordingly, it is possible to obtain an image based on a portion of the first image without processing the first image. For example, even in a case where it is difficult to cut out a portion of the first image from the first image, it is possible to generate training data using an image similar to the portion of the first image. Furthermore, in a case where the selected image is a 3D image, the posture of the 3D image can be easily changed (for example, tilt, etc.). Therefore, when the selected image is reflected on the second image, a more natural image can be generated, and therefore, more effective training data can be generated.

[0049] Furthermore, for example, when obtaining the second image, an image in which the position or posture of the camera device when shooting is the same as or similar to the position or posture of the camera device when shooting the first image is obtained as the second image.

[0050] This makes it possible to generate training data that is effective in a learning process for a passenger vehicle such as a bus or a taxi that travels on a predetermined route.

[0051] And for example, when obtaining the second image, an image whose time period or weather when shooting is the same as or similar to the time period or weather when shooting the first image is obtained as the second image. And for example, when obtaining the second image, an image whose luminance or brightness in the camera direction when shooting is the same as or similar to the luminance of the first image or the brightness in the camera direction when shooting the first image is obtained as the second image. And for example, when obtaining the second image, an image whose area data around a portion corresponding to the following object data is the same as or similar to the area data around a portion of the first image is obtained as the second image. The object data is the same object data as the object data corresponding to the portion of the first image.

[0052] According to this, it is possible to select a second image that is closer to the environment when the first image was obtained. The image generated by reflecting the erroneously detected object on such a second image becomes an image similar to the environment in which the first image was taken. For example, in the case of fine-tuning, by re-learning using an image with a similar environment, the object detection performance of the learned model for object detection can be reliably improved. Therefore, more effective training data can be generated.

[0053] Furthermore, for example, when obtaining the second image, an image in which the size of a portion corresponding to the following object data is the same as or similar to the size of a portion of the first image is obtained as the second image, and the object data is the same as the object data corresponding to the portion of the first image.

[0054] This makes it possible to generate a natural image with less discomfort, and thus more effective training data can be generated.

[0055] Furthermore, for example, the first image and the second image are camera images, and when obtaining the second image, an image in which the camera parameters of the camera device when shooting are the same as or similar to the camera parameters of the camera device when shooting the first image is obtained as the second image.

[0056] This makes it possible to generate training data including images similar to the environment in which the first image was captured, based on information other than the subject (the erroneously detected object).

[0057] Furthermore, an information processing system involved in one form of the present disclosure includes an acquisition unit and a generation unit, wherein the acquisition unit acquires a first image and object data of an object reflected in the first image, and the generation unit extracts a portion of the first image corresponding to a difference between an object detection result and the object data, wherein the object detection result is obtained by inputting the first image into a learned model, which is a model that takes an image as input and outputs an object detection result, and obtains a second image, wherein the second image includes a portion corresponding to the following object data, which is the same object data as the object data corresponding to the extracted portion of the first image, and training data for the learned model is generated by reflecting an image based on the extracted portion of the first image on the portion of the obtained second image corresponding to the same object data.

[0058] Accordingly, the computer can execute the above-mentioned information processing method according to the program.

[0059] Moreover, these general or specific forms can be implemented by systems, devices, methods, integrated circuits, computer programs, or non-temporary recording media such as computer-readable CD-ROMs, or by any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0060] The following detailed description of the implementation mode is given with reference to the accompanying drawings. In addition, the implementation modes to be described below are all specific examples of the present disclosure. The numerical values, shapes, constituent elements, steps, the order of steps, etc. shown in the following implementation modes are all examples, and the main purpose is not to limit the present disclosure. Furthermore, the constituent elements of the following implementation modes that are not recorded in the independent technical solutions are described as arbitrary constituent elements. Furthermore, in all implementation modes, the various contents can be combined.

[0061] Furthermore, each figure is a schematic diagram and is not a strict illustration.

[0062] Furthermore, in this specification, terms indicating the relationship between identical elements, terms indicating the shape of an element such as a rectangle, numerical values, and numerical ranges do not only express strict meanings but also express substantially equivalent ranges, for example, may include differences of approximately several percent.

[0063] Furthermore, in this specification, the false detection of the object detection device (object detection unit) includes the case where no detection is performed. That is, the false detection includes the case where the object detection unit does not correctly detect the object, and the case where the object is not detected. Furthermore, in Embodiment 1, the case where the object is not correctly detected is described, and in Embodiment 2, the case where an object that is not the detection target is mistakenly detected is described.

[0064] (Implementation Method 1)

[0065] Below, refer to Figures 1 to 10 , the information processing method and information processing system involved in this embodiment are explained.

[0066] [1-1. Configuration of information processing system]

[0067] First, refer to Figure 1 The configuration of the information processing system 1 including the server device 30 according to the present embodiment will be described. Figure 1 It is a block diagram showing the functional configuration of the information processing system 1 according to the present embodiment.

[0068] like Figure 1 As shown, the information processing system 1 includes a mobile object 10 and a server device 30 .

[0069] The moving object 10 is, for example, a vehicle. The vehicle is, for example, an autonomous car that controls the driving of the vehicle without the need for a driver's operation, but may also be a vehicle that can be switched to autonomous driving or manual driving.

[0070] The mobile body 10 has an object detection unit 11 and a communication unit 12. In addition, the mobile body 10 also has a plurality of sensors including an optical sensor such as a camera (not shown) that generates an image of the surroundings of the mobile body 10. The object detection unit 11 detects an object existing around the mobile body 10 based on information obtained from the optical sensor or other sensors. In the following description, the optical sensor is a camera, the object detection unit 11 obtains an image from the camera, and detects an object existing around the mobile body 10 based on the obtained image.

[0071] The object detection unit 11 detects an object based on the input sensor data, i.e., an image, etc. The object detection unit 11 is an information processing unit that controls the processing of the object detection model (learned model), and is composed of a circuit loop. The object detection model is a mathematical model (reasoner) of a neural network type having multiple layers, including a mathematical model for performing object detection processing.

[0072] Basically, the object detection model has multiple processing layers consisting of an input layer, an intermediate layer, and an output layer. The object data to be processed by the object detection process is input to the input layer as input data (input information). Then, the processing result data of the object detection process is output from the output layer as output data (output information). The input layer and the output layer are different processing layers, and the processing layer between the input layer and the output layer is called an intermediate layer. The object detection model can have multiple intermediate layers.

[0073] For example, an object detection model that has been learned using learning data is installed in the object detection unit 11. In the present embodiment, the object detection model is learned based on an image as input data so as to be able to detect two object categories, namely, "passenger car" and "person".

[0074] Furthermore, the object detection unit 11 generates log information (see log information 100 ) for transmission to the server device 30 based on the image of the detection object and the detection result. Figure 3 ). The log information includes an image showing the detection object and a detection result in the object detection unit 11, which will be described in detail later.

[0075] The communication unit 12 is a communication line (in other words, a communication module) for the mobile body 10 to communicate with the server device 30. The communication unit 12 functions as a log transmitter, and transmits the log information generated by the object detection unit 11 to the server device 30 via the network 20. In addition, the communication unit 12 has no particular limitation on the timing of sending the log information, and the log information can be sent sequentially or periodically.

[0076] Furthermore, the mobile object 10 may include a storage unit (storage device) for storing log information.

[0077] The network 20 is a wide area network (WAN) such as the Internet. The mobile object 10 and the server device 30 are connected to each other via the network 20 (specifically, via a relay device (not shown) provided at the end of the network 20) ​​so as to be able to communicate wirelessly.

[0078] The server device 30 is an information processing device (training data set generating device) that generates a training data set for relearning a learned model (object detection model) for object detection. The server device 30 is operated by, for example, a manufacturer or other operator that produces an object detection model installed in the object detection unit 11 of the mobile body 10.

[0079] The server device 30 includes a communication unit 31 , a determination detection unit 32 , a false detection determination unit 33 , a generation unit 34 , and a storage unit 35 .

[0080] The communication unit 31 is a communication line (in other words, a communication module) for the server device 30 to communicate with the mobile body 10. The communication unit 31 functions as a log collector and receives log information from the mobile body 10 via the network 20. The communication unit 31 outputs the received log information to the detection unit 32 for determination and the false detection determination unit 33. In addition, the communication unit 31 is an example of an acquisition unit.

[0081] The judgment detection unit 32 is a processing unit that performs object detection processing on the image included in the log information. The judgment detection unit 32 has a larger calculation scale than the object detection unit 11 and can perform more accurate object detection. In the present embodiment, the judgment detection unit 32 has an object detection model that has been learned in a manner capable of performing image segmentation (semantic segmentation), and uses the object detection model to perform image segmentation on the image. Performing image segmentation means performing a process of labeling the semantics represented by each of the multiple pixels in the image. This is equivalent to labeling which category of target each pixel is, that is, which category it belongs to.

[0082] In addition, the judgment detection unit 32 has an object detection model that includes object categories that can be detected by the object detection unit 11 (in this embodiment, "passenger car" and "person") and is learned in a manner that can detect more object categories than the object category, and the learned model can be used to perform object detection processing.

[0083] The determination detection unit 32 outputs the detection result to the false detection determination unit 33. In the present embodiment, the determination detection unit 32 outputs the segmentation result (see Figure 8 ) is output to the false detection judgment unit 33.

[0084] The false detection judgment unit 33 is a processing unit that judges whether the detection result of the object detection unit 11 is correct (i.e., whether the object detection of the object detection unit 11 is a false detection) based on the log information obtained from the object detection unit 11 and the detection result of the judgment detection unit 32. The false detection judgment unit 33 makes the above judgment based on, for example, whether there is a difference between the detection results of the object detection unit 11 and the detection results of the judgment detection unit 32. The false detection judgment unit 33 outputs the judgment result to the generation unit 34.

[0085] The generation unit 34 is a processing unit that generates training data for relearning the object detection model of the object detection unit 11 based on the judgment result of the false detection judgment unit 33. In the present embodiment, there is a feature in the method of generating training data of the generation unit 34. The generation unit 34 cuts out the object from the image showing the object that is judged by the false detection judgment unit 33 as an incorrect detection result of the object detection unit 11, which will be described in detail later. Then, the generation unit 34 performs a process of increasing (increasing) the effective training data by overlapping or replacing the cut image with other images showing objects of the same object category as the object (for example, "passenger car"). In addition, cutting is an example of extraction.

[0086] The training data may be data used as input data for re-learning the object detection model, or may be test data for verifying the detection capability of the re-learned object detection model.

[0087] The storage unit 35 stores the image group (see Figure 7 ), the image group includes a plurality of images for making the training data appear to increase. The image group may also be stored in advance. In addition, the storage unit 35 may store the images included in the log information obtained from the false detection judgment unit 33 as an image group. In this case, the image is stored corresponding to the detection result of the judgment detection unit 32 in the image.

[0088] In addition, each of the plurality of images included in the image group stored in the storage unit 35 may be combined with the image described below. Figure 3The various information included in the log information shown corresponds to each of the multiple images included in the image group. For example, each of the multiple images included in the image group may also correspond to information such as the time (or time period) when the image is taken, weather, etc. Furthermore, each of the multiple images included in the image group may also correspond to information such as the position or posture of the camera when the image is taken. Furthermore, each of the multiple images included in the image group may also correspond to information such as the luminance in the image (for example, the luminance of an object in the image) or the brightness (for example, the brightness of an object in the object). The brightness in the image may be, for example, information obtained by a sensor other than a camera.

[0089] When storing the images included in the log information as an image group, the log information may also be stored (see Figure 3 ) itself. In this case, the object detection result by the determination detection unit 32 may be applied to the detection information included in the log information. In other words, the object detection result by the object detection unit 11 may not be stored in the storage unit 35.

[0090] Furthermore, the storage unit 35 may store programs and the like executed by each processing unit included in the server device 30. The storage unit 35 is composed of, for example, a semiconductor memory.

[0091] As described above, the server device 30 includes, for example, a judgment detection unit 32 that has higher accuracy than the object detection unit 11 that is a reinforcement target (a target of re-learning) for images of erroneous detection scene data. Therefore, the server device 30 uses the segmentation result or detection result of the judgment detection unit 32 to clip the erroneously detected image area, and generates training data for countermeasures against erroneous detection by reflecting the clipped image 120 on other images showing objects of the same object category. In this way, the effect of the re-learning process for the object detection unit 11 for countermeasures against erroneous detection can be improved.

[0092] [1-2. Operation of the server device]

[0093] Next, refer to Figures 2 to 10 , the operation of the server device 30 and the generated images, etc. are explained. Figure 2 This is a flowchart showing the operation of the server device 30 according to the present embodiment.

[0094] like Figure 2As shown, first, the server device 30 obtains log information from the mobile body 10 (S10). Specifically, the communication unit 31 receives the log information from the mobile body 10. Then, the communication unit 31 outputs the received log information to the judgment detection unit 32 and the false detection judgment unit 33. In addition, among the various data included in the log information, only the image may be output to the judgment detection unit 32.

[0095] Here, refer to Figure 3 , the log information received by the communication unit 31 from the mobile body 10 is described, that is, the log information generated by the object detection unit 11 of the mobile body 10 is described. Figure 3 1 is a diagram showing an example of the log information 100 obtained by the server device 30 according to the present embodiment.

[0096] like Figure 3 As shown, the log information 100 includes, for example, scene images, camera device information, time information, camera position information, detection information, camera parameters, and the like.

[0097] The scene image (hereinafter also simply referred to as an image) is an image 101 captured by a camera provided to the mobile body 10, and is an image 101 for which object detection processing is to be performed. The image 101 may also correspond to an object region 101a, which is a region corresponding to the detection information. Figure 3 In the example of FIG. 1 , the region where the target object 110 exists is the object region 101 a .

[0098] The camera information is information related to the camera that took the image. For example, the camera information is information for identifying the camera that took the image 101. Figure 3 The example is the front camera.

[0099] The time information is information indicating the time when the image 101 was captured.

[0100] The imaging position information is information indicating the position and orientation at which the image 101 was captured.

[0101] The detection information is information indicating the detection result of the object detection unit 11 for the image 101. Specifically, the detection information includes information indicating the detection result and the object region 101a corresponding to the detection result. Figure 3 , an example is shown in which the detection result of the object detection unit 11 for the target object 110 being a "passenger car" is a "person". That is, Figure 3The log information 100 shown is log information when an erroneous detection is made by the object detection unit 11. In addition, although the object region 101a is shown as a rectangle, the shape of the object region 101a is only an example, and a shape other than a rectangle may be used.

[0102] The imaging parameters are information indicating the imaging conditions of the camera when imaging the image 101. The imaging conditions include gain, white balance, and the like, for example.

[0103] Refer again Figure 2 The determination detection unit 32 uses the image 101 included in the log information 100 obtained from the communication unit 31 as input data and performs object detection processing (S20). Figure 4 An example of the image 200 input to the determination detection unit 32 according to the present embodiment is shown. Figure 5 The detection result of the determination detection unit 32 according to the present embodiment is shown. The image 200 is the same image as the image 101 included in the log information 100. The image 200 input to the determination detection unit 32 does not include the object region 101a.

[0104] By Figure 4 The image 200 shown is input to the determination detection unit 32, so that Figure 5 The segmentation result 300 shown is output. Figure 5 It is shown in Figure 4 The target object 110 shown is determined to be a “passenger car.” In this way, since the determination detection unit 32 can detect an object more accurately than the object detection unit 11 of the moving body 10 , an object that was erroneously detected by the object detection unit 11 can be correctly detected.

[0105] The determination detection unit 32 outputs the detection result (here, the segmentation result 300 ) to the false detection determination unit 33 .

[0106] Refer again Figure 2 Next, the false detection judgment unit 33 judges whether there is a false detection in the detection of the object detection unit 11 based on the detection result obtained from the judgment detection unit 32 and the log information 100 (i.e., the detection result of the object detection unit 11 of the moving body 10) (S30). The false detection judgment unit 33 makes the above judgment, for example, by judging whether there is a difference between the detection results of the judgment detection unit 32 and the object detection unit 11. In the present embodiment, the detection result of the judgment detection unit 32 for the target object 110 is "passenger car" (refer to Figure 5 ), the detection result of the object detection unit 11 of the moving body 10 is "person" (refer to Figure 3 ).

[0107] In this way, when there is a difference between the detection result of the detection unit 32 for determination and the detection result of the object detection unit 11, the false detection determination unit 33 determines that there is a false detection in the detection of the object detection unit 11. When the false detection determination unit 33 determines that there is a false detection ("Yes" in S30), the log information 100 corresponding to the image is collected as false detection scene data (S40). Then, the false detection determination unit 33 outputs the collected false detection scene data to the generation unit 34. For example, the false detection determination unit 33 outputs the false detection scene data and the segmentation result 300 for the false detection scene data to the generation unit 34. In addition, the falsely detected image refers to an image in which the falsely detected target object 110 is reflected. In addition, the difference between the detection result of the detection unit 32 for determination and the detection result of the object detection unit 11 means, for example, that the detection unit 32 for determination detects the object, but the object detection unit 11 does not detect the object. That is, a case where the object detection unit 11 fails to detect an object that it should detect is also included in erroneous detection by the object detection unit 11 .

[0108] Then, when the false detection determination unit 33 determines that there is no false detection (No in S30 ), the process ends.

[0109] Next, the generating unit 34 performs annotation processing on the erroneous detection scene data obtained by the erroneous detection judging unit 33 (S50). The generating unit 34 performs annotation processing based on the detection result of the judging detecting unit 32 on the erroneous detection scene data. In the present embodiment, the generating unit 34 performs annotation processing based on the segmentation result 300 on the erroneous detection scene data. Specifically, the object category for the target object 110 is changed from "person" to "passenger car". The annotated erroneous detection scene data (i.e., log information with incorrect detection results) can also be used as training data.

[0110] Here, in suppressing false detection in the object detection unit 11, that is, in generating an object detection model capable of high-precision object detection, more training data is advantageous. Although it is possible to extract training data from a plurality of log information 100 from the mobile body 10, it takes more time and manpower to extract effective training data. Therefore, the generation unit 34 performs a process of increasing the training data for suppressing false detection (increase process). The generation unit 34 generates effective training data according to prescribed rules in order to suppress false detection in the object detection unit 11. The process of generating effective training data in the generation unit 34 will be described below.

[0111] The generating unit 34 generates a clipped image with the erroneously detected object region 101a clipped out based on the segmentation result 300 for the image 101 of the erroneously detected scene data (S60). Figure 6 A cropped image 120 of a target object 110 is shown. Figure 6 An example of the clipped image 120 according to the present embodiment is shown.

[0112] In addition, when there is an object that obscures a portion of the target object 110 in the image 101 (for example, an object that is located between the moving body 10 and the target object 110 and overlaps with the target object 110 when viewed from the moving body 10), the generating unit 34 can clip the object and the target object 110 as a whole. For example, if there is another object that obscures the target object 110 between the target object 110 to be clipped and the vehicle itself (moving body 10), the other object can be clipped at a display area threshold (for example, the size of the clipped image 120) that is set in advance based on the segmentation result, so that the other object can be clipped as a whole. For example, the target object 110 and the other object can be objects of the same object category (for example, a passenger car). By reflecting the clipped image clipped as a whole to other images, a natural image with less sense of incongruity can be generated as training data.

[0113] Thus, the image (an example of an image based on a portion of the first image) superimposed or replaced with another image may be, for example, an image (for example, a clipped image 120) obtained by processing the object 110 (an example of a portion of the first image) of the image 101. In addition, a predetermined image processing may be performed on the clipped image 120.

[0114] Refer again Figure 2 Next, the generator 34 selects an image that reflects the clipped image 120 from other image groups that have the same object category (e.g., passenger car) as the erroneously detected object (here, the target object 110) (S70). In addition, the reflection includes overlapping or replacing.

[0115] Figure 7 An example of the image group 400 stored in the storage unit 35 is shown. Figure 8 4 shows the detection result of the determination detection unit 32 for the image 401 included in the image group 400. Specifically, Figure 8 Segmentation result 500 for image 401 is shown.

[0116] like Figure 7 As shown, there are target objects 410 and 420 in the image 401. Figure 8As shown, the object category of the target object 410 is "passenger car", and the object category of the target object 420 is "sign". The target object 410 is the same object category as the target object 110 in the image 101. The generator 34 determines whether the object categories of the target objects 110 and 410 are the same based on the detection results of the determination detection unit 32 in each target object. Here, the object category of the target objects 110 and 410 is "passenger car".

[0117] Thus, the generation unit 34 selects the image 401 having the same object category as the erroneously detected object category from the pre-stored image group 400, based on the segmentation results 300 and 500 using the judgment detection unit 32. The image 401 having the same object category as the erroneously detected object category is an example of an image that can reflect the clipped image 120.

[0118] In step S70, the generation unit 34 prioritizes and selects, as images that can be reflected, images that can reflect the target object 110 at its original size from the image group 400. Alternatively, the generation unit 34 may select, as images that can be reflected, images that can be reflected by enlarging or reducing the target object 110 at a predetermined magnification.

[0119] Refer again Figure 2 Next, the generation unit 34 reflects the clipped image 120 on the selected image group (S80). For example, the generation unit 34 reflects the clipped image 120 on the image 401 selected from the image group 400. Figure 7 The segmentation result 500 indicates the area where the “passenger car” exists.

[0120] Fig.9A An example of an image group 600 included in the training data is shown.

[0121] like Fig.9A As shown, the generating unit 34, for example, overlays the clipped image 120 obtained by clipping the target object 110 shown in the clipped image 120 onto the Figure 7 The image 601 is generated by using the object 410 of the image 401 shown in the figure. The generation unit 34 superimposes the object 110 on the position where the "passenger car" in the image 401 exists. The image 601 is an image in which the object 110 is superimposed on the position where the object 110 actually exists (in this embodiment, the lane where the "passenger car" exists). According to this, since an image close to the real world can be generated, when a general neural network is fine-tuned on the neural network dedicated to the mobile body 10, it is possible to fine-tune the method of detecting the object with higher accuracy.

[0122] In addition, when the clipped image 120 is superimposed on the target object 410, the superimposed image (for example Fig.9A In the image 601 shown in FIG. 60, it is desired that the target object 410 cannot be seen. Therefore, the generating unit 34 can further Figure 7 The clipped image 120 resized according to the size of the area where the "passenger car" of the segmentation result 500 exists is superimposed on or replaced with the image 401. The resizing includes enlargement and reduction.

[0123] The adjustment range (an example of the adjustment ratio) for resizing the clipped image 120 may be preset. Figure 6 The size of the clipped image 120 shown can be enlarged or reduced within ±20%. In addition, the enlargement and reduction processing is performed on the basis of maintaining the aspect ratio of the clipped image 120. Furthermore, in the case of overlapping the clipped images 120, the enlargement rate or reduction rate can be determined in such a way that the target object 410 is not visible in the overlapped image 601, or the area where the target object 410 is visible becomes smaller (the area that exceeds becomes smaller).

[0124] Furthermore, in order to prevent the object overlapped on the image 401 (for example, the passenger car shown in the clipped image 120) from becoming a relatively large object in the image 401, the generating unit 34 may interpolate and overlap using the surrounding pixels of the object 110 of the original image (for example, the image 101).

[0125] And, if Fig. 9B As shown, the generating unit 34 generates Figure 7 The annotation information corresponding to the image group 400 shown in FIG. 4 is used to generate the annotation information 700 corresponding to the image group 600. Fig. 9B As shown, “passenger car” is annotated for the target object 110 .

[0126] Refer again Figure 2 , the generating unit 34 is based on Fig.9A The image group 600 and the annotation information 700 corresponding to the image group 600 are used to generate training data (S90).

[0127] Furthermore, when the number of generated training data is less than a preset number, the generating unit 34 may generate additional training data using images having annotation information of a similar category. Fig.10As shown, it can be considered that the category structure of the object category is predetermined, and "bus" or "truck" with the same superordinate category as "passenger car" is used. The generation unit 34 can select an image with the object category of "bus" or "truck" from the image group 400, and perform the processing of steps S80 and S90 on the selected image. In addition, Fig.10 An example of a category structure for object detection is shown.

[0128] Furthermore, when the number of images selected in step S70 exceeds a preset number, the generator 34 may use all the selected images to generate training data, or may extract a preset number of images from the selected images. For example, the generator 34 may extract the images using a subordinate category of the object category. Fig.10 In the category structure shown, it is conceivable that the subordinate categories of "passenger car" are "sedan" and "SUV". When the number of images that can be reflected in step S70 exceeds the preset number, the generation unit 34 can further preferentially select images showing objects of the same or similar category as the subordinate category (e.g., "sedan") in the target object 110 whose object category is "passenger car", and perform the processing of steps S80 and S90 on the selected images. For example, the generation unit 34 preferentially selects images showing objects that are considered to be the same as "sedan".

[0129] in addition, Fig.10 The category structure shown is stored in the storage unit 35. In addition, the determination detection unit 32 can detect, for example, Fig.10 The object categories included in the category structure shown are detected.

[0130] The above describes an example of a process in which the generation unit 34 overlaps or replaces the clipped image 120 with another image of the same object category as the target object 110, but the present invention is not limited to this example. The generation unit 34 may, for example, reflect a CG (Computer Graphics) image such as a 3D model similar to the clipped image 120 on the image selected in step S70. The generation unit 34 may generate a CG image based on the clipped image 120, and when the storage unit 35 stores a CG image of the target object, a CG image having a feature value similar to that of the clipped image 120 may be selected. The selected CG image may be an image having a feature value most similar to that of the clipped image 120, or may be a plurality of similar CG images having a feature value greater than a specified value.

[0131] In this way, the image (an example of an image based on a portion of the first image) overlapped or replaced with another image (the image selected in step S70) can be, for example, an image selected from pre-prepared images and similar to the target object 110 (an example of a portion of the first image). In this case, the size and tilt (for example, the rotation angle) of the CG image can be adjusted. The adjustment range of the size and tilt of the CG image is an example of an adjustment ratio. In addition, the adjustment ratio is a ratio at which the CG image as an image will not be distorted, and can be set to a ratio at which the object can be detected as such an object.

[0132] In addition, there is no particular limitation on the timing of executing the processing in the judgment detection unit 32, the false detection judgment unit 33, and the generation unit 34. For example, it can be executed every time the communication unit 31 obtains the log information 100 from the object detection unit 11, it can be executed periodically, and it can be executed when a predetermined number or a predetermined volume of log information 100 is accumulated. And, for example, in the case where the log information 100 includes the reliability of the detection result of the object detection unit 11, the judgment detection unit 32 can also perform object detection processing according to the reliability. The judgment detection unit 32 can, for example, prioritize images including objects whose detection results have a reliability below a predetermined value to perform object detection processing, or can only perform object detection processing on images including objects whose detection results have a reliability below a predetermined value. Based on this, the server device 30 can efficiently generate training data.

[0133] In addition, although the example described above is that the false detection judgment unit 33 obtains the detection result from the judgment detection unit 32, compares the obtained detection result (an example of object data) with the detection information of the log information 100 (an example of the object detection result), and extracts the target object 110 (an example of the part of the first image) based on the comparison result, the present invention is not limited to this example. For example, the server device 30 may also obtain the detection information of the log information 100. The false detection judgment unit 33 may obtain information corresponding to the difference between the detection results of the judgment detection unit 32 and the object detection unit 11 (for example, information related to the target object 110) from the administrator who manages the server device 30. The false detection judgment unit 33 may receive an indication of the target object 110 to be re-learned via a receiving unit (not shown) as a user interface for receiving input from the administrator, and output the indication to the generating unit 34. The indication of the target object 110 refers to information including the position and object category of the target object 110 in the image 101 included in the log information 100.

[0134] In addition, although the example described above is that the server device 30 receives the log information 100 including the detection information from the mobile body 10, the present invention is not limited to this example. The server device 30 may also obtain an image captured by the mobile body 10 from the mobile body 10. For example, the server device 30 may include an object detection unit (not shown) having the same learned model as the object detection unit 11 included in the mobile body 10. Therefore, the false detection judgment unit 33 can judge whether there is a difference (that is, whether the detection result of the object detection unit is correct) by comparing the detection result of the object detection unit with the detection result of the judgment detection unit 32.

[0135] As described above, in the information processing method in the server device 30 involved in the present embodiment, the computer performs the following operations, namely, obtaining the image 101 (an example of the first image) and the object category (an example of object data) related to the target object 110 (an example of the object reflected in the image 101), extracting the part of the image 101 (for example, the target object 110) corresponding to the difference between the object detection result (for example, the segmentation result 300) obtained by inputting the image 101 into the learned model and the object data, the learned model is a model that takes the image 101 as input and outputs the object detection result, obtaining the image 401 (an example of the second image) including the part corresponding to the same object data as the object data (for example, "passenger car") corresponding to the extracted part of the image 101, and generating training data for the learned model by reflecting the clipped image 120 (an example of the image based on the part of the first image) of the extracted image 101 on the target object 410 (an example of the part corresponding to the same object data) in the obtained image 401.

[0136] Thus, training data close to the real world can be generated. Thus, the object detection performance of the learned model for object detection can be reliably improved in relearning. In addition, an effective learning processing environment in the false detection correction of the neural network type detector can be realized at a low cost. For example, the effective training data can be automatically processed to appear to increase.

[0137] For example, when the design of a passenger car is significantly changed, the object detection unit 11 of the mobile body 10 may not be able to detect the passenger car. For example, in the case of a neural network type object detection unit 11, the neural network can be re-learned (additional learning). By generating training data for re-learning through the above method, even for a passenger car with a significantly changed design, the detection performance can be reliably improved.

[0138] (Variation of Implementation Example 1)

[0139] The following reference FIG. 11A to FIG. 11F , the server device and the like involved in the modification of this embodiment are described. In addition, the configuration of the server device involved in this modification is the same as that of the server device 30 involved in the first embodiment, and the description thereof is omitted here. Furthermore, for the same processing as that of the server device 30 in the first embodiment, the description thereof may be omitted or simplified.

[0140] The server device involved in this modification example is Figure 2 Among the images selected in step S70 shown in the figure, training data is further preferentially selected through the following processing. FIG. 11A to FIG. 11F The processing shown in step S70 may be performed, for example, when the number of images selected in step S70 exceeds the number of training data generated. FIG. 11A to FIG. 11F The processing in Figure 2 The steps shown are performed between steps S70 and S80.

[0141] Fig.11A This is a flowchart showing a first example of the operation of the generation unit 34 according to the present modification.

[0142] like Fig.11A As shown, the generation unit 34 can further determine whether the position or posture of the camera device when shooting the image selected in step S70 is the same or similar (S110). Specifically, the generation unit 34 determines whether the position or posture of the camera device (for example, a camera included in the mobile object 10) when shooting the image 101 included in the log information 100 obtained from the mobile object 10 is the same or similar to the position or posture of the camera device (for example, a camera included in a mobile object different from the mobile object 10) when shooting the image selected in step S70.

[0143] Then, when the position or posture is the same or similar ("Yes" in S110), the generation unit 34 determines the image as an image that can be reflected (an example of the second image) (S120). Next, the generation unit 34 determines whether the images determined as images that can be reflected have reached a prescribed number (S130). When the prescribed number has been reached (S130), the generation unit 34 proceeds to the process of step S90. And, when they are not the same or similar ("No" in step S110), and when the prescribed number has not been reached ("No" in S130), the generation unit 34 returns to step S110 and executes the process of step S110 for the next image.

[0144] In addition, the position of the camera is the position of the camera obtained by GPS positioning results, etc. For example, the position of the camera when the image 101 is captured can be obtained from the camera position information included in the log information 100. In addition, the posture of the camera is, for example, the direction in which the optical axis of the camera is facing, such as the posture of the camera when the image 101 is captured, which can be obtained from the camera equipment information and the camera position information included in the log information 100.

[0145] In addition, similarity here means that the position or posture included in the log information 100 is within a predetermined range when the image selected in step S70 is captured. The predetermined range may be stored in the storage unit 35 in advance.

[0146] Accordingly, when there is a similar image (different in date and time) whose camera position information is within a predetermined range among the images selected in step S70, the image is preferentially selected. Therefore, it is possible to generate training data that is effective in the learning process for passenger vehicles such as buses and taxis traveling on a determined route.

[0147] then, Fig. 11B This is a flowchart showing a second example of the operation of the generator 34 according to the present modification.

[0148] like Fig. 11B As shown, the generation unit 34 can further determine whether the time period or weather when the image 101 included in the log information 100 obtained from the mobile body 10 was taken is the same or similar to the time period or weather when the image selected in step S70 was taken (S210). Specifically, the generation unit 34 determines whether the time period or weather when the image 101 included in the log information 100 obtained from the mobile body 10 was taken is the same or similar to the time period or weather when the image selected in step S70 was taken.

[0149] Then, when the time period or weather is the same or similar ("Yes" in S210), the generation unit 34 performs the processing after step S120. If the time period or weather is not the same or similar ("No" in step S210), or the predetermined number is not reached ("No" in S130), the generation unit 34 returns to step S210 and performs the processing of step S210 for the next image.

[0150] In addition, the time period when the image is taken may be a time, etc. For example, the time period when the image 101 is taken can be obtained from the time information included in the log information 100. Furthermore, the weather when the image is taken includes, for example, one of the following: temperature, humidity, wind state (wind direction or wind force), cloud state (cloudy, rainy clouds, etc.), the presence or absence of precipitation or snowfall, etc. For example, the weather when the image 101 is taken can be obtained from the time information and camera position information included in the log information 100, and the information in the database storing weather information.

[0151] In addition, similarity here means that, with respect to the time or weather included in the log information 100, the time period or weather when the image selected in step S70 was shot is within a pre-set specified range. Weather similarity, for example, means that the type of weather is the same (an example within a specified range). Moreover, weather similarity, for example, may also mean that the temperature and humidity are within a specified range. Examples of weather types include "cloudless", "sunny", "cloudy", "foggy", "rainy", "thunder", etc. The specified range may be pre-stored in the storage unit 35.

[0152] Accordingly, among the images selected in step S70, if there is an image similar to the one in which the shooting time period or weather is within a predetermined range, the image is preferentially selected. Therefore, an image close to the environment when the target object 110 is misdetected can be preferentially selected, so that more effective training data can be generated in the case where the factor of misdetection of the target object 110 depends on the environment around the target object 110.

[0153] then, Fig. 11C This is a flowchart showing a third example of the operation of the generator 34 according to the present modification.

[0154] like Fig. 11C As shown, the generating unit 34 can further determine whether the brightness or the brightness in the camera direction when shooting is the same or similar to the image selected in step S70 (S310). Specifically, the generating unit 34 determines whether the brightness or the brightness in the camera direction when shooting of the image 101 included in the log information 100 obtained from the mobile object 10 is the same or similar to the brightness of the image selected in step S70 or the brightness in the camera direction when shooting the image.

[0155] Then, when the luminance or brightness is the same or similar ("Yes" in S310), the generator 34 performs the processing after step S120. If the luminance or brightness is not the same or similar ("No" in step S310), or if the predetermined number is not reached ("No" in S130), the generator 34 returns to step S310 and performs the processing of step S310 for the next image.

[0156] In addition, the brightness at the time of shooting is the brightness on the image. For example, the brightness at the time of shooting in the image 101 can be obtained from the image 101 included in the log information 100. The brightness is, for example, the brightness of the clipped image 120, but is not limited to this, and can also be the brightness of the entire image 101. In addition, the brightness can be calculated by statistical processing. The brightness can be, for example, the maximum value, minimum value, average value, or median value of the brightness in the clipped image 120. In addition, the brightness in the camera direction at the time of shooting is, for example, information on the brightness obtained from a sensor other than the camera device that shot the image 101 (for example, a sensor that detects the amount of light incident on the detection range). Information related to brightness can be included in the log information 100, for example.

[0157] In addition, similarity here means that the brightness of the image 101 or the brightness in the shooting direction when the image 101 is shot is within a predetermined range. The predetermined range can be stored in the storage unit 35 in advance.

[0158] Accordingly, when there is a similar image whose luminance or brightness is within a predetermined range among the images selected in step S70, the image is preferentially selected. Therefore, it is possible to preferentially select an image close to the environment when the target object 110 was erroneously detected, and thus, in a case where the element of the erroneously detected target object 110 depends on the environment around the target object 110, more effective training data can be generated.

[0159] then, Fig.11D This is a flowchart showing a fourth example of the operation of the generation unit 34 according to the present modification.

[0160] like Fig.11D As shown, the generation unit 34 can further determine whether the size of the portion corresponding to the object data (e.g., object category) (e.g., the target objects 110 and 410) is the same or similar (S410) for the image selected in step S70. Specifically, the generation unit 34 determines whether the size of the target object 110 existing in the image 101 of the log information 100 obtained from the mobile body 10 is the same or similar to the size of the target object 410 existing in the image 401 selected in step S70.

[0161] Then, when the size of the portion corresponding to the object data is the same or similar ("Yes" in S410), the generation unit 34 performs the processing after step S120. If the size of the portion corresponding to the object data is not the same or similar ("No" in step S410), or the predetermined number is not reached ("No" in S130), the generation unit 34 returns to step S410 and performs the processing of step S410 for the next image.

[0162] The size of the target object 110 is a size on the image. For example, the size of the target object 110 can be obtained from the image 101 included in the log information 100 . Also, the size of the target object 410 can be obtained from the image 401 .

[0163] In addition, the similarity here means that the size of the target object 410 existing in the image selected in step S70 is within a predetermined range set in advance with respect to the target object 110. The predetermined range may be stored in the storage unit 35 in advance.

[0164] Thus, if there is a similar image in which the size of the target object 410 is within a predetermined range among the images selected in step S70, the image is preferentially selected. Therefore, the generation unit 34 can generate training data including natural images with little sense of discomfort.

[0165] then, Fig.11E This is a flowchart showing a fifth example of the operation of the generation unit 34 according to the present modification.

[0166] like Fig.11E As shown, the generation unit 34 can further determine whether the area data around the portion corresponding to the object data is the same or similar to the image selected in step S70 (S510). Specifically, the generation unit 34 generates a segmentation result 300 (for example, referring to Figure 5 ), and the segmentation result 500 of the image selected in step S70 (for example, refer to Figure 8 ) are identical or similar to each other.

[0167] Then, when the area data around the portion corresponding to the object data is the same or similar ("Yes" in S510), the generation unit 34 performs the processing after step S120. If the area data around the portion corresponding to the object data is not the same or similar ("No" in step S510), or the area data has not reached the predetermined number ("No" in S130), the generation unit 34 returns to step S510 and performs the processing of step S510 for the next image.

[0168] In addition, the area data around the portion corresponding to the object data can be obtained as long as the judgment detection unit 32 can output the segmentation result. In addition, the area data around the portion corresponding to the object data (the target object 110) in the image 101 is, for example, "lane", "forest", etc. In addition, the area data around the portion corresponding to the object data (the target object 410) in the image 401 is, for example, "lane", "forest", etc.

[0169] In addition, similarity here means that, for the area data around the portion corresponding to the object data of the image 101, the area data around the portion corresponding to the object data of the image selected in step S70 is within a predetermined range. The predetermined range may be a range of positions or sizes of area data of the same category (e.g., "lane", "forest", etc.). The predetermined range may be stored in advance in the storage unit 35.

[0170] Accordingly, if there is an image in the image selected in step S70 whose surrounding area data corresponding to the portion of the object data of the image is similar within a predetermined range, the image is preferentially selected. Therefore, since an image close to the environment when the target object 110 was erroneously detected can be preferentially selected, more effective training data can be generated in the case where the element of the erroneous detection of the target object 110 depends on the environment around the target object 110.

[0171] then, Fig.11F This is a flowchart showing a sixth example of the operation of the generation unit 34 according to the present modification.

[0172] like Fig.11F As shown, the generation unit 34 can further determine whether the camera parameters of the camera device when shooting the image selected in step S70 are the same or similar (S610). Specifically, the generation unit 34 determines whether the camera parameters of the camera device when shooting the image 101 included in the log information 100 obtained from the mobile body 10 are the same or similar to the camera parameters of the camera device when shooting the image selected in step S70.

[0173] Therefore, when the imaging parameters of the imaging device at the time of shooting are the same or similar (“Yes” in S610), the generation unit 34 performs the processing after step S120. In addition, when the imaging parameters of the imaging device at the time of shooting are not the same or similar (“No” in step S610), or when the predetermined number is not reached (“No” in S130), the generation unit 34 returns to step S610 and performs the processing of step S610 for the next image.

[0174] In addition, the imaging parameters of the imaging device during shooting are setting conditions of the imaging device, for example, the imaging parameters of the imaging device during shooting of the image 101 can be obtained from the imaging parameters included in the log information 100. The imaging parameters may be, for example, gain or white balance settings, but are not limited thereto.

[0175] Here, similarity means that the imaging parameters of the imaging device when the image 101 is captured are within a predetermined range set in advance. The predetermined range may be stored in the storage unit 35 in advance.

[0176] Accordingly, if there is a similar image in which the imaging parameters of the imaging device at the time of shooting the image are within a specified range among the images selected in step S70, the image is preferentially selected. Therefore, it is possible to generate training data similar to the environment when the image 101 was shot based on information other than the subject (for example, the target object 110). In addition, in the case where the element of the erroneous detection of the target object 110 does not depend on the subject such as the target object 110, for example, in the case where the imaging parameters are the elements, more effective training data can be generated.

[0177] In addition, between steps S70 and S80, two or more of steps S110, S210, S310, S410, S510, and S610 may be executed. Furthermore, the predetermined range may be set according to the object type of the target object 110 in the image 101 included in the log information 100.

[0178] In addition, processing other than the above may be performed between steps S70 and S80. For example, if there is a scene image that can be superimposed with a small protruding area of ​​the target object 410 in the image 601, the scene image may be preferentially selected.

[0179] (Implementation Method 2)

[0180] The following reference Figures 12 to 15 , the server device and the like involved in this embodiment are described. In addition, the configuration of the server device involved in this embodiment is the same as that of the server device 30 involved in Embodiment 1, and the description thereof is omitted here. Furthermore, in the same processing as that of the server device 30 in Embodiment 1, the description thereof may be omitted or simplified.

[0181] In this embodiment, a case where the object detection unit 11 erroneously detects an object that is not a detection target will be described. In addition, an example in which the object detection unit 11 is a detection unit that detects only "people" will be described below.

[0182] Fig.12This is a flowchart showing the operation of the server device according to the present embodiment.

[0183] like Fig.12 As shown, first, the server device 30 obtains log information from the mobile object 10 (S1010). Specifically, the communication unit 31 receives the log information from the mobile object 10. Then, the communication unit 31 outputs the received log information to the detection unit 32 for determination and the false detection determination unit 33.

[0184] Here, the log information received from the mobile body 10, that is, the log information generated by the object detection unit 11 of the mobile body 10, will be referred to. Fig.13 to explain. Fig.13 1 is a diagram showing an example of the log information 800 obtained by the server device 30 according to the present embodiment.

[0185] like Fig.13 As shown, the log information 800 , like the log information 100 , includes, for example, a scene image, camera device information, time information, camera position information, detection information, camera parameters, and the like.

[0186] As shown in the scene image, there are target objects 810 and 820 in the image 801. And, as shown in the detection information, the object detection unit 11 determines that the target object 820 existing in the area encircled by the object area 801a is a "person". The object category of the target object 820 is a "sign". The object detection unit 11 mistakenly detects the "sign" which is an object that should not be detected as a "person".

[0187] Refer again Fig.12 , step S1020 is performed with Figure 2 The same process as step S20 shown in FIG. 1 is omitted here for detailed description. Fig.14 The image 900 shown in FIG. 1 is input to the determination detection unit 32, and the segmentation result (for example, referring to FIG. 1 ) is obtained. Figure 8 ) is output. Fig.14 is an example of the image 900 input to the determination detection unit 32 according to the present embodiment. In addition, the image 900 is regarded as the same image as the image 401. In other words, the segmentation result of the image 900 is the same as that of the image 401. Figure 8 The segmentation result 500 shown is the same. In the following description, the segmentation result of the image 900 is referred to as the segmentation result 500 .

[0188] The false detection judgment unit 33 judges whether there is a false detection in the detection of the object detection unit 11 based on the detection result (here, the segmentation result 500) obtained from the judgment detection unit 32 and the log information (i.e., the detection result of the object detection unit 11 of the mobile body 10) (S1030). For example, the false detection judgment unit 33 judges whether there is a difference between the detection result and the log information (specifically, the detection information as the object detection result). In this embodiment, for the target object 820, the detection result of the judgment detection unit 32 is a "flag" (refer to Figure 8 ), however, the detection result of the object detection unit 11 of the moving body 10 is "person" (refer to Fig.13 ).

[0189] In this way, when the detection result of the detection unit 32 for determination is different from the detection result of the object detection unit 11, the false detection determination unit 33 determines that there is a false detection in the detection of the object detection unit 11. When the false detection determination unit 33 determines that there is a false detection ("Yes" in S1030), the log information 800 is collected as false detection scene data (S1040). Then, the false detection determination unit 33 outputs the collected false detection scene data to the generation unit 34. And, when the false detection determination unit 33 determines that there is no false detection ("No" in S1030), the processing ends.

[0190] Next, the generating unit 34 performs annotation processing (S1050) on the false detection scene data obtained from the false detection judging unit 33. In the present embodiment, since the object detecting unit 11 detects only "people", there is no detection category (object category) such as "mark" in the object detecting unit 11. In this case, generally, the annotation of the area is not performed, but in the present embodiment, an annotation such as "background" is performed to indicate that it is not a "person".

[0191] The processing of steps S1060 to S1090 is Figure 2 The processing of S60 to S90 shown in FIG. 1 is the same as that of S60 to S90, and detailed description is omitted here. The generating unit 34 generates a clipped image (not shown) in which the target object 820 is clipped, and reflects the generated clipped image to the image (an example of the second image) selected in step S1070 (S1080). Based on this, the generating unit 34 generates a clipped image having a plurality of clips including the target object 820. Fig.15 In other words, the generator 34 generates a large amount of training data through the above-mentioned processing. Fig.15 An example of an image group 1000 included in the training data is shown.

[0192] In addition, compared with the object detector of the relearning processing object (for example, the object detection unit 11), by using the detection results of an object detector with a large number of detection categories and a large computational scale (for example, the judgment detection unit 32), a large amount of training data for countermeasures against false detections can be further effectively generated.

[0193] (Other Embodiments)

[0194] As mentioned above, the present disclosure has been described based on the embodiments and the like, but the present disclosure is not limited to the above-mentioned embodiments and the like.

[0195] For example, in the above-mentioned embodiments, the mobile body is described as an example of a vehicle (self-driving car), but it is not limited to this. As long as it is a mobile body with an object detection unit (object detection device), it can be applied to the present disclosure. For example, the mobile body can be a flying object (for example, a drone, etc.) or a ship, or it can be a robot that moves by itself based on the images taken of the surroundings (for example, an automatic vacuum cleaner, etc.). In other words, the server device (training data set generation device) involved in the present disclosure can be applied not only to the field of self-driving cars, but also to different product fields such as robots and drones.

[0196] Furthermore, the object detection unit included in the moving object in the above-mentioned embodiment may output the segmentation result as the detection result.

[0197] Furthermore, the order of the plurality of processes described in the above embodiment is an example, and the order of the plurality of processes may be changed, or the plurality of processes may be executed in parallel.

[0198] Furthermore, each component element described in each embodiment can be implemented by software, and can typically be implemented as an LSI of an integrated circuit. These can be a single chip, or part or all of them can be included in a single chip. Although it is referred to as LSI here, it can also be referred to as IC, system LSI, super LSI, and extra-large LSI depending on the degree of integration. Furthermore, the method of integrated circuitization is not limited to LSI, and can also be implemented by a dedicated circuit or a general-purpose processor. It is also possible to use a programmable FPGA (Field Programmable gate Array) after LSI manufacturing, and a reconfigurable processor that can reconfigure the connection or setting of the circuit unit inside the LSI. Moreover, with the advancement of semiconductor technology or the derivation of other technologies, if there is a technology for integrated circuitization that can replace LSI, of course, this technology can also be used to integrate the components.

[0199] Furthermore, the division of the functional blocks in the block diagram is an example, and multiple functional blocks can be implemented as one functional block, or one functional block can be divided into multiple blocks, or a part of the functions can be moved to other functional blocks. Furthermore, the functions of multiple functional blocks with similar functions can be processed in parallel or in time division by a single hardware or software.

[0200] Furthermore, the server device of the information processing system may be implemented as a single device or by multiple devices. For example, each processing unit of the server device may be implemented by two or more server devices. In the case where the information processing system is implemented by multiple server devices, the components of the information processing system may be arbitrarily distributed to multiple server devices. Furthermore, the communication method between the multiple server devices is not particularly limited.

[0201] Moreover, the technology disclosed in the present invention may be the above-mentioned program, or may be a non-temporary computer-readable recording medium recording the above-mentioned program. Furthermore, the above-mentioned program can be circulated through a transmission medium such as the Internet. For example, the above-mentioned program and the digital signal composed of the above-mentioned program can be transmitted via an electric communication line, a wireless or wired communication line, a network represented by the Internet, data broadcasting, etc. Furthermore, the above-mentioned program and the digital signal composed of the above-mentioned program can be transferred by being recorded in a recording medium, or transferred via a network, etc., so that it can be executed by other independent computer systems.

[0202] The present disclosure can be applied to an information processing method and the like for generating training data for re-learning a learned model.

[0203] Explanation of symbols

[0204] 1 Information Processing System

[0205] 10 Moving body

[0206] 11 Object Detection Unit

[0207] 12 Ministry of Communications

[0208] 20 Network

[0209] 30 Server Devices

[0210] 31 Communication Department (Acquisition Department)

[0211] 32. Judgment detection unit

[0212] 33 False detection judgment unit

[0213] 34 Generation Department

[0214] 35 Storage

[0215] 100, 800 Log information

[0216] 101, 200, 401, 601, 801, 900, 1001 images

[0217] 101a, 801a Object area

[0218] 110, 410, 420, 810, 820 Object

[0219] 120 Cut image

[0220] 300, 500 segmentation results

[0221] 400, 600, 1000 image groups

[0222] 700 Annotation information

Claims

1. An information processing method, wherein a computer performs the following operations: obtaining log information, the log information including a first image captured by a first camera device and detection information of a target object detected by an object detection unit and reflected in the first image, the detection information including a first object category of the target object, Obtaining an object detection result obtained by inputting the first image into a learned model, wherein the learned model receives an image as input to output the object detection result, wherein: The object detection result includes a detection result of the target object reflected in the first image detected by the learned model, and the detection result of the target object includes a second object category of the target object. determining whether the detection information of the target object is incorrect by determining whether there is a difference between the first object category of the target object and the second object category of the target object, extracting, by a generating unit generating training data, a portion of the first image including an object region corresponding to the target object for which the detection information is determined to be incorrect, The information processing method further includes executing, by the generating unit: obtaining a second image including a portion corresponding to an object included in the same object category as the second object category of the target object corresponding to the extracted portion of the first image, The training data of the learned model is generated by cutting out the extracted portion of the first image and overlapping or replacing the image based on the extracted portion of the first image with the portion of the second image obtained that corresponds to the object included in the same object category.

2. The information processing method according to claim 1, comparing the second object category with the first object category, Based on the comparison result, a portion of the first image is extracted.

3. The information processing method according to claim 1 or 2, The image based on the portion of the first image is an image obtained by processing the portion of the first image.

4. The information processing method according to claim 1 or 2, The image based on the portion of the first image is an image similar to the portion of the first image and selected from images prepared in advance.

5. The information processing method according to claim 1 or 2, When obtaining the second image, an image in which the position or posture of the second camera device when shooting is the same as or similar to the position or posture of the first camera device when shooting the first image is obtained as the second image.

6. The information processing method according to claim 1 or 2, When obtaining the second image, an image captured in a time period or weather that is the same as or similar to that of the first image is obtained as the second image.

7. The information processing method according to claim 1 or 2, When obtaining the second image, an image whose luminance or brightness in the camera direction when shooting is the same as or similar to the luminance of the first image or the brightness in the camera direction when shooting the first image is obtained as the second image.

8. The information processing method according to claim 1 or 2, When obtaining the second image, an image in which the size of a portion corresponding to the following object is the same as or similar to the size of a portion of the first image is obtained as the second image, and the object is an object included in the same object category as the second object category of the object object corresponding to the portion of the first image.

9. The information processing method according to claim 1 or 2, When obtaining the second image, an image in which area data around a portion corresponding to the following object is the same as or similar to area data around a portion of the first image is obtained as the second image, wherein the object is an object included in the same object category as the second object category corresponding to the portion of the first image.

10. The information processing method according to claim 1 or 2, The first image and the second image are camera images. When acquiring the second image, an image captured by the second imaging device with imaging parameters that are the same as or similar to imaging parameters of the first imaging device when capturing the first image is acquired as the second image.

11. An information processing system comprising: an acquisition unit that acquires log information, the log information including a first image captured by a first camera device and detection information of a target object detected by an object detection unit and reflected in the first image, the detection information including a first object category of the target object, A detection unit for determination is configured to obtain an object detection result by inputting the first image into a learned model, wherein the learned model receives an image as input and outputs the object detection result, wherein: The object detection result includes a detection result of the target object reflected in the first image detected by the learned model, and the detection result of the target object includes a second object category of the target object; a false detection judging unit, for judging whether the detection information of the target object is incorrect by judging whether there is a difference between the first object category of the target object and the second object category of the target object; as well as The generation part generates training data for the learned model in the following ways: extracting a portion of the first image including an object region corresponding to the target object for which the detection information is determined to be incorrect, obtaining a second image including a portion corresponding to an object included in the same object category as the second object category of the target object corresponding to the extracted portion of the first image, The extracted portion of the first image is cut out, and an image based on the extracted portion of the first image is overlapped or replaced with a portion of the second image obtained that corresponds to an object included in the same object category.

12. A computer program product, Used to cause a computer to execute the information processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium storing a program, When the program is executed by a computer, the computer executes the information processing method according to any one of claims 1 to 10.

Images