Information processing method and information processing system
By processing the image and position information of multiple imaging devices by computers, selecting the detection results of objects with high consistency in the common area, solving the problem of learning data stability caused by low sensor accuracy, and improving the reliability and efficiency of object detection.
Patent Information
- Application Number
- CN201910670434.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-08-02
- Filing Date
- 2019-07-24
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2039-12-05
AI Technical Summary
In the prior art, the low accuracy of the sensor detection results makes it difficult to obtain learning data stably, and high-precision sensors are required to increase costs and loads.
By computer processing the image and position information of multiple imaging devices at the same time, performing object detection processing, and calculating the consistency of object detection results in common areas, thereby selecting stable and efficient learning data.
It realizes stable acquisition of learning data under low-precision sensors, reduces unnecessary processing, and improves the reliability and efficiency of object detection results.
Smart Images

Figure CN110795985B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an information processing method and an information processing system for selecting an image to be used as learning data. Background Art
[0002] Patent Document 1 discloses a database construction system that automatically collects trained learning data for machine learning that uses detection results of a certain sensor among a plurality of types of sensors as training data and performs object recognition based on outputs of other sensors.
[0003] Prior art literature
[0004] Patent Literature
[0005] Patent Document 1: (Japanese) Patent Publication No. 2017-102838 Summary of the invention
[0006] However, in the technology of Patent Document 1, if the precision or accuracy (hereinafter referred to as precision) of the detection result of the sensor serving as the teacher data is low, the detection result of the sensor is not adopted as the learning data, so whether the learning data is obtained depends on the precision of the sensor. Therefore, in the technology of Patent Document 1, it is difficult to obtain the learning data stably.
[0007] Therefore, the present disclosure aims to provide an information processing method and an information processing system that are unlikely to affect the accuracy of object detection processing using a sensor and can stably obtain photographic images that serve as learning data.
[0008] The information processing method involved in the present disclosure uses a computer to obtain a plurality of images generated by photographing at the same time, and the positions and photographing directions of a plurality of photographing devices that generated each of the plurality of images, obtain a plurality of object detection results respectively obtained by a plurality of object detection processes respectively performed using the plurality of images, and perform a learning data selection process using the obtained plurality of images, the positions and the photographing directions, and the plurality of object detection results, the learning data selection process comprising: calculating a common area commonly reflected in each of the plurality of images based on the positions and the photographing directions of the plurality of photographing devices, determining a degree of consistency between the plurality of object detection results in the common area, and selecting an image to be learning data from the plurality of images based on the degree of consistency.
[0009] In addition, these overall or specific methods can also be implemented through systems, methods, integrated circuits, computer programs or computer-readable recording media such as CD-ROMs, or can be implemented through any combination of systems, methods, integrated circuits, computer programs and recording media.
[0010] The information processing method and the information processing system according to the present disclosure are unlikely to affect the accuracy of object detection processing using a sensor, and can stably obtain photographic images serving as learning data. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 This is a diagram showing the appearance of the information processing system according to the first embodiment.
[0012] Figure 2 This is a block diagram showing an example of the hardware configuration of the information processing device according to the embodiment.
[0013] Figure 3 This is a block diagram showing an example of the hardware configuration of the vehicle according to the first embodiment.
[0014] Figure 4 This is a block diagram showing an example of the functional configuration of the information processing system according to the first embodiment.
[0015] Figure 5 This is a diagram showing an example of the structure of the first data.
[0016] Figure 6 This is a diagram viewed from above, showing an example of the relationship between a specific imaging device and other imaging devices whose imaging ranges overlap.
[0017] Fig. 7A Indicates a plurality of images captured by a plurality of imaging devices at the same timing.
[0018] Figure 7B This is a diagram showing an example of an object position model obtained using a plurality of images, and the positions and imaging directions of a plurality of imaging devices at the same timing.
[0019] Figure 7C A plurality of projected images are shown in which the calculated common area is projected onto each of the plurality of images.
[0020] Fig. 8A This is a diagram for explaining a first example of a method for selecting images to be used as learning data.
[0021] Figure 8B This is a diagram for explaining a second example of the method of selecting images to be used as learning data.
[0022] Fig. 9This is a sequence diagram showing an example of the operation in the information processing system according to the embodiment.
[0023] Fig.10 This is a flowchart showing an example of details of the learning data selection process in the information processing device according to the embodiment.
[0024] Fig.11 This is a block diagram showing an example of the functional configuration of the information processing system according to Modification 1.
[0025] Fig.12 This is a sequence diagram showing an example of the operation in the information processing system according to Modification 1.
[0026] Description of labels:
[0027] 1.1A Information Processing System
[0028] 100, 100A Information processing device
[0029] 101, 201 processor
[0030] 102, 202 main memory
[0031] 103, 203 storage
[0032] 104, 204 communication interface
[0033] 110, 110a Acquisition section
[0034] 120 Execution Judgment Unit
[0035] 130 Executive Department
[0036] 131 Calculation Department
[0037] 132 consistency determination unit
[0038] 133 Selection Department
[0039] 200a, 200b, 200c, 201a, 201b, 201c Camera device
[0040] 205 Camera
[0041] 206 IMU
[0042] 207 GNSS
[0043] 210 Detection Department
[0044] 220 Storage
[0045] 230 Object Detection Unit
[0046] 240 Sending Department
[0047] 300 Communication Network
[0048] 310 Base Station
[0049] 400 objects
[0050] 401 Common Area
[0051] 410, 420, 430 camera range
[0052] 411, 421, 431 Object range. DETAILED DESCRIPTION
[0053] (Knowledge and insights that form the basis of the present invention)
[0054] In recent years, in the fields of autonomous driving, surveillance cameras, robots, etc., object detection using machine learning such as deep learning has been used for images captured by cameras. In such object detection, a large amount of teacher data for machine learning is required. Therefore, a large number of images captured by various cameras are collected, and people give correct answers to the collected images to generate teacher data.
[0055] However, it costs money for people to assign correct answers to images, so it is not preferable to simply generate teacher data based on all the large number of images obtained. In addition, even if teacher data with correct answers to all the large number of images is obtained without considering the cost, the processing load spent on machine learning becomes larger because machine learning needs to be performed on the large number of teacher data obtained, and the processing time also becomes longer. Therefore, in order to efficiently perform machine learning, it is necessary to select images that are effective for machine learning from a large number of images.
[0056] Here, a large number of images used in machine learning need to be composed of multiple images taken under various different conditions, that is, multiple images with diversity. In other words, using multiple images taken under different conditions is more effective in achieving efficient machine learning than using multiple images taken under similar conditions.
[0057] As described above, in the database construction system of Patent Document 1, the recognition data of the object with high reliability based on the output data of the first sensor is associated with the teacher data and the input data, which is the output data obtained from the second sensor, thereby collecting the teacher learning data.
[0058] However, when the reliability of the object recognition data based on the output data of the first sensor is not high, there is a concern that erroneous teacher data may be generated. For example, the reliability of the object recognition data depends on the quality of the first sensor, so the first sensor is required to have a quality above a certain level. That is to say, in the prior art, the detection results of a sensor with high accuracy are required, which costs more. In addition, in Patent Document 1, when the quality of the detection results of the first sensor is low, the detection results of the first sensor are not adopted as learning data, so whether the learning data is obtained depends on the quality of the first sensor. Therefore, it is difficult to obtain learning data stably.
[0059] In order to solve such a problem, an information processing method involved in one embodiment of the present invention uses a computer to obtain multiple images generated by photographing at the same time, and the positions and photographing directions of multiple camera devices that generated each of the multiple images, obtain multiple object detection results respectively obtained by multiple object detection processes respectively performed using the multiple images, and use the obtained multiple images, the positions and the photographing directions, and the multiple object detection results to perform learning data selection processing, the learning data selection processing including: calculating a common area commonly reflected in each of the multiple images based on the positions and the photographing directions of the multiple camera devices, determining the consistency between the multiple object detection results in the common area, and selecting an image from the multiple images to become learning data based on the consistency.
[0060] Therefore, when a plurality of imaging devices capture images of a common object, a plurality of object detection results of a plurality of object detection processes for the common object can be obtained. Thus, based on the degree of consistency between the plurality of object detection results for the common object, for example, an image that is the subject of the object detection process with inconsistent object detection results can be selected as an image to be used as learning data. Therefore, in the information processing method, it is difficult to affect the accuracy of the object detection process using the sensor, and a photographic image to be used as learning data can be obtained stably.
[0061] Furthermore, it is also possible to determine whether to execute the learning data selection processing based on a specific object detection result performed using an image generated by capturing images using a specific camera device among the multiple camera devices. If it is determined that the learning data selection processing is to be executed, the learning data selection processing is executed.
[0062] Therefore, the learning data selection process can be performed in the case where the object detection result has low accuracy and the desired learning data can be obtained more reliably. In addition, by not performing the learning data selection process in the case where the object is not desired to be learned, the execution of unnecessary processing can be reduced.
[0063] Furthermore, the other imaging devices other than the specific imaging device among the plurality of imaging devices may be imaging devices capable of capturing an image of a range overlapping with the imaging range of the specific imaging device.
[0064] Therefore, other imaging devices can be selected based on a specific imaging device, thereby obtaining an object detection result that is effective in determining the degree of coincidence.
[0065] Furthermore, the other imaging device may be an imaging device located within a predetermined range from a reference point based on the position of the specific imaging device.
[0066] Therefore, it is possible to select another imaging device that is highly likely to capture the same object as the object captured by the specific imaging device.
[0067] Furthermore, imaging may be performed by another imaging device other than the specific imaging device among the plurality of imaging devices.
[0068] Therefore, it is possible to make the imaging conditions consistent between a specific imaging device and other imaging devices. For example, it is possible to make the imaging times uniform so as to be regarded as the same time.
[0069] Furthermore, the specific object detection result may include a likelihood regarding the object detection result, and in determining whether to execute the learning data selection process, if the likelihood satisfies a prescribed condition, it is determined that the learning data selection process is to be executed.
[0070] Therefore, the learning data selection process can be performed in the situation where learning is desired, such as when the likelihood is low, and the desired learning data can be obtained more reliably. In addition, by not performing the learning data selection process when learning is not desired, unnecessary processing can be reduced.
[0071] In addition, in determining whether to execute the data selection processing for learning, statistical information of at least one of the shooting time of the multiple images, the position of the camera device that generated the images through the shooting, and the attributes or forms of the objects detected in the multiple object detection processes can also be obtained, and the obtained statistical information can be used to determine whether to execute the data selection processing for learning.
[0072] Therefore, the learning data selection process can be performed in the situation where learning is desired, such as when the shooting conditions or the shooting content are statistically suitable for learning, and the desired learning data can be obtained more reliably. In addition, by not performing the learning data selection process in the situation where learning is not desired, the execution of unnecessary processing can be reduced.
[0073] Furthermore, the statistical information may include scarcity information indicating the scarcity of the at least one information, and in determining whether to execute the data selection process for learning, if the scarcity indicated by the scarcity information satisfies a prescribed condition, it is determined that the data selection process for learning is to be executed.
[0074] Therefore, the learning data selection process can be executed when the scarcity satisfies the prescribed condition, and the desired learning data can be obtained more reliably. In addition, by not executing the learning data selection process when the scarcity does not satisfy the prescribed condition, the execution of unnecessary processes can be reduced.
[0075] Furthermore, in the learning data selection process, when there are inconsistent object detection results among the plurality of object detection results, an image to be the learning data may be selected from among the plurality of images.
[0076] Therefore, images including images of the object detection processing subject that may become object detection results with the possibility of false detection or missed detection can be selected as learning data. Thus, images captured under conditions different from those of images already selected as learning data can be selected as learning data.
[0077] In addition, when the performance between the multiple camera devices is the same as each other and the performance between the multiple object detection processes is the same as each other, images can also be selected from the multiple images to serve as learning data based on the number or proportion of consistent results among the multiple object detection results.
[0078] Therefore, for example, images in which the number of matching object detection results is less than a predetermined number or images in which the ratio of matching object detection results is lower than a predetermined ratio can be selected as learning data.
[0079] In addition, when the performances of the multiple camera devices are different from each other, or the performances of the multiple object detection processes are different from each other, the parameters of the learning data selection process can be determined based on the performances of the multiple camera devices or the performances of the multiple object detection processes, and the learning data selection process using the determined parameters can be executed.
[0080] Therefore, even when the performances of a plurality of imaging devices or a plurality of object detection processes are different from each other, by considering the performances, images including images of the object of the object detection process that may become an object detection result with a possibility of false detection or missed detection can be selected as learning data. Thus, images captured under conditions different from those of the images already selected as learning data can be selected as learning data.
[0081] In addition, these overall or specific methods can also be implemented by systems, devices, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or by any combination of systems, devices, integrated circuits, computer programs, or recording media.
[0082] Hereinafter, an information processing method and an information processing system according to one embodiment of the present invention will be described in detail with reference to the drawings.
[0083] In addition, the embodiments described below all represent a specific example of the present invention. The numerical values, shapes, materials, structural elements, configuration positions and connection methods of structural elements, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit the present invention. In addition, among the structural elements in the following embodiments, the structural elements that are not recorded in the independent claims representing the highest concept are described as arbitrary structural elements.
[0084] (Implementation Method)
[0085] Below, use Figures 1 to 10 , describing implementation mode one.
[0086] [1-1. Structure]
[0087] Figure 1 This is a diagram showing the appearance of the information processing system according to the first embodiment.
[0088] Specifically, in Figure 1 , an information processing device 100, imaging devices 200a, 200b, a communication network 300, and a base station 310 of a mobile communication system are shown. For example, the information processing system 1 includes the information processing device 100 and imaging devices 200a, 200b among these components. Figure 1 In FIG. 1 , the imaging devices 200a and 200b are shown as vehicles equipped with the camera 205, but they are not limited to vehicles and may be other imaging devices such as surveillance cameras. In addition, two imaging devices 200a and 200b are shown, but three or more imaging devices may be used.
[0089] The information processing device 100 is a device that acquires a plurality of images captured by the imaging devices 200a and 200b and selects learning data for machine learning from the acquired plurality of images. The information processing device 100 is, for example, a server.
[0090] Each of the imaging devices 200a and 200b is a vehicle, and is provided with a camera 205, and performs automatic driving or driving assistance using, for example, the result of object detection or object recognition obtained using image data obtained by the camera 205. In addition, each of the imaging devices 200a and 200b only needs to be provided with the camera 205, and does not necessarily have to have the function of performing automatic driving or driving assistance.
[0091] Communication network 300 may be a general network such as the Internet or a dedicated network. Base station 310 is a base station used in a mobile communication system such as a third generation mobile communication system (3G), a fourth generation mobile communication system (4G), or LTE (registered trademark).
[0092] Next, regarding a specific example of the hardware configuration of the information processing device 100, Figure 2 Provide explanation.
[0093] Figure 2 This is a block diagram showing an example of the hardware configuration of the information processing device according to the embodiment.
[0094] like Figure 2 As shown, the information processing device 100 includes a processor 101 , a main memory 102 , a storage 103 , and a communication interface (IF) 104 as a hardware configuration.
[0095] The processor 101 is a processor that executes a control program stored in the memory 103 or the like.
[0096] The main memory 102 is a volatile storage area used as a work area when the processor 101 executes a control program.
[0097] The memory 103 is a nonvolatile storage area that holds various data such as a control program, image data, and point group data.
[0098] The communication interface 104 is a communication interface for communicating with the camera devices 200a, 200b, etc. via a communication network. The communication interface 104 is, for example, a wired LAN interface. In addition, the communication interface 104 may also be a wireless LAN interface. In addition, the communication interface 104 is not limited to a LAN interface, and may be any communication interface as long as it can establish a communication connection with a communication network.
[0099] Next, regarding a specific example of the hardware configuration of the imaging devices 200a, 200b, etc., Figure 3 Provide explanation.
[0100] Figure 3 This is a block diagram showing an example of the hardware configuration of the vehicle according to the first embodiment.
[0101] like Figure 3 As shown, each of the imaging devices 200a, 200b, 200c, etc., includes a processor 201, a main memory 202, a storage 203, a communication interface (IF, Interface) 204, a camera 205, an IMU (Inertial Measurement Unit) 206, and a GNSS (Global Navigation Satellite System) 207 as a hardware structure. Each of the imaging devices 200a, 200b, etc. has the same structure as each other, so in the following, the imaging device 200a is described, and the description of the structure of the other imaging devices 200b, 200c, etc. is omitted.
[0102] The processor 201 is a processor that executes a control program stored in the memory 203 or the like. The processor 201 may include a processor used in an ECU for performing automatic driving or driving assistance of the imaging device 200a.
[0103] The main memory 202 is a volatile storage area used as a work area when the processor 201 executes a control program.
[0104] The memory 203 is a nonvolatile storage area that holds various data such as a control program, image data, and point group data.
[0105] The communication interface 204 is a communication interface for communicating with the information processing device 100 via the communication network 300. That is, the communication interface 204 can be a communication interface that can be connected to the communication network 300 for communication. Specifically, the communication interface 204 is a communication interface that is connected to the communication network 300 for communication through a communication connection with a base station 310 of a mobile communication system. The communication interface 204 is, for example, a wireless communication interface suitable for a communication standard used in a mobile communication system such as a third generation mobile communication system (3G), a fourth generation mobile communication system (4G), or LTE (registered trademark). In addition, the communication interface 204 may also be, for example, a wireless LAN (Local Area Network) interface suitable for IEEE802.11a, b, g, n, ac standards, or a communication interface that is connected to the communication network 300 for communication through a communication connection with a router (e.g., a mobile wireless LAN router) not shown in the figure.
[0106] The camera 205 is held in the imaging device 200 a , and is an optical sensor including an optical system such as a lens and an image sensor, and is an example of a first sensor.
[0107] The IMU 206 is a sensor device including an acceleration sensor and a gyro sensor. The acceleration sensor is a sensor that detects acceleration in each of the three different directions of the imaging device 200a. The gyro sensor is a sensor that detects angular velocity in rotation around three axes with the three different directions of the imaging device 200a as axes.
[0108] The GNSS 207 receives information indicating the position of the GNSS 207 from artificial satellites including GPS (Global Positioning System) satellites. That is, the GNSS 207 detects the current position of the imaging device 200a.
[0109] Next, regarding the functional structure of the information processing system 1, Figure 4 Provide explanation.
[0110] Figure 4 is a block diagram showing an example of a functional structure of an information processing system according to an embodiment of the present invention. Figure 4 In, omitted Figure 1 The communication network 300 and the base station 310 in it.
[0111] First, the functional structure of the imaging device 200a will be described.
[0112] The imaging device 200a includes, as a functional configuration, a detection unit 210, a storage unit 220, an object detection unit 230, and a transmission unit 240. Other imaging devices 200b, 200c, and the like also include the same functional configuration as the imaging device 200a.
[0113] The detection unit 210 detects an image captured by the camera 205 of the imaging device 200a. In addition, the detection unit 210 detects the position and the imaging direction of the imaging device 200a when the image is generated by the imaging of the camera 205. For the position and the imaging direction of the imaging device 200a when the image is generated by the imaging, the position and the imaging direction detected at the timing closest to the timing when the image is captured can be used, or the latest position and the latest imaging direction among the multiple positions and the multiple imaging directions detected at the timing when the image is captured can be used. For example, the position of the imaging device 200a is detected by the GNSS 207. For example, the imaging direction of the imaging device 200a can be detected using the detection value of the IMU 206, the detection value of the GNSS 207, or the combination of the detection values of the IMU 206 and the GNSS 207. In addition, when the imaging device 200a has a geomagnetic sensor, the imaging direction can also be detected using the detection value of the geomagnetic sensor.
[0114] The shooting direction is the shooting direction of the camera 205, and is a direction predetermined in the camera device 200a. When the camera 205 is arranged in a vehicle as the camera device 200a in a state where the shooting direction is fixed, the direction in which the camera 205 faces relative to the vehicle is the shooting direction. For example, when the camera 205 shoots the front of the vehicle, the direction in front of the vehicle becomes the shooting direction.
[0115] The detection unit 210 stores the plurality of images in the storage unit 220 in such a manner that the plurality of frames constituting the plurality of images or the plurality of still images are associated with the time when the frame or the still image is captured, i.e., the capturing timing. In addition, the detection unit 210 stores the plurality of positions in the storage unit 220 in association with the time when the position is detected, i.e., the detection timing. Similarly, the detection unit 210 stores the plurality of capturing directions in the storage unit 220 in association with the time when the capturing direction is detected, i.e., the detection timing. When the plurality of images are associated with the plurality of positions and the plurality of capturing directions in accordance with the detection timing, the positions and the capturing directions obtained at the timing closest to the timing when the image is obtained may be associated, or the plurality of images detected within a specified time interval may be associated with the plurality of positions and the plurality of capturing directions.
[0116] The detection unit 210 is implemented by, for example, a combination of the camera 205 , the IMU 206 , the GNSS 207 , and the like.
[0117] The storage unit 220 stores the plurality of images detected by the detection unit 210 together with the respective imaging timings of the plurality of images. In addition, the storage unit 220 stores the plurality of positions and imaging directions detected by the detection unit 210 together with the respective detection timings of the plurality of positions and imaging directions. In addition, the storage unit 220 stores the object detection results detected by the object detection unit 230 performing the object detection processing on the image in correspondence with the image. The storage unit 220 is realized by, for example, the storage 203.
[0118] The object detection unit 230 performs object detection processing using a predetermined object detection model on each of the plurality of images detected by the detection unit 210 to detect an object included in the image. The object detection unit 230 may perform object detection processing on, for example, the plurality of images stored in the storage unit 220, or may perform object detection processing on the plurality of images detected by the detection unit 210. The object detection unit 230 performs object detection processing on, for example, an image to recognize the attribute or form of an object included in the image. The attribute of an object may be, for example, a person, a vehicle, etc., and may also be the gender, age, etc. of a person, the type of the vehicle body, etc. The form of an object may be, for example, standing, sitting, sleeping, walking, running, etc. when the object is a person, and may be, for example, driving, slowing down, stopping, etc. when the object is a vehicle. When recognizing the attribute of an object, for example, the object detection unit 230 evaluates the likelihood that the detected object belongs to the attribute for each of the attributes of a plurality of types of objects, and recognizes the attribute of the detected object based on the evaluated likelihood. The object detection unit 230 may, for example, identify an attribute whose likelihood is the largest and whose likelihood is greater than the other likelihoods by a predetermined difference as an attribute of the object. Likelihood is the accuracy, reliability, score, etc. of object detection, and the higher the value, the higher the evaluation of the object detection result. In this way, the likelihood may also be included in the object detection result.
[0119] The object detection unit 230 associates the object detection result obtained by performing object detection processing on the image with the image and stores it in the storage unit 220. The object detection result and the image may be associated with each other by assigning an identifier for identifying the image to the object detection result or by assigning an identifier common to the image to the object detection result.
[0120] The object detection unit 230 is implemented by, for example, a combination of the processor 201 , the main memory 202 , and the storage 203 .
[0121] The sending unit 240 will Figure 5The first data including multiple images, multiple positions, multiple camera directions, and multiple object detection results stored in the storage unit 220 is sent to the information processing device 100. The sending unit 240 may also send the first data obtained by the detection by the detection unit 210 to the information processing device 100 regularly, such as every day, every week, etc. In addition, when the vehicle serving as the camera device 200a is parked at the storage location of the vehicle such as the driver's home, the sending unit 240 may send the first data obtained by the driving of the vehicle during the period before parking to the information processing device 100 via a wireless LAN router provided in the driver's home, for example. In addition, the sending unit 240 may also be such that, at this time, if there is first data that has been detected before but has not yet been sent to the information processing device 100, the first data is sent to the information processing device 100. In addition, Figure 5 This is a diagram showing an example of the structure of the first data.
[0122] The transmission unit 240 is realized by, for example, the processor 201 , the main memory 202 , the storage 203 , the communication interface 204 , and the like.
[0123] Next, the functional configuration of the information processing device 100 will be described.
[0124] The information processing device 100 includes an acquisition unit 110 , an execution determination unit 120 , and an execution unit 130 as a functional configuration.
[0125] The acquisition unit 110 acquires the first data from each of the plurality of imaging devices 200a, 200b, 200c, etc. Thus, the acquisition unit 110 acquires a plurality of images generated by imaging, the positions and imaging directions of the plurality of imaging devices 200a, 200b, 200c, etc. that generated each of the plurality of images by the imaging, and a plurality of object detection results obtained by the plurality of object detection processes in the plurality of imaging devices 200a, 200b, 200c, etc.
[0126] The execution determination unit 120 determines whether to execute the learning data selection process performed by the execution unit 130 based on the specific object detection result executed using the image generated by the imaging of a specific imaging device among the multiple imaging devices 200a, 200b, 200c, etc. The execution determination unit 120 may determine to execute the learning data selection process, for example, when the likelihood included in the object detection result of the specific imaging device satisfies a predetermined condition. In other words, the specific imaging device is an imaging device that outputs an object detection result that satisfies the predetermined condition among the multiple object detection results.
[0127] An object detection result that satisfies a prescribed condition is, for example, a likelihood having a difference with the maximum likelihood among the likelihoods of the evaluated multiple attributes that is less than a prescribed difference. In other words, the object detection result is a case where it is difficult to determine the attribute or form of the object as one of the multiple attributes or multiple forms, and there is no confidence in the accuracy of the detection result. Specifically, there is basically no difference between the likelihood of the first attribute and the likelihood of the second attribute, and it is impossible to determine whether the attribute of the object as the object detection result should be set as the first attribute or the second attribute. The same is true for the form of the object.
[0128] In addition, the object detection result that satisfies the prescribed conditions is, for example, a likelihood having the largest likelihood having a difference of a prescribed value or more from other likelihoods, or a likelihood having a likelihood exceeding a prescribed likelihood. In other words, the object detection result is a result that easily determines the attribute or form of the object as one attribute or one form among multiple attributes or multiple forms, and the accuracy of the detection result is confident.
[0129] In addition, the execution determination unit 120 may also obtain statistical information of image attribute information representing at least one of the attributes or forms of the object detected in the multiple object detection processes in the multiple camera devices 200a, 200b, 200c, etc., representing the shooting time of multiple images, the position of the camera that generates the image by shooting, and the image attribute information, and use the obtained statistical information to determine whether to execute the learning data selection process. In addition, the statistical information may also include sparseness information indicating the sparseness of the image attribute information. The execution determination unit 120 may also determine to execute the learning data selection process when the sparseness indicated by the sparseness information satisfies a prescribed condition. The sparseness indicated by the sparseness information satisfies the prescribed condition when the first data contains an image attribute having sparseness. Here, the image attribute having sparseness means that, in the shooting time, the position of the camera, the attribute or form of the object, or a combination of two or more thereof, the degree of deviation (or, deviation value) of the image attribute of the target image relative to the multiple image attributes of the multiple images selected as the learning data is greater than a prescribed threshold.
[0130] The execution unit 130 executes the learning data selection process using the plurality of first data acquired by the acquisition unit 110. The execution unit 130 may execute the learning data selection process when the execution determination unit 120 determines that the learning data selection process is to be executed.
[0131] Specifically, the execution unit 130 includes a calculation unit 131 , a match degree determination unit 132 , and a selection unit 133 .
[0132] In the learning data selection process, the calculation unit 131 calculates the common area commonly shown in each of the multiple images based on the respective positions and shooting directions of the multiple camera devices 200a, 200b, 200c, etc. at the specified timing among the multiple first data. The calculation unit 131 extracts multiple first data of multiple images captured at the specified timing based on the multiple first data acquired by the acquisition unit 110. The specified timing is a time that is considered to be the same. In other words, the specified timing may not necessarily be the same time, and may include all timings included in a specified time width based on a certain timing. The specified time width may also be, for example, a time based on the shooting interval of each camera device of the multiple camera devices 200a, 200b, 200c, etc. for shooting images, for example, a time of 1 to 10 times the shooting interval.
[0133] In addition, in the present embodiment, the plurality of camera devices 200a, 200b, 200c, etc. are vehicles and therefore move. Therefore, the plurality of images captured by the plurality of camera devices 200a, 200b, 200c, etc. often do not have a common area. Therefore, if the calculation unit 131 performs a process of calculating a common area on all of the plurality of first data acquired by the acquisition unit 110, the efficiency is low. Therefore, before the process of calculating the common area, the process of extracting a plurality of first data including a plurality of images having a common area may be performed.
[0134] The calculation unit 131 analyzes the positions and shooting directions of the camera devices of the plurality of first data, for example, to extract the plurality of first data shot by the plurality of camera devices 200a, 200b, 200c, etc., which can shoot the mutually overlapping ranges, and establishes the correspondence between the extracted plurality of first data to generate the second data. Therefore, the calculation unit 131 extracts the first data obtained by the following camera device, which is a camera device other than the specific camera device that shoots the image that becomes the object of the learning data selection process in the judgment in the execution judgment unit 120, and is a camera device that can shoot the range that overlaps with the shooting range of the specific camera device. Therefore, even if the acquisition unit 110 acquires the first data from a plurality of camera devices whose shooting ranges do not overlap, the calculation unit 131 can easily calculate the common area.
[0135] In addition, when the calculation unit 131 analyzes the positions and imaging directions of the plurality of imaging devices of the first data and selects a plurality of imaging devices whose imaging ranges overlap, the calculation unit 131 selects an imaging device that is located within a predetermined range from a reference point based on the position of a specific imaging device. Figure 6As shown, the imaging devices 200b and 200c located within a radius R1 from the reference point B1 at a distance D1 in front of the specific imaging device 200a are selected as other imaging devices 200b and 200c that overlap with the imaging range of the specific imaging device 200a. Figure 6 This is a diagram viewed from above, and is a diagram showing an example of the relationship between a specific imaging device and other imaging devices whose imaging ranges overlap.
[0136] Here, regarding a specific example of a method for calculating a common area, Figure 7A to Figure 7C Provide explanation.
[0137] Figure 7A to Figure 7C This is a diagram for explaining a specific example of a method for calculating a common area.
[0138] Fig. 7A The plurality of images P10, P20, and P30 captured at the same timing by the imaging devices 200a, 200b, and 200c are shown. The plurality of images P10, P20, and P30 are images including, for example, a common object 400.
[0139] Figure 7B This is a diagram showing an example of an object position model obtained using a plurality of images P10 , P20 , and P30 , and the positions and imaging directions of the imaging devices 200 a , 200 b , and 200 c at the same timing described above. Figure 7B A model of the object's position as viewed from above is shown.
[0140] Specifically, the calculation unit 131 uses the image P10 and the position and the imaging direction of the imaging device 200a to calculate the imaging range 410 and the object range 411. Specifically, the imaging range 410 indicates the imaging range estimated based on the position and the imaging direction of the imaging device 200a that captured the image P10. In addition, the object range 411 indicates the range where the object 400 exists obtained based on the area of the object 400 in the image P10 and the imaging range 410.
[0141] Similarly, the calculation unit 131 calculates the imaging range 420 and the object range 421 based on the image P20 and the position and imaging direction of the imaging device 200b. The imaging range 420 indicates the imaging range estimated based on the position and imaging direction of the imaging device 200b that captures the image P20. In addition, the object range 421 indicates the range where the object 400 exists obtained based on the area of the object 400 in the image P20 and the imaging range 420.
[0142] Similarly, the calculation unit 131 calculates the imaging range 430 and the object range 431 based on the image P30 and the position and imaging direction of the imaging device 200c. The imaging range 430 indicates the imaging range estimated based on the position and imaging direction of the imaging device 200c that captures the image P30. In addition, the object range 431 indicates the range where the object 400 exists obtained based on the area of the object 400 in the image P30 and the imaging range 430.
[0143] The calculation unit 131 determines that the object 400 is an object commonly captured in the plurality of images P10, P20, and P30, and calculates the object ranges 411, 421, and 431. The calculation unit 131 may not calculate the object ranges 411, 421, and 431.
[0144] Next, the calculation unit 131 uses the calculated imaging ranges 410, 420, and 430 to calculate the common area 401 thereof. Figure 7C As shown, projection images P11, P21, and P31 are generated by projecting the calculated common area 401 onto the images P10, P20, and P30, respectively.
[0145] The consistency determination unit 132 determines the consistency between the plurality of object detection results in the common region. The consistency determination unit 132 may, for example, compare the plurality of object detection results to determine the consistency between the plurality of object detection results. In addition, the object detection result in the common region is, for example, an object detected in a region on the image overlapping the common region.
[0146] The selection unit 133 selects an image to be used as learning data from among the plurality of images based on the degree of consistency determined by the degree of consistency determination unit 132. When there is an inconsistency among the plurality of object detection results included in the second data, the selection unit 133 selects an image to be used as learning data from among the plurality of images included in the second data.
[0147] For example, when the imaging performances of the cameras 205 of multiple imaging devices 200a, 200b, 200c, etc. are the same, and the processing performances of the object detection processing of the multiple imaging devices 200a, 200b, 200c, etc. are the same, the selection unit 133 selects an image to be learning data from the multiple images based on the number or proportion of consistent results among the multiple object detection results included in the second data.
[0148] use Fig. 8A and Figure 8B , an example of selecting an image to be used as learning data is described.
[0149] Fig. 8A and Figure 8BThis is a diagram for explaining an example of a method of selecting images to be used as learning data.
[0150] The consistency determination unit 132 Fig. 8A As shown in the first example, when the object detection result in the common area 401 of the projection image P11 is a vehicle, and the object detection results in the common area 401 of the projection images P21 and P31 are no vehicles, it is also possible to determine by majority decision that the object detection result in the common area 401 is no vehicle, and determine that the possibility of the object detection result in the projection image P11 including a false detection is high. Figure 8B As shown in the second example, when the object detection result in the common area 401 of the projection images P11 and P31 is a vehicle, and the object detection result in the common area 401 of the projection image P21 is no vehicle, it is also possible to determine by majority decision that the object detection result in the common area 401 is a correct answer, and determine that the object detection result in the projection image P21 has a high possibility of including an erroneous detection. In addition, the selection unit 133 selects the image determined to have a high possibility of including an erroneous detection in the object detection result as an image to be used for learning data based on the determination result in the consistency determination unit 132. In addition, when the multiple object detection results in the common area 401 contain different results, the selection unit 133 may select all multiple images of the objects that are the objects of the detection of the multiple object detection results as images to be used for learning data.
[0151] Furthermore, for example, when the imaging performances of the cameras 205 of the plurality of imaging devices 200a, 200b, 200c, etc. are different from each other, or the processing performances of the object detection processing of the plurality of imaging devices 200a, 200b, 200c, etc. are different from each other, the selection unit 133 determines the parameters of the selection processing for selecting an image to be learning data from among the plurality of images according to the imaging performances of the plurality of imaging devices 200a, 200b, 200c, etc., or the processing performances of the object detection processing of the plurality of imaging devices 200a, 200b, 200c, etc., and performs the selection processing under the determined parameters. For example, the selection unit 133 may select an image corresponding to an object detection result different from the object detection result with good accuracy as the object detection result with high weighting is considered to be a more accurate object detection result according to the imaging performances of the plurality of imaging devices 200a, 200b, 200c, etc., or the processing performances of the object detection processing. That is, the selection unit 133 selects images corresponding to object detection results different from the following object detection results as images set as learning data. The above-mentioned object detection results are object detection results in which the higher the performance, the larger the coefficient is multiplied by the object detection result to obtain an evaluation value above a specified threshold.
[0152] [1-2. Action]
[0153] Next, the operation of the information processing system 1 according to the embodiment will be described.
[0154] Fig. 9 This is a sequence diagram showing an example of the operation in the information processing system according to the embodiment.
[0155] First, in each camera device such as camera 200a, 200b, 200c, the detection unit 210 detects an image, and the position and direction of the camera device at the timing of photographing the image (S11), and stores the photographed timing, the detected image, and the position and direction of the camera device in the storage unit 220.
[0156] Next, the object detection unit 230 of the camera device detects the object by performing object detection processing on the image detected by the detection unit 210, thereby obtaining an object detection result (S12). The obtained object detection result is stored in the storage unit 220 in a state corresponding to the image that is the object of the object detection processing. Thus, in the camera device, first data is generated in which the timing of imaging, the detected image, the position and imaging direction of the camera device, and the object detection result are associated with each other.
[0157] Then, the transmitting unit 240 of the camera device transmits the generated first data to the information processing device 100 ( S13 ).
[0158] In the information processing device 100 , the acquisition unit 110 acquires the first data transmitted from each of the imaging devices such as the imaging devices 200 a , 200 b , and 200 c ( S21 ).
[0159] Next, the execution determination unit 120 of the information processing device 100 determines whether to execute the learning data selection processing (S22) performed by the execution unit 130 based on the specific object detection result executed using the image generated by the imaging of a specific imaging device among multiple imaging devices 200a, 200b, 200c, etc.
[0160] If the execution determination unit 120 determines that the learning data selection process is to be executed (Yes in S22 ), the execution unit 130 of the information processing device 100 executes the learning data selection process ( S23 ). The details of the learning data selection process will be described later.
[0161] On the other hand, when the execution determination unit 120 determines that the learning data selection process is not to be executed (No in S22 ), the execution unit 130 of the information processing device 100 does not execute the learning data selection process.
[0162] In addition, whether the learning data selection processing is executed or not, in the information processing system 1, the processing of the camera devices 200a, 200b, 200c, etc. in steps S11 to S13 and the processing of the information processing device 100 in steps S21 to S23 are repeatedly executed.
[0163] In addition, the details of the processing performed by each processing unit of steps S11 to S13 and S21 to S23 are described in detail using Figures 4 to 8B The functional configurations of the imaging devices 200a, 200b, 200c, etc. and the functional configuration of the information processing device 100 have been described in detail and are therefore omitted.
[0164] Fig.10 This is a flowchart showing an example of details of the learning data selection process in the information processing device according to the embodiment.
[0165] When the learning data selection process of step S23 is started, the calculation unit 131 of the execution unit 130 extracts a plurality of first data that are associated with an image generated by imaging by a specific imaging device at the same time and have a common imaging range from the plurality of first data acquired by the acquisition unit 110. Thus, the calculation unit 131 extracts the plurality of captured images (S31).
[0166] Next, the calculation unit 131 calculates a common area commonly shown in each of the plurality of images based on the positions and imaging directions of the plurality of imaging devices 200a, 200b, 200c, etc., using the extracted plurality of first data (S32).
[0167] Then, the consistency determination unit 132 determines the consistency between the plurality of object detection results in the common region ( S33 ).
[0168] Thereafter, the selection unit 133 selects an image to be learning data from among the plurality of images based on the degree of match determined by the degree of match determination unit 132 ( S34 ), and the learning data selection process ends.
[0169] In addition, the details of the processing performed by each processing unit in steps S31 to S34 are described in detail in the Figures 4 to 8B The functional configurations of the imaging devices 200a, 200b, 200c, etc. and the functional configuration of the information processing device 100 have been described in detail and are therefore omitted.
[0170] [1-3. Effects, etc.]
[0171] According to the information processing method involved in this embodiment, in the data selection process for learning, the common area commonly reflected in each of the multiple images is calculated based on the positions and camera directions of the multiple cameras, the consistency between the multiple object detection results in the common area is determined, and the image to be the learning data is selected from the multiple images based on the determined consistency.
[0172] Therefore, when a plurality of camera devices capture images of a common object, a plurality of object detection results based on a plurality of object detection processes for the common object can be obtained. Thus, based on the degree of consistency between the plurality of object detection results for the common object, for example, an image that is the subject of the object detection process with inconsistent object detection results can be selected as an image to be used as learning data. Therefore, in the information processing method, it is difficult to affect the accuracy of the object detection process using the sensor, and a photographic image to be used as learning data can be obtained stably.
[0173] Furthermore, according to the information processing method involved in the present embodiment, it is determined whether to perform the learning data selection process based on the specific object detection result performed using the image generated by the camera of a specific camera among the multiple camera devices, and the learning data selection process is performed when it is determined to perform the learning data selection process. Therefore, the learning data selection process can be performed in the situation where the object detection result is low in accuracy, and the desired learning data can be obtained more reliably. In addition, by not performing the learning data selection process in the situation where it is not desired to learn, the execution of unnecessary processing can be reduced.
[0174] In addition, according to the information processing method involved in this embodiment, the other camera devices other than the specific camera device among the multiple camera devices are camera devices that can capture the range that overlaps with the camera range of the specific camera device. Therefore, the other camera devices can be selected based on the specific camera device. Thus, an object detection result that is effective for determining the degree of consistency can be obtained.
[0175] In addition, according to the information processing method involved in this embodiment, the other camera is a camera located within a predetermined range from a reference point based on the position of the specific camera. Therefore, it is possible to select another camera that is likely to capture the same object as the object captured by the specific camera.
[0176] In addition, according to the information processing method involved in this embodiment, a specific object detection result includes the likelihood of the object detection result, and in the determination of whether to perform the learning data selection process, when the likelihood satisfies the prescribed conditions, it is determined that the learning data selection process is to be performed. Therefore, the learning data selection process can be performed in a situation where learning is desired, such as when the likelihood is low, and the desired learning data can be obtained more reliably. In addition, by not performing the learning data selection process in a situation where learning is not desired, the execution of unnecessary processing can be reduced.
[0177] Furthermore, according to the information processing method involved in the present embodiment, in determining whether to perform the learning data selection process, statistical information of at least one of the shooting time of multiple images, the position of the camera device that generated the image by shooting, and the attribute or form of the object detected in the multiple object detection processes is obtained, and the obtained statistical information is used to determine whether to perform the learning data selection process. Therefore, the learning data selection process can be performed in the situation where learning is desired, such as when the shooting conditions or the shooting content are statistically suitable for learning, and the desired learning data can be obtained more reliably. In addition, by not performing the learning data selection process in the situation where learning is not desired, the execution of unnecessary processing can be reduced.
[0178] Furthermore, according to the information processing method involved in the present embodiment, the statistical information includes rarity information indicating the rarity of at least one information, and in the determination of whether to perform the learning data selection process, when the rarity indicated by the rarity information satisfies a prescribed condition, it is determined that the learning data selection process is to be performed. Therefore, the learning data selection process can be performed when the rarity satisfies the prescribed condition, and the execution of unnecessary processing can be reduced.
[0179] Furthermore, according to the information processing method involved in the present embodiment, in selecting an image to be learning data, when there are inconsistent object detection results among a plurality of object detection results, an image to be learning data is selected from a plurality of images. Therefore, it is possible to select as learning data an image including an image that is the subject of object detection processing that is an object detection result with the possibility of misdetection or missed detection. Thus, it is possible to select as learning data an image that was captured under conditions different from those of the image that has been selected as learning data.
[0180] Furthermore, according to the information processing method involved in this embodiment, when the performances of the plurality of camera devices are the same and the performances of the plurality of object detection processes are the same, an image is selected from the plurality of images as learning data based on the number or ratio of the consistent object detection results. Therefore, for example, an image in which the number of consistent object detection results is less than a prescribed number or the ratio of consistent object detection results is lower than a prescribed ratio can be selected as learning data.
[0181] Furthermore, according to the information processing method involved in the present embodiment, when the performances of a plurality of camera devices are different from each other, or the performances of a plurality of object detection processes are different from each other, the parameters of the selection process for selecting an image to be learning data from among a plurality of images are determined according to the performances of the plurality of camera devices or the performances of the plurality of object detection processes, and the selection process under the determined parameters is performed. Therefore, even when the performances of a plurality of camera devices or the performances of a plurality of object detection processes are different from each other, since each performance is taken into consideration, it is possible to select as learning data an image including an image of an object detection process that is a result of object detection with the possibility of misdetection or missed detection. Thus, an image captured under a condition different from an image that has been selected as learning data can be selected as learning data.
[0182] [1-4. Modifications]
[0183] [1-4-1. Modification 1]
[0184] In the information processing system 1 according to the above-described embodiment, each of the imaging devices 200a, 200b, 200c, etc. has a structure of the object detection unit 230 that performs object detection processing on the image detected by the detection unit 210, but the present invention is not limited to this. For example, the above structure may also be used as follows. Fig.11 As shown, the imaging devices 201a, 201b, 201c, etc. are configured without the object detection unit 230. In the information processing system 1A in this case, the acquisition unit 110a of the information processing device 100A performs object detection processing on the image included in each of the plurality of acquired first data. Fig.11 This is a block diagram showing an example of the functional configuration of the information processing system according to Modification 1.
[0185] Fig.12 This is a sequence diagram showing an example of the operation in the information processing system according to Modification 1.
[0186] First, in each camera device such as camera devices 201a, 201b, 201c, etc., the detection unit 210 detects an image, and the position and direction of the camera device at the timing of photographing the image (S11), and stores the photographed timing, the detected image, and the position and direction of the camera device in the storage unit 220.
[0187] Next, the transmission unit 240 of the imaging device 200a transmits first data including the timing of imaging, the detected image, and the position and imaging direction of the imaging device to the information processing device 100A (S13a).
[0188] In the information processing device 100A, the acquisition unit 110 acquires the first data transmitted from each of the imaging devices such as the imaging devices 201a, 201b, and 201c (S21a).
[0189] Next, the acquisition unit 110a of the information processing device 100A detects an object by executing an object detection process on an image included in each of the plurality of acquired first data, thereby obtaining an object detection result (S21b).
[0190] The subsequent processing of steps S22 and S23 is the same as the processing in the embodiment, and therefore the description thereof is omitted.
[0191] [1-4-2. Modification 2]
[0192] In the information processing system 1 involved in the above-mentioned embodiment, when a vehicle as one of the plurality of camera devices 200a, 200b, 200c, etc. is in motion (for example, the engine is being driven and the power is turned on), it is set that the camera 205 always takes pictures at a predetermined period, but it is not limited thereto. For example, in a specific camera device, the same determination as that of the execution determination unit 120 may be performed, so that when an image for which the learning data selection process is to be performed is obtained, a camera request is sent to other camera devices located around the specific camera device, so that the other camera devices perform the camera. The camera request from the specific camera device may be sent to the other camera devices via the information processing device 100 or directly to the other camera devices.
[0193] As a result, other imaging devices only need to perform imaging when receiving an imaging request, and thus the frequency of imaging processing can be reduced.
[0194] [1-4-3. Modification 3]
[0195] In the information processing system 1 according to the above-mentioned embodiment, all the first data obtained in the plurality of imaging devices 200a, 200b, 200c, etc. are transmitted to the information processing device 100, but the present invention is not limited thereto. For example, in a specific imaging device, the same determination as that of the execution determination unit 120 is performed, so that at the timing when an image for which the learning data selection process is to be performed is obtained, an imaging request is transmitted to other imaging devices located around the specific imaging device, so that the other imaging devices can transmit only the first data at the timing to the information processing device 100. In this way, the communication load from the imaging device to the information processing device 100 can be reduced, and the information processing device 100 can suppress the storage capacity of the storage that stores the first data in advance from being tight.
[0196] In addition, in each of the above-mentioned embodiments, each structural element may be formed by dedicated hardware or implemented by executing a software program suitable for each structural element. Each structural element may also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory. Here, the software for implementing the information processing method and information processing device of each of the above-mentioned embodiments is the following program.
[0197] That is, the program causes a computer to execute the following information processing method, which includes: obtaining a plurality of images generated by photographing at a time that is considered to be the same, and the positions and photographing directions of a plurality of photographing devices that generate each of the plurality of images, obtaining a plurality of object detection results respectively obtained by a plurality of object detection processes respectively performed using the plurality of images, performing a learning data selection process using the obtained plurality of images, the positions and the photographing directions, and the plurality of object detection results, the learning data selection process calculating a common area commonly reflected in each of the plurality of images based on the positions and the photographing directions of the plurality of photographing devices, determining a degree of consistency between the plurality of object detection results in the common area, and selecting an image from the plurality of images to become learning data based on the degree of consistency.
[0198] The above is an explanation of the information processing method and information processing system involved in one or more embodiments of the present invention based on the implementation mode, but the present invention is not limited to this implementation mode. As long as it does not deviate from the purpose of the present invention, the implementation mode obtained by applying various modifications thought of by those skilled in the art to this implementation mode and the mode constructed by combining the structural elements in different implementation modes can also be included in the scope of one or more embodiments of the present invention.
[0199] In the above embodiment, an example in which a photographic image is used as learning data is described, but the learning data is not limited thereto. For example, the learning data may be point group data obtained from an optical sensor other than an image sensor such as LIDAR.
[0200] In addition, the processing performed in the execution determination unit 120 and the consistency determination unit 132 can also use machine learning. In machine learning, for example, there can be listed teacher learning that uses teacher data that assigns labels (output information) to input information to learn the relationship between input and output, untaught learning that constructs data structures based only on untagged inputs, semi-taught learning that can handle both labeled and untagged data, and reinforcement learning that learns continuous actions that can obtain the most rewards by obtaining feedback (rewards) for actions selected based on observation results of the state. In addition, as specific methods of machine learning, there are neural networks (including deep learning using multi-layer neural networks), genetic programming, decision trees, Bayesian networks, support vector machine devices (SVM), etc. In the present disclosure, any of the specific examples listed above can be used.
[0201] Industrial Applicability
[0202] The present disclosure is useful as an information processing method and an information processing system that can stably obtain photographic images serving as learning data from a large number of obtained images without affecting the accuracy of object detection processing using a sensor.
Claims
1. An information processing method, Using a computer, acquiring a plurality of images generated by imaging that is considered to be at the same time, and positions and imaging directions of a plurality of imaging devices that generated each of the plurality of images, acquiring a plurality of object detection results respectively obtained by performing a plurality of object detection processes respectively using the plurality of images, performing a learning data selection process using the acquired plurality of images, the positions and the imaging directions, and the plurality of object detection results, The learning data selection process includes: calculating a common area commonly shown in each of the plurality of images based on the positions and the imaging directions of the plurality of imaging devices, determining the consistency between the plurality of object detection results in the common area, selecting an image to be used as learning data from among the plurality of images based on the degree of consistency, In the learning data selection process, when there are inconsistent object detection results among the plurality of object detection results, an image to be used as learning data is selected from the plurality of images, In a case where the performances of the multiple camera devices are different from each other, or the performances of the multiple object detection processes are different from each other, the higher the performance of the camera device or the higher the performance of the object detection process, the more heavily weighted the object detection result is, and the more accurate the object detection result is, and an image corresponding to an object detection result different from the accurate object detection result is selected as an image set as learning data.
2. The information processing method according to claim 1, and then, determining whether to execute the learning data selection process based on a specific object detection result executed using an image generated by imaging by a specific imaging device among the plurality of imaging devices, If it is determined that the learning data selection process is to be executed, the learning data selection process is executed.
3. The information processing method according to claim 2, The other imaging devices other than the specific imaging device among the plurality of imaging devices are imaging devices capable of capturing an image of a range overlapping with the imaging range of the specific imaging device.
4. The information processing method according to claim 3, The other imaging devices are imaging devices located within a predetermined range from a reference point based on the position of the specific imaging device.
5. The information processing method according to any one of claims 2 to 4, Furthermore, the other imaging devices other than the specific imaging device among the plurality of imaging devices are caused to execute imaging.
6. The information processing method according to any one of claims 2 to 4, The specific object detection result includes a likelihood of the object detection result. In determining whether to execute the learning data selection process, if the likelihood satisfies a predetermined condition, it is determined that the learning data selection process is to be executed.
7. The information processing method according to any one of claims 2 to 4, In determining whether to execute the learning data selection process, acquiring statistical information of at least one of the shooting time of the plurality of images, the position of the camera device that generated the images by the shooting, and the attributes or forms of the objects detected in the plurality of object detection processes, Using the acquired statistical information, it is determined whether or not to execute the learning data selection process.
8. The information processing method according to claim 7, The statistical information includes scarcity information indicating scarcity of the at least one information, In determining whether to execute the learning data selection process, if the scarcity indicated by the scarcity information satisfies a predetermined condition, it is determined that the learning data selection process is to be executed.
9. The information processing method according to claim 1, When the performances of the plurality of imaging devices are the same and the performances of the plurality of object detection processes are the same, an image is selected from the plurality of images to serve as learning data based on the number or ratio of coincidences among the plurality of object detection results.
10. An information processing system comprising: an acquisition unit that acquires a plurality of images generated by imaging that is considered to be at the same time, positions and imaging directions of a plurality of imaging devices in each of the plurality of images that generated the plurality of images, and a plurality of object detection results respectively obtained by a plurality of object detection processes respectively performed using the plurality of images; and an execution unit that executes a learning data selection process using the plurality of images, the positions and the imaging directions, and the plurality of object detection results acquired by the acquisition unit, The execution unit performs, as the learning data selection process: calculating a common area commonly shown in each of the plurality of images based on the positions and the imaging directions of the plurality of imaging devices, determining the consistency between the plurality of object detection results in the common area, selecting an image to be used as learning data from among the plurality of images based on the degree of consistency, In the learning data selection process, when there are inconsistent object detection results among the plurality of object detection results, the execution unit selects an image to be used as learning data from among the plurality of images, In a case where the performances of the multiple camera devices are different from each other, or the performances of the multiple object detection processes are different from each other, the higher the performance of the camera device or the higher the performance of the object detection process, the more heavily weighted the object detection result is, and the more accurate the object detection result is. The execution unit selects an image corresponding to an object detection result different from an object detection result with good accuracy as an image set as learning data.
