Information processing device and method

Abstract

To efficiently collect images for learning of a machine learning model for use in image recognition.SOLUTION: A terminal receives a local image feature amount of one or more first objects included in a first image from a server, acquires a local image feature amount of one or more objects included in a captured image for each of a plurality of captured images of a camera on the basis of a detection result of objects by a first sensor that detects the objects by emitting a predetermined signal for a range including an imaging range of the camera, and transmits, among the plurality of captured images, a second captured image, in which the local image feature amount of the one or more objects included in the captured image is similar to the local image feature amount of one or more first objects, to the server as one of learning data for a machine learning model for use in image recognition. The server transmits the local image feature amount of the one or more first objects to the terminal, receives the second captured image from the terminal, and stores the second captured image in a storage section as one of the learning data for the machine learning model.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to machine learning models used for image recognition. [Background technology]

[0002] A database construction system has been disclosed that automatically adjusts and collects supervised learning data for machine learning to perform object recognition from the output of one sensor using the detection results of another sensor as training data, and constructs a database of supervised learning data (e.g., Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-102838 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to provide an information processing device and method capable of efficiently collecting images for training a machine learning model used in image recognition. [Means for solving the problem]

[0005] One aspect of the present disclosure is receiving, from a server, local image features of one or more first objects included in a first image; Obtaining a detection result of an object by a first sensor that detects an object by transmitting a predetermined signal in a range including an imaging range of the camera; acquiring, for each of a plurality of captured images of the camera, local image features of one or more objects included in the captured images based on the detection result of the object by the first sensor; transmitting, to the server, a second captured image, among the plurality of captured images, in which a local image feature of one or more objects included in the captured image is similar to a local image feature of the one or more first objects, as one piece of learning data for a machine learning model used for image recognition; A control unit that executes The information processing device includes:

[0006] Another aspect of the present disclosure is Transmitting local image features of one or more first objects included in a first image to a first terminal; receiving, from the first terminal, a second captured image of a camera, in which local image features of one or more objects included in the image are similar to local image features of the one or more first objects; storing the second captured image in a storage unit as one piece of learning data for a machine learning model used for image recognition; A control unit that executes Equipped with The local image feature of one or more objects included in the image is acquired based on a detection result of the object by a first sensor that detects the object by emitting a predetermined signal in a range including an imaging range of the camera. It is an information processing device.

[0007] Another aspect of the present disclosure is The device is receiving, from a server, local image features of one or more first objects included in a first image; Obtaining a detection result of an object by a first sensor that detects an object by transmitting a predetermined signal in a range including an imaging range of the camera; acquiring, for each of a plurality of captured images of the camera, local image features of one or more objects included in the captured images based on the detection result of the object by the first sensor; transmitting, to the server, a second captured image, among the plurality of captured images, in which a local image feature of one or more objects included in the captured image is similar to a local image feature of the one or more first objects, as one piece of learning data for a machine learning model used for image recognition; Run The server, transmitting local image features of the one or more first objects to the terminal; receiving the second captured image from the terminal; storing the second captured image in a storage unit as one piece of learning data for the machine learning model; Execute This is the method. Effect of the Invention

[0008] According to the present disclosure, it is possible to efficiently collect images for training a machine learning model used for image recognition. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of a learning data collection system for a machine learning model according to the first embodiment, and an example of a hardware configuration of a center server and a vehicle. [Diagram 2] FIG. 2 is a diagram illustrating an example of a flow of a learning data collection process according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a functional configuration of the center server and the vehicle. [Figure 4] FIG. 4 is a diagram illustrating an example of information included in a collection plan. [Diagram 5] FIG. 5 is an example of a flowchart of a process of collecting re-learning data by the center server. [Figure 6] FIG. 6 is an example of a flowchart of a process for collecting re-learning data for a vehicle. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] A machine learning model can improve the accuracy of estimation by learning. For example, when a machine learning model performs image recognition to determine the type of an object captured from a captured image and its position in the captured image, the recognition result is evaluated based on whether the type and position of the object are correct or not. Re-learning to improve the estimation accuracy of a machine learning model used for image recognition requires selecting images similar to images with poor recognition results and creating teacher data corresponding to the selected images.

[0011] The selection of images for re-learning is often done by humans. However, when selecting images for re-learning from images captured by an in-vehicle camera, for example, a certain length of video must be watched. This results in high human resource costs. In addition, since the criteria for determining whether an image is similar to an image with poor recognition results vary from person to person, it is difficult to ensure consistency in the selection criteria for images for re-learning. In other words, the collection of images for learning in a machine learning model that performs image recognition processing can be inefficient.

[0012] In view of the above problem, one aspect of the present disclosure notifies a terminal of local image features of one or more first objects contained in an image to be trained by a machine learning model, and causes the terminal to transmit to a server an image captured by a camera in which the local image features of the objects contained in the image are similar to the local image features of the first object. This makes it possible to efficiently collect learning images that are similar to the image to be trained and are useful for improving the accuracy of the machine learning model.

[0013] More specifically, one aspect of the present disclosure is an information processing device including a control unit. The control unit receives local image features of one or more first objects included in a first image from a server. The control unit acquires a detection result of an object by a first sensor that detects an object by emitting a predetermined signal in a range including an imaging range of a camera. The control unit acquires local image features of one or more objects included in each of a plurality of captured images of the camera based on the detection result of the object by the first sensor. The control unit transmits to the server, as one piece of learning data for a machine learning model used for image recognition, a second captured image in which the local image features of one or more objects included in the captured image are similar to the local image features of one or more first objects among the plurality of captured images.

[0014] The information processing device in one of the aspects is, for example, a computer connected to an in-vehicle camera such as an ECU or a data communication device (DCM) mounted on a vehicle, or a computer connected to a surveillance camera. The vehicle is, for example, an automobile, a motorcycle, a bicycle, or a railroad car. The information processing device in one of the aspects may be a computer connected to a camera mounted on an aircraft, a ship, or the like other than a vehicle. The information processing device in one of the aspects may be a computer such as a smartphone or a tablet terminal. The control unit is, for example, a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), and an FPGA (Field Programmable Gate Array). The circuit may be a circuit such as a MOS Array.

[0015] The camera may be provided in the information processing device, or may be a device separate from the information processing device and connected to the information processing device. Alternatively, the camera may be a camera that is not provided in the information processing device or connected to it. The first sensor is, for example, a millimeter wave radar, a sonar, or a LiDAR (Light Detection And Ranging). The first sensor is The predetermined signal used is a radio wave or a sound wave signal.

[0016] Machine learning models include, for example, convolutional neural networks (CNs). However, in one aspect of the present disclosure, the machine learning model is not limited to a specific algorithm.

[0017] A local image feature is a feature in a range corresponding to one object in an image. For example, the local image feature is a value related to the color or shape of the object. More specifically, the local image feature includes AKAZE, SIFT (Scaled Invariance Feature Transform), HOG (Histograms of Oriented Gradients), SURF (Speed-Up Robust Features), and LBP (Local Binary Pattern). However, the local image feature is not limited to these. In addition, the local image feature is not limited to one type of feature, and multiple types of feature may be used.

[0018] According to one aspect of the present disclosure, an information processing device detects one of a plurality of captured images by a camera. The second captured image similar to the first image is transmitted to the server as one of the data for learning the machine learning model. For example, when the first image is an image with low estimation accuracy by the machine learning model, the server can collect captured images similar to the first image from the information processing device, which are useful for improving the estimation accuracy of the machine learning model. This makes it possible to efficiently collect data for learning the machine learning model. In addition, by using the object detection result by the first sensor, it becomes easier to identify the position of the object in the captured image by the camera, and the processing load can be reduced.

[0019] Another aspect of the present disclosure is an information processing device including a control unit that executes the following: transmitting local image features of one or more first objects included in a first image to a first terminal; receiving a second captured image from a camera from the first terminal, in which the local image features of one or more objects included in the image are similar to the local image features of the one or more first objects; and storing the second captured image in a storage unit as one piece of learning data for a machine learning model used for image recognition. The information processing device is, for example, a server. The server is, for example, a dedicated computer, a computer such as a PC, etc. The first terminal may acquire the local image features of one or more objects included in the image based on a detection result of the object by the first sensor.

[0020] Another aspect of the present disclosure can be specified as a method in which a computer executes the processes executed by each of the information processing devices. Another aspect of the present disclosure can be specified as a program for causing a computer to execute the method, and a non-transitory computer-readable recording medium having the program recorded thereon.

[0021] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The configurations of the following embodiments are merely examples, and the present disclosure is not limited to the configurations of the embodiments.

[0022] First Embodiment FIG. 1 is a diagram showing an example of a system configuration of a machine learning model learning data collection system 100 according to the first embodiment, and an example of a hardware configuration of a center server 1 and a vehicle 2. The learning data collection system 100 is a system that collects images to be used as learning data for a machine learning model that performs image recognition. The learning data collection system 100 includes a center server 1 and a vehicle 2. The learning data collection system 100 includes a plurality of vehicles 2, but in FIG. 1, for convenience, one vehicle 2 is extracted and shown.

[0023] The center server 1 and the vehicle 2 are connected to a network N1 and can communicate with each other through the network N1. The network N1 is, for example, a public line network such as the Internet. A vehicle information DB 3, a weather information DB 4, and a map information DB 5 are also connected to the network N1.

[0024] The vehicle 2 is a so-called connected car, which has a communication function. Information related to the traveling of the vehicle 2 is periodically collected and stored in the vehicle information DB 3. The vehicle information stored in the vehicle information DB 3 includes, for example, the position information of the vehicle 2, a time stamp, and sensor information such as speed.

[0025] In the first embodiment, a machine learning model used for image recognition for safe driving of a vehicle, such as an advanced driver assistance system (ADAS), is assumed. Therefore, the center server 1 collects images captured by the vehicle 2's on-board camera as learning data for the machine learning model used for image recognition. In the first embodiment, the center server 1 sets, among the images captured by the vehicle 2's on-board camera, images similar to images with low accuracy of the estimation results by the machine learning model as targets to be collected as re-learning data for the machine learning model. The center server The center server 1 notifies the vehicle 2 of local image features of one or more objects included in an image in which the accuracy of the estimation result by the machine learning model is low. The vehicle 2 identifies, from among multiple captured images by the on-board camera, captured images in which the local image features of one or more objects included in the image are similar to the local image features of one or more objects notified from the center server 1, and transmits the captured images to the center server 1.

[0026] This allows the center server 1 to collect, from the vehicle 2, images similar to images with low accuracy of the estimation result by the machine learning model, which are suitable as re-learning data for the machine learning model. Since suitable images as re-learning data for the machine learning model are received from the vehicle 2 with pinpoint accuracy, it is possible to reduce the human cost of manually selecting suitable images as re-learning data from video data of a predetermined length, for example. Therefore, the learning data collection system 100 can improve the efficiency of collecting re-learning data.

[0027] Next, the hardware configuration of each device will be described. The center server 1 can be configured using a dedicated information processing device (computer) such as a server machine. The center server 1 may be a collection (cloud) of one or more computers. The center server 1 is an example of an "information processing device."

[0028] The center server 1 includes, as its hardware configuration, a processor 101, a memory 102, an auxiliary storage device 103, and a communication unit 104. The memory 102 and the auxiliary storage device 103 are computer-readable recording media.

[0029] The auxiliary storage device 103 stores various programs and data used by the processor 101 when executing each program. The auxiliary storage device 103 is, for example, an erasable programmable read-only memory (EPROM), a hard disk drive, or a solid state drive (SSD). The programs held in the auxiliary storage device 103 include, for example, These include an operating system (OS), an application program, and a learning data collection program, etc. The learning data collection program is a program that collects images captured by an in-vehicle camera from the vehicle 2 that are suitable for learning data for a machine learning model.

[0030] The memory 102 is a storage device that provides the processor 101 with a storage area and a working area for loading programs stored in the auxiliary storage device 103, and is used as a buffer. The memory 102 includes, for example, semiconductor memories such as a Read Only Memory (ROM) and a Random Access Memory (RAM).

[0031] The processor 101 executes various processes by loading a program held in the auxiliary storage device 103 into the memory 102 and executing the program. The processor 101 is, for example, a CPU, a GPU, or a DSP (Digital Signal Processor). The number of processors 101 is not limited to one, and multiple processors may be provided. The processor 101 is an example of a "control unit."

[0032] The communication unit 104 is, for example, a network interface card (NIC) or an optical line interface. The communication unit 104 is, for example, a wireless communication unit that connects to a wireless network such as a wireless LAN and a mobile wireless communication network such as 5G, LTE (Long Term Evolution), 6G, etc. The hardware configuration of the center server 1 is not limited to that shown in Fig. 1. The center server 1 may be a device equipped with an electric circuit such as a dedicated FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) that executes the corresponding process.

[0033] Next, the vehicle 2 includes a DCM 210, an ECU 212, a camera 230, a millimeter wave radar 240, and a position acquisition sensor 250. The DCM 210 and the ECU 220 are connected to each other, for example, via an in-vehicle LAN. The ECU 220, the camera 230, the millimeter wave radar 240, and the position acquisition sensor 250 are connected to each other, for example, via a Controller Area Network (CAN). However, the DCM 210, the ECU 220, the camera 230, 1, for the sake of convenience, the hardware components related to the processing according to the first embodiment are extracted and shown, and the vehicle 2 also includes hardware components for controlling traveling, etc.

[0034] The position acquisition sensor 250 is, for example, a GPS (Global Positioning System) receiver. The detection value of the position acquisition sensor is position information of the current location of the vehicle 2. The position information is, for example, latitude and longitude. The position acquisition sensor 250 acquires the position information at a predetermined period and transmits it to the ECU 220 to the DCM 210. The position information is transmitted as one piece of vehicle information from the DCM 210 to the center server 1. The camera 230 may be, for example, a camera shared with a drive recorder or the like.

[0035] The millimeter wave radar 240 performs object detection processing at a predetermined cycle. In the object detection processing, the millimeter wave radar 240 irradiates radio waves in a frequency band with wavelengths in millimeter units and measures the reflected waves to acquire the presence of an object in the irradiation direction and position information such as the distance and angle of the object. In the first embodiment, it is assumed that the imaging direction of the camera 230 and the irradiation direction of the millimeter wave radar 240 face in the same direction and that the imaging range of the camera 230 is included in the irradiation range of the millimeter wave radar 240.

[0036] The ECU 220 is, for example, an ECU that performs processing of an ADAS system or a multimedia ECU. However, the ECU 220 is not limited to these. In the first embodiment, the ECU 220 performs processing to identify an image having a local image feature similar to the local image feature notified from the center server 1 from the captured image of the camera 230 using the object detection result by the millimeter wave radar 240.

[0037] The ECU 220 includes, as a hardware configuration, a processor 221, a memory 222, an auxiliary storage device 223, an interface 224 with an in-vehicle network, and an interface 225 with a CAN. The processor 221, the memory 222, and the auxiliary storage device 223 are similar to the processor 101, the memory 102, and the auxiliary storage device 103, respectively. The auxiliary storage device 223 stores an application program for a client of the learning data collection system 100. The application program for a client of the learning data collection system 100 is installed by the center server 1 through, for example, OTA (Over The Air). The application program for a client of the learning data collection system 100 is a program that identifies an image having a local image feature similar to a local image feature notified from the center server 1 from an image captured by the camera 230. The ECU 220 is an example of an "information processing device." The processor 221 is an example of a "control unit."

[0038] The DCM 210 is responsible for the communication function of the vehicle 2. The DCM 210 includes, as its hardware configuration, a processor 211, a memory 212, an auxiliary storage device 213, a wireless communication unit 214, and an interface 215 with an in-vehicle network. The processor 211, the memory 212, and the auxiliary storage device 213 are similar to the processor 101, the memory 102, and the auxiliary storage device 103, respectively. The wireless communication unit 214 is, for example, a wireless communication circuit that connects to a wireless network such as a wireless LAN and a mobile wireless communication network such as 5G, LTE, or 6G. Note that the hardware configuration of the center server 1 and the vehicle 2 shown in FIG. 1 is an example. Therefore, the configuration is not limited to that shown in FIG.

[0039] 2 is a diagram showing an example of a flow of a process for collecting learning data according to the first embodiment. First, the vehicle 2 periodically transmits vehicle information at a predetermined cycle, and the vehicle information is stored in the vehicle information DB 3.

[0040] In S11, the center server 1 extracts an image with low estimation accuracy from the teacher data of the test image. The teacher data of the image is, for example, image data to which each object included in the image is tagged with the type of object and has position information in the image. A predetermined evaluation index is used to determine the estimation accuracy. The estimation accuracy of the machine learning model can be improved by having the machine learning model learn images similar to the image with low estimation accuracy. Hereinafter, the image with low estimation accuracy is referred to as a question image. The image of the re-learning data to be collected is an image similar to the question image. Hereinafter, the image of the re-learning data to be collected may also be referred to as an image similar to the question image. The question image is an example of a "first image".

[0041] In S12, the center server 1 performs scene analysis on the subject image. The scene analysis is performed, for example, by a method of estimation from the position information of the shooting time and the shooting location, or by using a predetermined image scene recognition technology. The scene analysis may be performed, for example, by a human's visual judgment. By the scene analysis, for example, information on the weather, the time of day, and the driving scene is obtained. The driving scene is, for example, information on the location where the vehicle 2 is driving. The information on the driving scene is, for example, information on whether the vehicle is driving on a highway, driving on a main road, driving through a tunnel, or driving on a mountain road. The information obtained as a result of the scene analysis is stored in the collection plan DB 17.

[0042] In S13, the center server 1 extracts local image features of one or more objects included in the subject image corresponding to the training data extracted in S11 from the subject image. The local image features are acquired using a feature extractor using an algorithm such as AKAZE, SIFT, HOG, SURF, and LBP. The local image feature groups of one or more objects extracted from the subject image are stored in the collection plan DB 17.

[0043] A collection plan for re-learning data for the machine learning model is created by an administrator of the learning data collection system 100. The collection plan includes information on when to start and end collection of re-learning data, from which vehicle 2 the re-learning data is to be collected, how much re-learning data is to be collected, and what kind of images the re-learning data to be collected are. The collection plan is created, for example, by referring to the result of the scene analysis in S12 and using the group of local image features of the subject image acquired in S13.

[0044] In S21, the center server 1 refers to the vehicle information DB 3, the weather information DB 4, and the map information DB 5, and identifies a vehicle to be instructed to collect re-learning data according to the collection plan. This makes it possible to identify a vehicle 2 in a situation similar to the scene analysis result of the challenge image as a target, and to suppress instructing a vehicle 2 that is unlikely to capture an image similar to the challenge image to collect re-learning data. In S22, the center server 1 transmits a data collection start instruction and local image features of the challenge image to the vehicle 2 identified in S21, for example, via OTA.

[0045] In S31, the vehicle 2 detects an object using the millimeter wave radar 240. In object detection by the millimeter wave radar 240, for example, the presence of an object is detected, but the type of object is not detected. In object detection by the millimeter wave radar 240, the size and position of the detected object are also acquired. The millimeter wave radar 240 is an example of a "first sensor."

[0046] In S32, the vehicle 2 converts the object detection result in S31 from the coordinates of the millimeter wave radar 240 to the coordinates of the camera 230 using a predetermined coordinate conversion method. In S33, the vehicle 2 specifies the range in which the object is located in the captured image as the position of the object to be recognized, based on the coordinate conversion result. Note that the captured image used in S33 is captured at the time closest to the time when the object was detected by the millimeter wave radar 240 in S31.

[0047] In S34, the vehicle 2 extracts local image features for the positions of the objects identified in S33 in the captured image of the camera 230. The type of feature is the same as the local image features of the subject image. When multiple objects are detected by the millimeter wave radar 240, the position of each object in the captured image is identified, and local image features are extracted.

[0048] In S35, the vehicle 2 compares the similarity between the local image feature groups of the subject image and the local image feature groups of the images captured by the cameras 230 between the images captured by the multiple cameras 230. The similarity comparison between the subject image and the images captured by the cameras 230 may be performed using, for example, Bag-of-Visual Words (BoVW) Methods such as the following are used.

[0049] In S36, the vehicle 2 transmits to the center server 1, for example, data of a predetermined number of captured images having a top similarity or a captured image having a similarity equal to or greater than a predetermined threshold value among the multiple captured images. Position information of an object in the captured image obtained by coordinate transformation in S32 is also transmitted together with the captured image data. The object position information is, for example, information indicating an area in the captured image in which an object exists. For example, when the captured image is displayed, a frame showing the approximate shape of the object is displayed at the position of the object based on the object position information, making it easier to visually recognize the position where the object exists. The captured image transmitted from the vehicle 2 to the center server 1 is an example of a "second captured image."

[0050] In S41, the center server 1 stores the captured image data received from the vehicle 2 as one of the input data for learning data in the re-learning image information DB 18. The center server 1 also stores position information of an object included in the captured image received from the vehicle 2 in the re-learning image information DB 18 as auxiliary information for creating teacher data.

[0051] Thereafter, teacher data is created using the captured image data received from the vehicle 2 and position information of objects included in the captured image. For example, the types of objects included in the captured image are determined by a person, labeled, and the position information of each object is added to create the teacher data. For example, a frame indicating the range in which the object exists can be displayed on the captured image displayed on a display based on the position information of the objects included in the captured image. This makes it easier for the worker to identify the object to be labeled from the captured image, improving work efficiency.

[0052] By retraining the machine learning model using the image data and teacher data as input data for the re-learning data collected in this manner, it is possible to improve the estimation accuracy of the machine learning model. The machine learning model may be provided in the center server 1, or may be provided in a device different from the center server 1. Furthermore, the learning of the machine learning model may be performed by the center server 1, or may be performed by a device different from the center server 1.

[0053] 3 is a diagram showing an example of the functional configuration of the center server 1 and the vehicle 2. The center server 1 includes, as its functional configuration, an evaluation unit 11, a control unit 12, an image feature extraction unit 13, a scene analysis unit 14, a test image information DB 16, a collection plan DB 17, and a re-learning image information DB 18. These are executed, for example, by a processor 101 of the center server 1 running a predetermined program. In addition, each or a part of the functional components may correspond to different programs or different hardware components (such as FPGA).

[0054] The evaluation unit 11 evaluates the estimation result of the machine learning model of each image of the learned learning data according to a predetermined evaluation criterion, and extracts a challenge image. The control unit 12 controls the collection process of re-learning data to be used as a challenge image, which corresponds to the processes of S21, S22, and S41 in Fig. 2, for example. The process of the control unit 12 will be described in detail later. The image feature extraction unit 13 extracts local image features from the image. The scene analysis unit 14 performs scene analysis of the image using a predetermined method.

[0055] The test image information DB 16, collection plan DB 17, and re-learning image information DB 18 are created, for example, in a storage area of ​​the auxiliary storage device 103 of the center server 1. The test image information DB 16 is learning data including test images as input data and teacher data of the test images, and holds information on the learning data that has been learned by the machine learning model. The collection plan DB 17 holds a collection plan, as well as scene analysis results and local image feature groups for the subject image. Details of the information contained in the collection plan will be described later. Re-learning image information DB 18 stores images similar to the subject image collected from the vehicle 2 as input data, auxiliary information for creating the training data, and the training data.

[0056] Next, the vehicle 2 includes, as functional components, a control unit 21, an object detection unit 22, and an image feature extraction unit 23. These may be functions achieved by, for example, the processor 221 of the ECU 220 executing a predetermined program. Furthermore, each or a part of the functional components may correspond to a different program or a different hardware component (such as an FPGA).

[0057] The control unit 21 controls the process of identifying an image similar to the subject image from an image captured by the camera 230 and transmitting the image to the center server 1, which corresponds to the processes of S31 to S36 in Fig. 2, for example. The process of the control unit 21 will be described in detail later. The object detection unit 22 controls object detection by the millimeter wave radar 240. The image feature extraction unit 23 extracts local image features from an image. The local image features acquired by the image feature extraction unit 23 are the same type of local image features as the local image features acquired by the image feature extraction unit 13 of the center server 1.

[0058] The vehicle information DB 3 holds vehicle information periodically collected from the vehicle 2 at a predetermined cycle. The vehicle information includes, for example, the location information and speed of the vehicle 2. The weather information DB 4 holds information related to weather. The weather information stores information related to weather, sunshine, precipitation, wind speed, and lightning occurrence for each area. The map information DB 5 stores map information. The vehicle information DB 3, the weather information DB 4, and the map information DB 5 may be databases managed by the learning data collection system 100, or may each be databases managed by an external organization. The functional configurations of the center server 1 and the vehicle 2 are not limited to the example shown in FIG. 3.

[0059] 4 is a diagram showing an example of information included in a collection plan. The collection plan includes fields for a plan ID, a subject image ID, a start condition, a target vehicle condition, an end condition, and a local image feature. The plan ID field stores identification information of the collection plan. The subject image ID field stores identification information of the subject image.

[0060] The start condition field stores information that defines the start condition of the process of collecting re-learning data for the subject image. The start condition field includes a time period field. The time period field stores the time period when the process of collecting re-learning data for the subject image is started. The start condition field may be blank, in which case, when a start command is input from the administrator of the learning data collection system 100, a collection process of re-learning data for the target image is started. The information indicating the time period may be determined based on the time period when the target image is captured, which is obtained by scene analysis of the target image, or may be a time period arbitrarily designated by the administrator of the learning data collection system 100. The start condition field may be blank, in which case, when a start command is input from the administrator of the learning data collection system 100, a collection process of re-learning data for the target image is started.

[0061] The field of the target vehicle conditions stores information defining the conditions of the vehicle 2 that is the target of the instruction to collect the re-learning data corresponding to the task image. The field of the target vehicle conditions includes fields of area, weather, and driving scene. The field of area stores information indicating a geographical range for collecting the re-learning data corresponding to the task image. The field of weather stores information indicating the weather that is the target of the collection of the re-learning data corresponding to the task image. The field of driving scene stores information indicating the driving scene that is the target of the collection of the re-learning data corresponding to the task image. The information stored in the fields of area, weather, and driving scene may be determined based on, for example, a scene analysis result of the task image, or may be arbitrarily specified by the administrator of the learning data collection system 100. Note that the fields of area, weather, and driving scene may be empty. When the target vehicle conditions are not set, for example, the vehicle 2 that is the target of the instruction to collect the re-learning data corresponding to the task image may be randomly specified.

[0062] The field of the termination condition stores information that defines the termination condition of the collection process of re-learning data for the subject image. The field of the termination condition includes fields for the number of collected images and the number of collected vehicles. The field of the number of collected images stores a lower limit value of the number of images to be collected as re-learning data for the subject image. When the number of captured images collected from the vehicles 2 reaches the value in the field of the number of collected images, the end of the collection process of the re-learning data for the subject image is determined. The field of the number of collected vehicles stores the number of vehicles 2 to be collected as re-learning data for the subject image. When captured images are collected from the number of vehicles 2 equal to the value in the field of the number of collected vehicles, the end of the collection process of the re-learning data for the subject image is determined. The termination condition may be defined by either the number of collected images or the number of collected vehicles, or may be defined by both. When the termination condition is defined by both the number of collected images and the number of collected vehicles, the end of the collection process of the re-learning data for the subject image is determined when both are satisfied. The field for the end condition may be left blank, in which case, for example, a predetermined number of collected sheets may be used as the end condition.

[0063] A group of local image features extracted from the subject image is stored in the field of the local image feature amount. The group of local image features stored in the field of the local image feature amount is notified to the vehicle 2.

[0064] 4 is an example, and each condition included in the collection plan can be flexibly set depending on what kind of images are desired to be collected as learning data by the learning data collection system 100. For example, the number of captured images to be collected from one vehicle 2 may be defined as the end condition.

[0065] Furthermore, the image data collected from the vehicle 2 may be further narrowed down using, for example, vehicle information to acquire re-learning data. The vehicle information includes, for example, information such as the amount of depression of the brake pedal and the amount of downward rotation of the steering wheel, and it is possible to detect sudden braking or sudden steering. For example, if it is desired to collect images of sudden braking and sudden steering as learning data, the vehicle information may be attached to the image data from the vehicle 2, and the image data collected from the vehicle 2 that is accompanied by vehicle information indicating sudden braking or sudden steering may be acquired as re-learning data. Alternatively, the image data may be acquired by issuing a command to start data collection together with a command to start data collection, and the image data may be acquired by issuing a command to start data collection together with a command to start data collection. An instruction to transmit images during raking and sudden steering may be transmitted so that the vehicle 2 transmits images similar to the subject images during sudden braking and sudden steering.

[0066] Fig. 5 is an example of a flowchart of a process for collecting re-learning data in the center server 1. The process shown in Fig. 5 is executed for each collection plan stored in the collection plan DB 17. The process shown in Fig. 5 is repeatedly executed at a predetermined period. The process shown in Fig. 5 is executed by, for example, the processor 101 of the center server 1, but for convenience, the process will be described mainly with respect to functional components.

[0067] In OP101, the control unit 12 determines whether or not a start condition included in the collection plan is satisfied. If the start condition is satisfied (OP101: YES), the process proceeds to OP102. If the start condition is not satisfied (OP101: NO), the process shown in FIG. 5 ends.

[0068] In OP102, the control unit 12 extracts a plurality of vehicles 2 that satisfy the target vehicle conditions included in the collection plan. In OP103, the control unit 12 transmits a data collection start instruction and a group of local image features of the subject image to each vehicle 2 extracted in OP102. In OP104, the control unit 12 executes a receiving process of receiving image data as re-learning data from the vehicle 2 and storing the image data in the re-learning image information DB 18.

[0069] In OP105, the control unit 12 determines whether or not a termination condition included in the collection plan is satisfied. If the termination condition is satisfied (OP105: YES), the process proceeds to OP106. If the termination condition is not satisfied (OP105: NO), the process proceeds to OP104.

[0070] In OP106, a data collection end instruction is transmitted to the target vehicle, and then the process shown in FIG.

[0071] Fig. 6 is an example of a flowchart of a process for collecting re-learning data for the vehicle 2. The process shown in Fig. 6 is repeatedly executed at a predetermined cycle. The process shown in Fig. 6 is executed mainly by the processor 221 of the ECU 220, but for convenience, the process will be described mainly by the functional components.

[0072] In OP201, the control unit 21 determines whether or not a data collection start instruction has been received from the center server 1. When a data collection start instruction has been received from the center server 1 (OP201: YES), the process proceeds to OP202. Together with the data collection start instruction, a group of local image features of the subject image is also received. When a data collection start instruction has not been received from the center server 1 (OP201: NO), the process shown in FIG. 6 ends.

[0073] In OP202, the control unit 21 determines whether or not a predetermined time has elapsed. The predetermined time is set arbitrarily within a range of, for example, 1 second to 10 seconds. If the predetermined time has elapsed (OP202: YES), the process proceeds to OP203. The control unit 21 is in a standby state until the predetermined time has elapsed.

[0074] In OP203, the control unit 21 acquires a plurality of object detection results of the millimeter wave radar 240 performed within a most recent predetermined time length from the object detection unit 22. In OP204, the control unit 21 converts each of the object detection results of the millimeter wave radar 240 acquired in OP203 into camera coordinates. In OP205, the control unit 21 specifies the position of an object in each of a plurality of images captured by the camera 230 for the most recent predetermined time length based on the corresponding object detection result.

[0075] In OP206, the control unit 21 requests the image feature extraction unit 23 for each of the most recent multiple captured images of the camera 230 for a predetermined time length, and acquires local image features extracted from the position of the object identified in OP205. In OP207, the control unit 21 compares the similarity of the local image feature group of the subject image between the most recent multiple captured images of the camera 230 for a predetermined time length, and determines a captured image similar to the subject image. The captured image similar to the subject image may be one image or may be multiple images. In OP208, the control unit 21 transmits the captured image similar to the subject image and the position information of the object in the captured image obtained in OP204 to the center server 1.

[0076] In OP209, the control unit 21 determines whether or not a data collection end instruction has been received from the center server 1. If a data collection end instruction has been received from the center server 1 (OP209: YES), the processing shown in FIG. 6 ends. If a data collection end instruction has not been received from the center server 1 (OP209: NO), the processing proceeds to OP202. As a result, the processing from OP203 is performed after a predetermined time has elapsed. By executing the processing from OP203 to OP209 at predetermined time intervals, it is possible to suppress the load caused by the processing for collecting re-learning data from affecting other processing of the ECU 220.

[0077] Note that the process of acquiring data for relearning in the center server 1 shown in FIG. 5 and the process of acquiring data for relearning in the vehicle 2 shown in FIG. 6 are each merely examples and can be modified as appropriate depending on the embodiment.

[0078] <Effects of the First Embodiment> In the first embodiment, the center server 1 can pinpoint images captured by an on-board camera similar to a subject image and collect them from the vehicle 2 as learning data for a machine learning model. This makes it possible to collect images suitable for learning a machine learning model and with consistent selection criteria without human intervention, improving the efficiency of the collection process of learning data for the machine learning model. Furthermore, by learning using learning data with consistent selection criteria, the estimation accuracy of the machine learning model can be further improved.

[0079] Furthermore, in the first embodiment, the vehicle 2 uses the object detection result by the millimeter wave radar 240 to identify the position of the object in the corresponding captured image of the camera 230. This makes it easier to identify a target area from which to extract local image features in the captured image of the camera 230, reducing the load of the re-learning data collection process in the vehicle 2 and shortening the time required for the process. Furthermore, by transmitting position information of the object in the captured image together with the captured image determined to be a similar image to the subject image, it becomes easier for a person to visually recognize the position of the object in the captured image, and the efficiency of the human work involved in creating the teacher data can be improved.

[0080] Furthermore, the collection process of re-learning data according to the first embodiment can be executed by the vehicle 2 simply by installing the client program of the learning data collection system 100 over the air. Therefore, in the first embodiment, the collection of re-learning data can be achieved without changing the hardware configuration of the vehicle 2.

[0081] Furthermore, in the first embodiment, a collection plan is created based on the scene analysis results of the subject image, and the center server 1 instructs a vehicle in a situation similar to the situation in which the subject image was captured to start collecting re-learning data based on the collection plan, at a timing that takes into account the situation in which the subject image was captured. This can improve the probability that images collected from the vehicle 2 are more suitable as learning data. Furthermore, even if the processing capacity of the vehicle 2 is not high, captured images can be collected as more suitable learning data.

[0082] <Other embodiments> The above-described embodiment is merely an example, and the present disclosure can be modified and implemented as appropriate without departing from the spirit and scope of the present disclosure.

[0083] In the first embodiment, the center server 1 collects captured images from the vehicle 2 as learning data for the machine learning model, but the terminal from which the learning data is collected is not limited to the vehicle 2, and can be changed as appropriate depending on the purpose of using the machine learning model. For example, the technology described in the first embodiment may be applied to a system that collects captured images from cameras mounted on motorcycles, bicycles, railway cars, etc. as learning data. In addition, the technology may be applied to a system that collects captured images from cameras mounted on aircraft, ships, etc. as learning data, not limited to vehicles moving on land. Or, the technology may be applied to a system that collects captured images from surveillance cameras, fixed cameras, etc. as learning data.

[0084] In the first embodiment, the vehicle 2 uses the object detection results of the millimeter wave radar 240 to identify the position of an object in a captured image, but instead of the millimeter wave radar 240, sensors used in other ADAS systems, such as sonar or LiDAR, may be used.

[0085] In the first embodiment, the ECU 220 performs the process of collecting re-learning data in the vehicle 2, but the hardware components that execute the process of collecting re-learning data in the vehicle 2 are not limited to the ECU 220. For example, the DCM 210, a drive recorder system, a car navigation system, or the like may execute the process of collecting re-learning data in the vehicle 2.

[0086] In the first embodiment, the center server 1 instructs the vehicle 2 to start and end the process of collecting data for re-learning, but is not limited to this. For example, a start condition and an end condition for each vehicle 2 may be transmitted together with the data collection start instruction, and the vehicle 2 may determine the start and end of the process of collecting data for re-learning. The start condition may be defined by, for example, the time period, area, weather, and driving scene included in the collection plan. The end condition may be defined by, for example, the number of images similar to the subject image collected per vehicle 2.

[0087] The processes and means described in this disclosure can be freely combined and implemented as long as no technical contradiction occurs.

[0088] Furthermore, a process described as being performed by one device may be shared and executed by multiple devices. Alternatively, a process described as being performed by different devices may be executed by one device. In a computer system, the hardware configuration (server configuration) by which each function is realized can be flexibly changed.

[0089] The present disclosure can also be realized by supplying a computer program implementing the functions described in the above embodiments to a computer, and having one or more processors of the computer read and execute the program. Such a computer program may be provided to the computer by a non-transitory computer-readable storage medium connectable to the system bus of the computer, or may be provided to the computer via a network. Non-transitory computer-readable storage media include, for example, any type of disk, such as a magnetic disk (floppy disk, hard disk drive (HDD), etc.), an optical disk (CD-ROM, DVD disk, Blu-ray disk, etc.), a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, an optical card, and any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0090] 1. Center server 2. Vehicle 11. Evaluation section 12, 21 Control section 13, 23··Image feature extraction section 14. Scene Analysis Section 16 Test image information DB 17 Collection Plan DB 18. Image information DB for re-learning 22. Object detection unit 100··Learning data collection system 101, 221 Processor 102, 222...Memory 103, 223...Auxiliary storage device 104 Communications Department 210··DCM 220··ECU 230··Camera 240 mm wave radar 250·· Position acquisition sensor

Claims

1. receiving, from a server, local image features of one or more first objects included in a first image; Obtaining a detection result of an object by a first sensor that detects an object by transmitting a predetermined signal within a range including an imaging range of the camera; acquiring, for each of a plurality of captured images of the camera, local image features of one or a plurality of objects included in the captured images based on the detection result of the object by the first sensor; transmitting, to the server, a second captured image, among the plurality of captured images, in which a local image feature of one or more objects included in the captured image is similar to a local image feature of the one or more first objects, as one piece of learning data for a machine learning model used for image recognition; A control unit that executes An information processing device comprising:

2. The control unit is identifying one or more positions in the captured images of one or more objects included in the captured images based on the detection result of the object by the first sensor between the plurality of captured images, acquiring local image features of the one or more objects included in the captured images from the identified one or more positions in the captured images, and comparing a similarity between the local image features of the one or more objects included in the acquired captured images and the local image features of the one or more first objects; determining, as the second captured image, a captured image in which the local image feature amount of one or more objects included in the captured image is similar to the local image feature amount of the one or more first objects based on a result of the comparison; and Further execute transmitting, to the server, position information in the second captured image of one or more objects included in the second captured image and identified based on the detection result of the object by the first sensor, together with the second captured image; The information processing device according to claim 1 .

3. Transmitting local image features of one or more first objects included in a first image to a first terminal; receiving, from the first terminal, a second captured image of a camera, in which local image features of one or more objects included in the image are similar to local image features of the one or more first objects; storing the second captured image in a storage unit as one piece of learning data for a machine learning model used for image recognition; A control unit that executes Equipped with The local image feature amounts of one or more objects included in the image are acquired based on a detection result of the object by a first sensor that detects the object by emitting a predetermined signal in a range including an imaging range of the camera. Information processing device.

4. The control unit is and further comprising identifying the first terminal from the plurality of terminals based on at least one of a location of the terminal, a weather condition, and a time of day. The second captured image is sent from the first terminal together with the second captured image. receiving positions in the second captured image of one or more objects included in the second captured image identified based on a detection result; The learning data includes teacher data corresponding to the second captured image that is created based on positions in the second captured image of one or more objects included in the second captured image. The information processing device according to claim 3 .

5. The device is receiving, from a server, local image features of one or more first objects included in a first image; Obtaining a detection result of an object by a first sensor that detects an object by transmitting a predetermined signal within a range including an imaging range of the camera; acquiring, for each of a plurality of captured images of the camera, local image features of one or a plurality of objects included in the captured images based on the detection result of the object by the first sensor; transmitting, to the server, a second captured image, among the plurality of captured images, in which a local image feature of one or more objects included in the captured image is similar to a local image feature of the one or more first objects, as one piece of learning data for a machine learning model used for image recognition; Run The server, transmitting local image features of the one or more first objects to the device; receiving the second captured image from the terminal; storing the second captured image in a storage unit as one piece of learning data for the machine learning model; Execute method.

Images