Information processing device, information processing system, information processing method, and information processing program

The system optimizes learning data use by matching sensor information with appropriate recognizers based on scene and metadata analysis, enhancing recognition performance and reducing costs in obstacle detection systems.

JP7586095B2Active Publication Date: 2024-11-19SONY GROUP CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2021567309
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-12-24
Filing Date
2020-12-15
Publication Date
2024-11-19
Estimated Expiration
2040-12-15

AI Technical Summary

Technical Problem

Existing information processing systems face inefficiencies in using learning data for obstacle detection due to varying attributes based on acquisition situations, limiting the effective use of data across different regions and environments.

Method used

An information processing device and system that utilizes a recognition unit with multiple recognizers, a learning data application determination unit, and a transmission determination unit to selectively apply and transmit sensor information for relearning, ensuring efficient use of learning data by matching it to appropriate recognizers based on scene information and metadata analysis.

Benefits of technology

Enhances the efficiency of learning data utilization by applying sensor information to the most suitable recognizers, improving recognition performance while reducing communication costs and ensuring data relevance across diverse environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007586095000001
    Figure 0007586095000001
  • Figure 0007586095000002
    Figure 0007586095000002
  • Figure 0007586095000003
    Figure 0007586095000003
Patent Text Reader

Abstract

The purpose of the present invention is to provide an information processing device, an information processing system, an information processing method, and an information processing program which enable efficient use of learning data. An information processing device according to this disclosure comprises: a recognition unit (101) that performs, with a first recognizer trained in advance, object recognition processing using sensor information acquired by a sensor; and a learning data application determination unit (22d) that determines whether the sensor information is applicable, as learning data, to a second recognizer which is different from the first recognizer.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] An information processing system has been proposed that uses an on-board camera mounted on a vehicle to analyze images captured while the vehicle is traveling and determines whether or not an obstacle such as a person is present in the vehicle's traveling direction. In addition, a technology has been proposed for such an information processing system that detects obstacles based on captured images using a recognition model learned by machine learning. Also, a configuration is known in which the recognition model is constructed by machine learning on a network, for example, on a server.

[0003] Incidentally, when detecting obstacles and the like based on captured images using machine learning, it is preferable to collect learning data from time to time, perform re-learning, and update the recognition model. For example, the server acquires and collects data acquired in the vehicle via communication, and uses the collected data as learning data to perform re-learning and update the recognition model. The updated recognition model is transmitted to the vehicle via communication. By using this updated recognition model, the vehicle side can more accurately detect obstacles and the like. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-202595 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the data sent for re-learning has various attributes depending on the circumstances in which the data was acquired, such as the region in which the data was sent, and it could only be used to the extent that the attributes were valid.

[0006] An object of the present disclosure is to provide an information processing device, an information processing system, an information processing method, and an information processing program that enable efficient use of learning data. [Means for solving the problem]

[0007] The information processing device according to the present disclosure includes a recognition unit that performs an object recognition process using sensor information acquired by a sensor with a first recognizer that has been trained in advance, and a learning data application determination unit that determines whether or not the sensor information can be applied as learning data to a second recognizer that is different from the first recognizer; a transmission determination unit that determines whether or not to transmit the sensor information to a server capable of generating the first recognizer so as to apply the sensor information to the second recognizer as the learning data, based on the sensor information; Equipped with The transmission determination unit transmits to the server, among the sensor information, sensor information in which a number of recognizers capable of performing the recognition process designated by the learning data application determination unit from the second recognizer is equal to or greater than a predetermined number. . [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of an overall view of a driving assistance system related to each embodiment. [Diagram 2] 1 is a block diagram showing a configuration of an example of a driving assistance system according to each embodiment. [Diagram 3] FIG. 2 is a block diagram showing an example of a hardware configuration of a terminal device applicable to each embodiment. [Figure 4] FIG. 2 is a block diagram showing an example of a hardware configuration of a server system applicable to an embodiment. [Diagram 5] 1 is a block diagram showing a configuration of an example of a driving assistance system according to a first embodiment. [Figure 6] FIG. 2 is a functional block diagram illustrating an example of a function of the server system according to the first embodiment. [Figure 7] 4 is a flowchart illustrating an example of a process in the server system according to the first embodiment. [Figure 8]FIG. 2 is a functional block diagram illustrating an example of a function of a server system according to a first modified example of the first embodiment. [Figure 9] 4 is a flowchart illustrating an example of a process in the server system according to the first embodiment. [Figure 10] FIG. 11 is a functional block diagram illustrating an example of a function of a server system according to a second modified example of the first embodiment. [Figure 11] 10A and 10B are schematic diagrams illustrating examples of images based on image data included in sensor information. [Figure 12] 11 is a flowchart illustrating an example of a process in a server system according to a second modified example of the first embodiment. [Figure 13] FIG. 11 is a block diagram showing a configuration of an example of a driving assistance system according to a second embodiment. [Figure 14] FIG. 11 is a functional block diagram illustrating an example of functions of a terminal device according to a second embodiment. [Figure 15] 10 is a flowchart illustrating an example of a process in a terminal device according to the second embodiment. [Figure 16] FIG. 13 is a functional block diagram illustrating an example of functions of a terminal device according to a first modified example of the second embodiment. [Figure 17] 13 is a flowchart of an example showing a process in a terminal device according to a first modified example of the second embodiment; [Figure 18] FIG. 13 is a functional block diagram illustrating an example of functions of a terminal device according to a second modification of the second embodiment. [Figure 19] 13 is a flowchart illustrating an example of a process in a terminal device according to a second modified example of the second embodiment. [Figure 20] FIG. 13 is a functional block diagram illustrating an example of functions of a terminal device according to a third modified example of the second embodiment. [Figure 21] 13 is a flowchart illustrating an example of a process in a terminal device according to a third modified example of the second embodiment. [Figure 22] FIG. 11 is a block diagram showing a configuration of an example of a driving assistance system according to a third embodiment. [Figure 23] FIG. 13 is a functional block diagram illustrating an example of functions of a terminal device according to a third embodiment. [Figure 24] 13 is a flowchart illustrating an example of a process in a terminal device according to the third embodiment. [Diagram 25] FIG. 13 is a functional block diagram illustrating an example of functions of a terminal device according to a first modified example of the third embodiment. [Figure 26] 13 is a flowchart illustrating an example of a process in a terminal device according to a first modified example of the third embodiment. [Figure 27] FIG. 13 is a functional block diagram illustrating an example of functions of a terminal device according to a second modified example of the third embodiment. [Figure 28] 13 is a flowchart illustrating an example of a process in a terminal device according to a second modified example of the third embodiment. [Figure 29] 1 is a block diagram showing a schematic configuration example of a vehicle control system that is an example of a mobile object control system to which the technology according to the present disclosure can be applied. [Diagram 30] FIG. 4 is a diagram showing an example of an installation position of an imaging unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0010] Hereinafter, embodiments of the present disclosure will be described in the following order. 1. Technologies Applicable to Each Embodiment 1-1. Overview of systems applicable to each embodiment 1-2. Overview of the system according to each embodiment 1-3. Hardware configuration example 2. First embodiment 2-1. First modified example 2-2. Second modified example 3. Second embodiment 3-1. First modified example 3-2. Second modified example 3-3.Third modified example 4. Third embodiment 4-1. First modified example 4-2. Second modified example 5. Fourth embodiment

[0011] [1. Technologies Applicable to Each Embodiment] (1-1. Overview of Systems Applicable to Each Embodiment) First, a technique applicable to each embodiment of the present disclosure will be described. Fig. 1 is a schematic diagram of an example showing an overall view of a driving assistance system related to each embodiment. In Fig. 1, the driving assistance system includes a server system 2 and one or more vehicles 10, each of which is connected to a network 1. The network 1 is, for example, a V2X network for performing vehicle-to-vehicle and road-to-vehicle (V2X) communication.

[0012] The server system 2 may be, for example, a cloud system including a plurality of computers and storage devices that are connected to each other via a network and operate in cooperation with each other. The server system 2 may also be configured as a server device using a single computer. The server system 2 may generate a recognizer that performs object recognition based on sensor information by machine learning.

[0013] A vehicle 10 is equipped with a terminal device 11 connected to a network 1. The terminal device 11 is configured as an information processing device including, for example, a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), and various interfaces, and includes a recognizer that performs object recognition processing based on the output of a sensor 12. The sensor 12 is, for example, a camera. In the following description, the sensor 12 is described as being the camera 12. The recognizer is installed in the terminal device 11, for example, by downloading a recognizer generated by a server system 2 via the network 1. However, the present invention is not limited to this, and the terminal device 11 may be equipped with a recognizer in advance.

[0014] The terminal device 11 performs object recognition processing using a recognizer based on the captured image acquired by the camera 12. The terminal device 11 can perform driving assistance in the vehicle 10 by this object recognition processing. Here, it is preferable to update the recognizer from time to time using new learning data, since this can improve the performance of the recognition processing. Therefore, the terminal device 11 transmits image data of the captured image acquired by the camera 12 to the server system 2 via the network 1 (steps S1 and S2).

[0015] The server system 2 accumulates the image data transmitted from the terminal device 11 in the storage unit 20. The storage unit 20 may be, for example, a storage device using a hard disk drive or a large-scale flash memory, and further accumulates a recognizer and parameter information for generating a recognizer that performs object recognition processing. The server system 2 uses, in the learning unit 3, each piece of data stored in the storage unit 20, including the image data transmitted from the terminal device 11, as learning data (step S3), and performs optimization and re-learning of the recognizer (step S4). Through this optimization and re-learning, a re-learned recognizer 21 in which the recognizer is updated is generated (step S5).

[0016] The server system 2 transmits the generated re-trained recognizer 21 to the terminal device 11 via the network 1 (steps S6 and S7). The terminal device 11 updates the existing recognizer with the re-trained recognizer 21 transmitted via the network 1. In this way, by re-training the existing recognizer using newly collected learning data, it is possible to improve the performance of the recognition process by the recognizer.

[0017] In such a driving assistance system, while it is possible to improve the machine learning method and retrain the recognizer, retraining using new training data is also effective. In order to improve the performance of the recognizer by retraining using new training data, it is desirable to continue collecting training data (e.g., image data) that contributes to performance improvement.

[0018] At this time, the time series data and image data transmitted from the terminal device 11 to the server system 2 for use in relearning are generally large in size, which may result in high communication costs. In addition, the time series data and image data have various attributes, which may differ depending on various factors, such as for each vehicle 10 or each region. As an example, recognition data regarding traffic road signs may differ depending on countries, and image data including traffic road signs obtained in country A may not be usable as learning data for recognizing traffic road signs in country B.

[0019] In each embodiment of the present disclosure, an object is to enable efficient use of new learning data to be used for re-learning.

[0020] (1-2. Overview of the System According to Each Embodiment) Next, an overview of the driving support system according to each embodiment will be described. Fig. 2 is a block diagram showing the configuration of an example of the driving support system according to each embodiment. In Fig. 2, parts closely related to each embodiment and each modified example thereof are indicated by bold frames.

[0021] 2, the server system 2 includes a learning data application determination unit 22 in addition to the above-mentioned learning unit 3 and storage unit 20. The storage unit 20 accumulates learning data and stores a plurality of recognizers. As will be described in detail later, the learning data application determination unit 22 specifies a recognizer to be re-trained using sensor information from among the recognizers stored in the storage unit 20, based on sensor information transmitted from the terminal device 11. The server system 2 transmits information indicating the recognizer specified by the learning data application determination unit 22 to the terminal device 11.

[0022] 2, the terminal device 11 includes a recognition unit 101, a control unit 102, an accumulation determination unit 103, an accumulation unit 104, a storage unit 105, an accumulated information output unit 106, a transmission determination unit 107, and a communication unit 110. The terminal device 11 may further include a learning data application determination unit 108.

[0023] The recognition unit 101 includes a recognizer downloaded from the server system 2 , for example, and performs object recognition processing based on sensor information including image data of an image captured by the camera 12 .

[0024] At this time, the recognition unit 101 can use RAW data as the captured image captured by the camera 12. That is, image data captured by the camera 12 and used for visual recognition is generally subjected to image processing such as demosaic processing and compression processing of each of the RGB color data to, for example, 8 bits. In contrast, such image processing is often unnecessary for recognition processing, and RAW data using the output of the image sensor of the camera 12 in a substantially unchanged state can achieve a higher recognition rate (recognition score), particularly for recognition at night or at a distance. The recognition score is a value indicating the degree of recognition, and ranges from 0 to 1, for example. A higher recognition score indicates a higher degree of recognition.

[0025] Furthermore, when taking into consideration the recognition of distant objects in particular, it is preferable that the resolution of the camera 12 (the resolution of the image sensor) is high.

[0026] The object recognition result by the recognition unit 101 is passed to a control unit 102 that performs driving control of the vehicle 10. The control unit 102 performs automatic driving control such as obstacle avoidance based on the object recognition result passed from the recognition unit 101.

[0027] Here, the sensor information includes the image data, as well as metadata, which is attribute information related to the camera 12 and the image captured by the camera 12. The metadata can include information related to the performance of the camera, such as the model number, resolution, and frame rate of the camera 12, and information indicating image processing (white balance processing, gain adjustment processing, etc.) executed by the camera 12. The metadata can further include information related to the vehicle 10, such as information indicating the model and destination of the vehicle 10, information related to the current location of the vehicle 10, and the traveling speed of the vehicle 10.

[0028] The accumulation determination unit 103 determines whether or not the sensor information output from the camera 12 is to be accumulated by the accumulation unit 104 in, for example, the memory unit 105, which is a storage device. A specific example will be described later, but the accumulation determination unit 103 determines whether or not the sensor information is to be accumulated in the memory unit 105 by the accumulation unit 104, for example, based on image data and metadata included in the sensor information. The accumulated information output unit 106 outputs the sensor information accumulated in the memory unit 105 to the outside. The accumulated information output unit 106 may output the sensor information to the outside by wired communication, or can also output the sensor information to the outside by wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0029] The transmission determination unit 107 determines whether or not to transmit the sensor information output from the camera 12 to the server system 2. For example, the transmission determination unit 107 determines whether or not to transmit the sensor information to the server system 2 based on the recognition result by the recognition unit 101 and the sensor information. When the transmission determination unit 107 determines to transmit the sensor information to the server system 2, the transmission determination unit 107 adds information indicating the recognizer transmitted from the learning data application determination unit 22 in the server system 2 to the sensor information to be transmitted. The sensor information determined not to be transmitted to the server system 2 by the transmission determination unit 107 is accumulated in the storage unit 105 by the accumulation unit 104.

[0030] According to this driving assistance system, in the terminal device 11, the transmission determination unit 107 determines whether or not to transmit the sensor information, so that it is possible to reduce communication costs related to the transmission of the sensor information. Also, in the server system 2, the learning data application determination unit 22 determines a recognizer to which the sensor information transmitted from the terminal device 11 is applied, so that the sensor information can be applied to a recognizer that performs recognition processing different from the recognizer included in the terminal device 11, and the sensor information can be used efficiently.

[0031] (1-3. Hardware configuration example) Next, an example of the hardware configuration of the server system 2 and the terminal device 11 applicable to each embodiment will be described. Fig. 3 is a block diagram showing an example of the hardware configuration of the terminal device 11 applicable to each embodiment. In Fig. 3, the terminal device 11 includes a CPU 1001, a ROM 1002, a RAM 1003, interfaces (I / F) 1004 and 1005, a communication interface 1006, and a GNSS receiver 1007, which are communicatively connected to each other via a bus 1030.

[0032] The CPU 1001 uses the RAM 1003 as a work memory in accordance with a program previously stored in the ROM 1002 to control the overall operation of the terminal device 11. The ROM 1002 is capable of rewriting stored data and programs under the control of the CPU 1001, and can store and update, for example, a program constituting the recognizer downloaded from the server system 2 described above. Alternatively, the RAM 1003 may include a non-volatile area, and the program constituting the recognizer may be stored in the non-volatile area of ​​the RAM 1003.

[0033] The interface 1004 is an interface for communicating with the camera 12, and image data of an image captured by the camera 12 and sensor information including the image data are input from the interface 1004 to the terminal device 11. The CPU 1001 can also send instructions such as image capture to the camera 12 via the interface 1004.

[0034] The interface 1005 is an interface for the control system of the vehicle 10 in which the terminal device 11 is mounted. Information such as the object recognition result by the recognizer is passed to the control system of the vehicle 10 via the interface 1005. The interface 1005 controls communication with the network 1 according to instructions from the CPU 1001.

[0035] The GNSS receiver 1007 receives signals from the Global Navigation Satellite System (GNSS) and acquires location information indicating the current location. The GNSS receiver 1007 can also acquire the current altitude and time along with the current location. If the vehicle 10 on which the terminal device 11 is mounted has a separate location information acquisition means, the GNSS receiver 1007 in the terminal device 11 can be omitted by acquiring the location information from the location information acquisition means.

[0036] In this manner, the terminal device 11 includes the CPU 1001, the ROM 1002, the RAM 1003, and various interfaces, and functions as a computer (information processing device).

[0037] The above-mentioned recognition unit 101, accumulation determination unit 103, accumulation unit 104, accumulated information output unit 106, transmission determination unit 107, and communication unit 110 included in terminal device 11 are realized by an information processing program that runs on CPU 1001. Without being limited to this, for example, some or all of accumulation determination unit 103, accumulation unit 104, accumulated information output unit 106, transmission determination unit 107, and communication unit 110 may be configured by hardware circuits that operate in cooperation with each other.

[0038] Fig. 4 is a block diagram showing an example of a hardware configuration of a server system 2 applicable to the embodiment. In Fig. 4, the server system 2 is shown configured as a single computer (information processing device). In Fig. 4, the server system 2 includes a CPU 2000, a ROM 2001, a RAM 2002, a storage device 2004, and a communication interface 2005, which are communicatively connected to each other via a bus 2010.

[0039] The CPU 2000 uses the RAM 2002 as a work memory in accordance with a program previously stored in the ROM 2001 or the storage device 2004 to control the overall operation of the server system 2. The program and each parameter constituting the recognizer generated by machine learning in accordance with the program by the CPU 2000 are stored in, for example, the storage device 2004.

[0040] The communication interface 2005 is an interface that controls communication with the network 1 .

[0041] The learning unit 3 and the learning data application determination unit 22 included in the server system 2 described above are realized by an information processing program that runs on the CPU 2000. However, without being limited thereto, a part or the whole of the learning unit 3 and the learning data application determination unit 22 may be configured by hardware circuits that operate in cooperation with each other.

[0042] [2. First embodiment] Next, a first embodiment of the present disclosure will be described. In each embodiment of the present disclosure, a recognizer that applies sensor information acquired in the terminal device 11 as learning data for re-learning can be specified from a plurality of recognizers. In the first embodiment, the recognizer is specified in the server system 2. FIG. 5 is a block diagram showing an example of a configuration of a driving assistance system according to the first embodiment. In the first embodiment, as shown by solid coloring in FIG. 5, the learning data application determination unit 108 on the terminal device 11 side shown in FIG. 2 is not used, but the learning data application determination unit 22a on the server system 2 side is used.

[0043] In the first embodiment, the scene at the time of capturing the image data included in the sensor information transmitted from the terminal device 11 is judged based on a predetermined rule or learning. Then, for example, for each judged scene, a recognizer to which the sensor information is applied as learning data is designated.

[0044] Here, the scene indicates the situation when sensing, i.e., capturing an image by the camera 12, and includes, for example, the location and environmental conditions where the image was captured. More specifically, the scene information indicating the scene may include the following information: Information obtained from outside the camera 12, such as the current position obtained by the GNSS receiver 1007 or the like, information indicating the current time, and the system status of the in-vehicle system installed in the vehicle 10. Internal sensor information such as exposure time and white balance processing when capturing an image in the camera 12. - Information estimated based on predetermined rules from sensed information (e.g. image data from captured images), or estimated through learning.

[0045] The learning data application determination unit 22a estimates an applicable recognizer by searching a rule base or a database based on the scene information acquired as described above. However, the learning data application determination unit 22a can also directly specify an applicable recognizer based on the scene information through learning.

[0046] Fig. 6 is a functional block diagram of an example for explaining functions of the server system 2 according to the first embodiment. In Fig. 6, the server system 2 includes a communication unit 200, a learning data application determination unit 22aa, a learning data accumulation unit 201, a learning unit 202, and a learning result utilization unit 203. The learning data application determination unit 22a includes a metadata analysis unit 221 and a learning data determination unit 222. The communication unit 200, the learning data application determination unit 22a (the metadata analysis unit 221 and the learning data determination unit 222), the learning data accumulation unit 201, the learning unit 202, and the learning result utilization unit 203 are realized by an information processing program running on a CPU 2000.

[0047] Here, the server system 2 has a plurality of recognizers 2101, 2102, 2103, 2104, .... In the following, when it is not necessary to distinguish between the recognizers 2101, 2102, 2103, 2104, ..., they will be represented by the recognizer 210 in the description. The learning data storage unit 201 stores learning data for generating the plurality of recognizers 210 for each recognizer 210.

[0048] The communication unit 200 controls the communication interface 2005 to receive, for example, sensor information transmitted from the terminal device 11. The metadata analysis unit 221 extracts metadata from the sensor information and analyzes the extracted metadata. The learning data determination unit 222 designates, based on the analysis result of the metadata, a recognizer 210 that can apply the sensor information among the multiple recognizers 210.

[0049] At this time, the learning data determination unit 222 can designate a recognizer different from the recognizer of the terminal device 11 that transmitted the sensor information as the recognizer 210 to which the sensor information can be applied. For example, the learning data determination unit 222 determines which of the above-mentioned scene information the sensor information corresponds to, and designates the recognizer 210 to which the sensor information can be applied according to the scene information to which the sensor information corresponds.

[0050] The learning data determination unit 222 adds information indicating, for example, the designated recognizer 210 to the sensor information and stores the information in the learning data storage unit 201 .

[0051] The learning unit 202 uses the learning data stored in the learning data storage unit 201 for each designated recognizer 210 to re-learn the recognizer 210, and updates the recognizer 210. The learning result utilization unit 203 performs utilization processing of the recognizer 210 that has been re-learned and updated by the learning unit 202. For example, the learning result utilization unit 203 transmits the updated recognizer 210 to the corresponding terminal device 11.

[0052] 7 is a flowchart of an example showing processing in the server system 2 according to the first embodiment. In a first step S100, the server system 2 performs communication processing by the communication unit 200 and receives sensor information transmitted from the terminal device 11 via the network 1. The processing in step S100 includes a process of receiving each piece of sensor information transmitted from each of the multiple terminal devices 11.

[0053] In the next step S101, the server system 2 acquires image data and metadata from each piece of sensor information acquired in the communication process of step S100 by the metadata analysis unit 221. In the next step S102, the server system 2 executes analysis processing of each piece of metadata acquired in step S101 by the metadata analysis unit 221.

[0054] The next step S103st indicates the start of a loop process between step S103ed and step S103st, which indicates that the process is executed for all the sensor information acquired in the communication process of step S100. The next step S104st indicates the start of a loop process between step S104ed and step S104st, which indicates that the process is executed for all the application destinations of each sensor information, that is, each recognizer 210.

[0055] In step S105, the server system 2 executes a metadata application determination process by the learning data determination unit 222. That is, in step S105, the learning data determination unit 222 checks whether one piece of sensor information (assumed to be target sensor information) among the pieces of sensor information acquired in the communication process in step S100 is applicable to the target recognizer 210 among the recognizers 210, based on the metadata.

[0056] In the next step S106, if the learning data determination unit 222 determines that the target sensor information is applicable to the target recognizer 210 (step S106, "Yes"), the process proceeds to step S107. In step S107, the server system 2 causes the learning data accumulation unit 201 to execute accumulation processing of the learning data to be applied. That is, in step S107, the learning data accumulation unit 201 accumulates the target sensor information as learning data for the target recognizer 210 determined to be applicable in step S106.

[0057] After the process of step S107, the server system 2 returns the process from step S104ed to step S104st, and the processes of steps S105 and S106 are executed with the next recognizer 210 of the recognizers 210 as the target recognizer 210.

[0058] On the other hand, if it is determined in step S106 that the target sensor information is not applicable to the target recognizer 210 (step S106, "No"), the process returns from step S104ed to step S104st.

[0059] When the processing of steps S104st to S104ed for each recognizer 210 is completed for the target sensor information, the server system 2 returns the processing from step S103ed to step S103st, and repeats the processing of steps S103st to S103ed with the next sensor information among the sensor information acquired in the communication processing of step S100 as the target sensor information.

[0060] When the processing of steps S103st to S103ed for each piece of sensor information is completed, the server system 2 shifts the processing to step S108. In step S108, the server system 2 performs re-learning processing of the corresponding recognizers 210 by the learning unit 202 using each piece of sensor information stored in the learning data storage unit 201 by the above-mentioned processing as learning data. When the processing of step S108 is completed, the server system 2 shifts the processing to step S109, and executes a utilization processing of each recognizer 210 re-learned in step S108 by the learning result utilization unit 203.

[0061] In this way, in the first embodiment, each piece of sensor information transmitted from each terminal device 11 is applied as learning data for the recognizer 210 that is made applicable based on the metadata. Therefore, each piece of sensor information transmitted from each terminal device 11 can be applied as learning data to a recognizer different from the recognizer used for the sensor information by the terminal device 11 that transmitted the sensor information, and the sensor information can be used more efficiently as learning data.

[0062] (2-1. First modified example) Next, a first modified example of the first embodiment will be described. In the first modified example of the first embodiment, when the recognizer is designated by the server system 2, the server system 2 designates, from among a plurality of recognizers 210 possessed by the server system 2, a recognizer 210 capable of converting the sensor information transmitted from the terminal device 11 in an applicable manner as a recognizer that applies the sensor information as learning data. In the first modified example of the first embodiment, the configuration of the driving assistance system according to the first embodiment described using FIG. 5 can be applied.

[0063] Fig. 8 is a functional block diagram of an example for explaining the function of the server system 2 according to the first modification of the first embodiment. In the first modification of the first embodiment, as shown in Fig. 8, the learning data application determination unit 108 on the terminal device 11 side shown in Fig. 2 is not used, but the learning data application determination unit 22b on the server system 2 side is used. Also, unlike the configuration of the learning data application determination unit 22a of the first embodiment described above, the learning data application determination unit 22b includes a domain analysis unit 230, a domain evaluation unit 231, and a learning data determination unit 232.

[0064] In a first modified example of the first embodiment, the server system 2 converts the domain of the sensor information using the applicability of domain adaptation. Here, a domain indicates an area (category) for limiting a target, such as a region (country, etc.) or an environment (day / night, highway / general road, etc.). Domain adaptation is a technique for converting a target area (object, country, task, ...) of a recognizer. As an example, domain adaptation enables a recognizer trained on Japanese data to recognize on American data. In this case, two domains, Japan and America, are converted.

[0065] That is, the learning data application determination unit 22b according to the first modification of the first embodiment hits the domain combination rule for which domain adaptation is possible for the target area of ​​the recognizer 210. As a specific example, consider N domains, domain #1, domain #2, domain #3, ..., domain #N. In these N domains, it is assumed that there are, for example, the following combinations of convertible domains represented by a directed graph: · Domain #1 → Domain #2, Domain #5, Domain #7, … Domain #2 → Domain #1, Domain #3, …

[0066] Let S_{ij} be the number of steps that indicate how many steps it can take to convert domain #i to domain #j when domain #i is input. Let α (<1) be the evaluation value per hop, and β_{ij} (<1) be the evaluation value when domain #i is converted to domain #j.

[0067] The evaluation value Ev of the input sensor information from domain #a to domain #b is calculated by the following equation (1). Ev=α (S_{ij}) +β_{ij} …(1)

[0068] A threshold value is set for this evaluation value Ev, and the recognizers 210 whose evaluation value Ev is higher than the threshold value are set as applicable recognizers 210.

[0069] The domain analysis unit 230 analyzes the sensor information received by the communication unit 200 from the terminal device 11, and determines the domain related to the sensor information. The domain analysis unit 230 may determine the domain from image data included in the sensor information, or may determine the domain from metadata. The domain evaluation unit 231 calculates an evaluation value Ev for each recognizer 210 included in the server system 2, for example, by the above-mentioned formula (1), based on the domain determined by the domain analysis unit 230.

[0070] The learning data determination unit 232 compares each evaluation value Ev calculated by the domain evaluation unit 231 with a threshold value, and designates a recognizer 210 whose evaluation value Ev is higher than the threshold value as a recognizer 210 to which the sensor information is applicable.

[0071] 9 is a flowchart of an example showing processing in the server system 2 according to the first modified example of the first embodiment. In a first step S200, the server system 2 performs communication processing by the communication unit 200 and receives sensor information transmitted from the terminal device 11 via the network 1. The processing in step S200 includes a process of receiving each piece of sensor information transmitted from each of the multiple terminal devices 11.

[0072] In the next step S201, in the server system 2, the domain analysis unit 230 acquires the sensor information received in the communication processing of step S200, and in the next step S202, analyzes the sensor information acquired in step S201 to determine the domain associated with the sensor information.

[0073] The next step S203st indicates the start of a loop process between step S203ed and step S203st, which is executed for all sensor information acquired in the communication process of step S200. The next step S204st indicates the start of a loop process between step S204ed and step S204st, which is executed for all application destinations of each sensor information, that is, each domain conversion destination.

[0074] In step S205, in the server system 2, the domain analysis unit 230 calculates a conversion path from the domain obtained from the target sensor information for the target domain as the conversion destination. In the next step S206, in the server system 2, the domain evaluation unit 231 calculates an evaluation value Ev for the conversion path of the domain calculated in step S205 according to the above-mentioned formula (1).

[0075] In the next step S207, the learning data determination unit 232 determines whether or not the evaluation value Ev calculated in step S206 exceeds a threshold value.

[0076] If the learning data determination unit 232 determines in step S207 that the evaluation value Ev exceeds the threshold value (step S207, "Yes"), the process proceeds to step S208. In step S208, the learning data accumulation unit 201 in the server system 2 executes accumulation processing of the learning data to be applied. That is, in step S208, the learning data accumulation unit 201 accumulates the target sensor information as learning data for the target recognizer 210 determined to be applicable in step S206.

[0077] After the process of step S208, the server system 2 returns the process from step S204ed to step S204st, and the processes of steps S205 to S207 are executed with the next domain among the domains as the target domain for conversion.

[0078] On the other hand, if it is determined in step S207 that the evaluation value Ev is equal to or less than the threshold value (step S207, "No"), the learning data determination unit 232 returns the process from step S204ed to step S204st.

[0079] When the processing of steps S204st to S204ed for each domain is completed for the target sensor information, the server system 2 returns the processing from step S203ed to step S203st, and repeats the processing of steps S203st to S203ed with the next sensor information among the sensor information acquired in the communication processing of step S200 as the target sensor information.

[0080] When the processing of steps S203st to S203ed for each piece of sensor information is completed, the server system 2 shifts the processing to step S209. In step S209, the server system 2 performs re-learning processing of the corresponding recognizers 210 by the learning unit 202 using each piece of sensor information stored in the learning data storage unit 201 by the above-mentioned processing as learning data. When the processing of step S208 is completed, the server system 2 shifts the processing to step S210, and executes a utilization processing of each recognizer 210 re-learned in step S209 by the learning result utilization unit 203.

[0081] In this way, according to the first modification of the first embodiment, the sensor information transmitted from the terminal device 11 can be applied to a domain different from the domain when the sensor information was acquired. This enables more efficient use of the sensor information.

[0082] (2-2. Second Modification) Next, a second modified example of the first embodiment will be described. In the second modified example of the first embodiment, when the server system 2 designates a recognizer, the server system 2 designates a recognizer to which the sensor information transmitted from the terminal device 11 is applied as learning data, from among the multiple recognizers 210 possessed by the server system 2, according to an object contained in image data possessed by the sensor information. In addition, in the second modified example of the first embodiment, the configuration of the driving assistance system according to the first embodiment described using FIG. 5 can be applied.

[0083] Fig. 10 is a functional block diagram of an example for explaining the function of the server system 2 according to the second modified example of the first embodiment. In the second modified example of the first embodiment, as shown in Fig. 10, the learning data application determination unit 108 on the terminal device 11 side shown in Fig. 2 is not used, but a learning data application determination unit 22c on the server system 2 side is used. Also, unlike the configuration of the learning data application determination unit 22a of the first embodiment described above, the learning data application determination unit 22c includes a sensing analysis unit 240 and a learning data determination unit 241.

[0084] 10, the sensing analysis unit 240 analyzes image data included in the sensor information received by the communication unit 200. Based on the information analyzed by the sensing analysis unit 240, the learning data determination unit 241 designates a recognizer 210 among the multiple recognizers 210 to which the sensor information can be applied.

[0085] A more detailed description will be given. In the second modified example of the first embodiment, image data included in the sensor information transmitted from the terminal device 11 is analyzed to determine objects and object classes included in the image data. Then, for each determined object and object class, a database is used to search whether the sensor information is applicable as learning data. As the contents of the database, pairs of candidate recognizers and objects can be applied. In this case, the contents of the database are not limited to a positive list in which applicable objects are listed, but may be a negative list (list of objects that cannot be applied), or a combination of a positive list and a negative list.

[0086] As an example of a database for a recognizer applicable to the second modified example of the first embodiment, for example, the following is considered. -Sign recognition device: When used in Japan, the negative list of signs from other countries Pedestrian detector: When used in Japan, the pedestrian positive list

[0087] Fig. 11 is a schematic diagram showing an example of an image 50 based on image data included in the sensor information. The image 50 shown in Fig. 11 includes a pedestrian 51, a traffic road sign 52, and a vehicle 53 as objects.

[0088] The sensing analysis unit 240 analyzes this image 50 to detect each object (pedestrian 51, traffic road sign 52, and vehicle 53). Here, consider a case where the detected pedestrian 51 and traffic road sign 52 are from the United States for a recognizer used in Japan. Based on the analysis result of the sensing analysis unit 240, the learning data determination unit 241 does not select the sign recognizer as a recognizer to be applied to the sensor information because a foreign country's sign is present in the image 50. On the other hand, because a pedestrian is present in the image 50, the learning data determination unit 241 selects the pedestrian detector as a recognizer to be applied to the sensor information.

[0089] 12 is a flowchart of an example showing processing in the server system 2 according to the second modified example of the first embodiment. In a first step S300, the server system 2 performs communication processing by the communication unit 200 and receives sensor information transmitted from the terminal device 11 via the network 1. The processing in step S300 includes a process of receiving each piece of sensor information transmitted from each of the multiple terminal devices 11.

[0090] In the next step S301, in the server system 2, the sensing analysis unit 240 acquires the sensor information received in the communication processing in step S300, and in the next step S302, analyzes the sensor information acquired in step S301 to determine objects and object classes included in the image data held by the sensor information. The following will explain the objects and object classes analyzed from the image data by taking objects as an example.

[0091] The next step S303st indicates the start of a loop process between step S303ed and step S303st, which is executed for all the sensor information acquired in the communication process of step S300. The next step S304st indicates the start of a loop process between step S304ed and step S304st, which is executed for all the application destinations of each sensor information, that is, each recognizer 210.

[0092] In step S305, in the server system 2, the learning data determination unit 241 checks whether the sensor information is applicable to the target recognizer 210 among the recognizers 210, based on each object contained in the image data of the target sensor information in step S302.

[0093] If the learning data determination unit 241 determines in step S306 that the target sensor information is applicable to the target recognizer 210 (step S306, "Yes"), the process proceeds to step S307. In step S307, the learning data accumulation unit 201 in the server system 2 executes accumulation processing of the learning data to be applied. That is, in step S307, the learning data accumulation unit 201 accumulates the target sensor information as learning data for the target recognizer 210 determined to be applicable in step S306.

[0094] After the process of step S307, the server system 2 returns the process from step S304ed to step S304st, and the processes of steps S305 and S306 are executed with the next recognizer 210 of the recognizers 210 as the target recognizer 210.

[0095] On the other hand, if the learning data determination unit 232 determines in step S306 that the target sensor information is not applicable to the target recognizer 210 (step S306, "No"), the process returns from step S304ed to step S304st.

[0096] When the processing of steps S304st to S304ed for each recognizer 210 is completed for the target sensor information, the server system 2 returns the processing from step S303ed to step S303st, and repeats the processing of steps S303st to S303ed with the next sensor information among the sensor information acquired in the communication processing of step S300 as the target sensor information.

[0097] When the processes of steps S303st to S303ed for each piece of sensor information are completed, the server system 2 shifts the process to step S308. In step S308, the server system 2 uses each piece of sensor information stored in the learning data storage unit 201 by the above-mentioned processes as learning data, and performs re-learning process of each corresponding recognizer 210 by the learning unit 202. When the process of step S308 is completed, the server system 2 shifts the process to step S309, and executes a utilization process of each recognizer 210 re-learned in step S308 by the learning result utilization unit 203.

[0098] In this way, in the second modification of the first embodiment, the recognizer 210 to which the sensor information can be applied is selected based on the object or object class included in the image data of the sensor information transmitted from the terminal device 11. This enables more efficient use of the sensor information.

[0099] [3. Second embodiment] Next, a second embodiment of the present disclosure will be described. In the above-mentioned first embodiment and each of its modified examples, the recognizer 210 to which the sensor information transmitted from the terminal device 11 is applied is specified on the server system 2 side. In contrast, in the second embodiment, the terminal device 11 specifies the recognizer 210 to which the sensor information to be transmitted is applied. Furthermore, in the second embodiment, the terminal device 11 determines whether or not to transmit the acquired sensor information to the server system 2.

[0100] Fig. 13 is a block diagram showing an example of a configuration of a driving support system according to the second embodiment. In the driving support system shown in Fig. 13, the learning data application determination unit 108 shown in Fig. 2 is used in the terminal device 11. In this case, for example, the learning data application determination unit 22 on the server system 2 side transmits information indicating the recognizer 210 possessed by the server system 2 to the learning data application determination unit 108 on the terminal device 11 side. In the second embodiment and each modified example described later, the transmission determination unit 107a plays a central role, as shown by filling in the figure.

[0101] The learning data application determination unit 108 determines, for example, in the same manner as in the first embodiment described above, based on metadata contained in the sensor information acquired from the camera 12, to which of the recognizers 210 contained in the server system 2 the sensor information should be applied. The transmission determination unit 107a adds information indicating the recognizer 210 to which the learning data application determination unit 108 has determined that the sensor information should be applied, to the sensor information to be transmitted to the server system 2, and transmits the information to the server system 2. At this time, the transmission determination unit 107a determines whether or not to transmit the sensor information to the server system 2, based on, for example, metadata contained in the sensor information. This enables efficient use of the sensor information and reduces communication costs for transmitting the sensor information.

[0102] Fig. 14 is a functional block diagram of an example for explaining functions of the terminal device 11 according to the second embodiment. In Fig. 14, the terminal device 11 includes an imaging unit 300, a recognition unit 101, a recognition result utilization unit 301, a learning data application determination unit 108a, a transmission determination unit 107a, and a communication unit 110. The imaging unit 300, the recognition unit 101, the learning data application determination unit 108a, the transmission determination unit 107a, and the communication unit 110 are realized by an information processing program that runs on a CPU 1001.

[0103] The imaging unit 300 controls the camera 12 to acquire image data of the captured image and metadata related to the imaging. The imaging unit 300 outputs the acquired image data and metadata as sensor information. The recognition unit 101 executes object recognition processing on the image data included in the sensor information output from the imaging unit 300, for example, using the recognizer 210 acquired from the server system 2 via the network 1. The recognition result utilization unit 301 is included in, for example, a control system of the vehicle 10 on which the terminal device 11 is mounted, and performs control such as obstacle avoidance according to the result of the object recognition processing.

[0104] The metadata analysis unit 250 and the learning data determination unit 222 have functions equivalent to those of the metadata analysis unit 221 and the learning data determination unit 222 described in the first embodiment, for example. For example, the metadata analysis unit 250 extracts metadata from the sensor information and analyzes the extracted metadata. The learning data determination unit 251 designates a recognizer 210 that can apply the sensor information among the multiple recognizers 210 included in the server system 2 based on the analysis result of the metadata.

[0105] The transmission determination unit 107a determines whether or not to transmit the sensor information acquired by the imaging unit 300 to the server system 2. For example, the transmission determination unit 107a determines whether or not to transmit the sensor information based on the sensor information. However, the transmission determination unit 107a can also determine whether or not to transmit the sensor information based on the recognition result of the recognition unit 101 based on the sensor information.

[0106] The communication unit 110 transmits the sensor information, which is determined to be transmitted by the transmission determination unit 107a, to the server system 2 via the network 1.

[0107] 15 is a flowchart of an example of processing in the terminal device 11 according to the second embodiment. In a first step S400, the terminal device 11 performs imaging processing using the imaging unit 300, and receives sensor information including image data of the captured image and metadata related to imaging by the camera 12 from the camera 12.

[0108] In the next step S401, the terminal device 11 performs object recognition processing based on the sensor information acquired in the imaging processing in step S400 by the recognition unit 101. In the next step S402, the terminal device 11 performs processing using the recognition result by the object recognition processing of the recognition unit 101 by the recognition result utilization unit 301.

[0109] In the next step S403, the metadata analysis unit 250 acquires metadata from the sensor information acquired in the imaging process in step S400, and executes an analysis process of the acquired metadata.

[0110] The next step S404st indicates the start of a loop process between step S404ed and all application destinations of the sensor information, that is, each recognizer 210 included in the server system 2.

[0111] In step S405, the terminal device 11 executes a metadata application determination process by the learning data determination unit 251. That is, in step S405, the learning data determination unit 251 checks whether the sensor information acquired in the imaging process in step S400 is applicable to the target recognizer 210 among the recognizers 210 based on the metadata. Here, the learning data determination unit 251 can apply the determination method described in the first embodiment. That is, the learning data determination unit 251 determines the scene at the time of capturing the image data included in the sensor information based on a predetermined rule or learning, and, for example, for each determined scene, specifies the recognizer 210 to which the sensor information is applied as learning data.

[0112] In the next step S406, if the learning data determination unit 251 determines that the sensor information is applicable to the target recognizer 210 (step S406, "Yes"), the process proceeds to step S407. In step S407, the accumulation determination unit 103 of the terminal device 11 executes accumulation processing of the learning data to which the sensor information is to be applied. That is, in step S407, the accumulation determination unit 103 determines to accumulate the sensor information as learning data for the target recognizer 210 determined to be applicable in step S406. The accumulation unit 104 stores and accumulates the sensor information in the storage unit 105 according to this determination.

[0113] After the process of step S407, the terminal device 11 returns the process from step S404ed to step S404st, and the processes of steps S405 to S407 are executed with the next recognizer 210 of the recognizers 210 as the target recognizer 210.

[0114] On the other hand, if it is determined in step S406 that the target sensor information is not applicable to the target recognizer 210 (step S406, "No"), the process returns from step S404ed to step S404st.

[0115] When the processing of steps S404st to S404ed for each recognizer 210 for the sensor information is completed, the terminal device 11 shifts the processing to step S408. In step S408, the terminal device 11 determines by the transmission determination unit 107a whether the number of recognizers 210 determined as application destinations of the sensor information in the processing of steps S404st to S404ed is sufficient (for example, whether the number exceeds a threshold value). When the transmission determination unit 107a determines that the number of recognizers 210 determined as application destinations of the sensor information is not sufficient, the transmission determination unit 107a does not transmit the sensor information to the server system 2, and ends the series of processing according to the flowchart of FIG. 15.

[0116] On the other hand, if the transmission determination unit 107a determines in step S408 that the number of recognizers 210 determined as the application destination of the sensor information is sufficient, the transmission determination unit 107a shifts the process to step S409. In step S409, the transmission determination unit 107a performs a transmission process of the sensor information. For example, the transmission determination unit 107a adds information indicating each recognizer 210 determined to be applicable by the processes of steps S404st to S404ed to the sensor information stored in the storage unit 105 in step S407.

[0117] In the next step S410, the sensor information to which the information indicating each recognizer 210 is added is read from the storage unit 105, and is transmitted by the communication unit 110 to the server system 2 via the network 1. The server system 2 stores the sensor information transmitted from the terminal device 11 in the storage unit 20 in association with each recognizer 210 as learning data for re-learning each recognizer 210 based on the information indicating each recognizer 210 added to the sensor information.

[0118] As described above, in the second embodiment, the sensor information including image data and metadata of a captured image output from the camera 12 is determined based on the metadata as to whether it is applicable to each recognizer 210 included in the server system 2. Therefore, the terminal device 11 can apply the sensor information acquired by the camera 12 as learning data to a recognizer different from the recognizer used by the terminal device 11 for the sensor information, and the sensor information can be used more efficiently as learning data. In addition, the terminal device 11 determines by the transmission determination unit 107a whether to transmit the sensor information to the server system 2 based on, for example, metadata included in the sensor information. This makes it possible to reduce communication costs for transmitting the sensor information.

[0119] In the above description, the transmission determination unit 107a determines whether or not to transmit the sensor information to the server system 2 based on the metadata contained in the sensor information, but this is not limited to this example. For example, the transmission determination unit 107a may determine whether or not to transmit the sensor information to the server system 2 based on a recognition score included in the recognition result of the recognition unit 101 for the image data including the sensor information. In this case, for example, the transmission determination unit 107a may transmit the sensor information to the server system 2 when the recognition score is equal to or lower than a predetermined value. By performing learning in the server system 2 based on sensor information with such a low recognition score, it is possible to further improve the recognition performance.

[0120] (3-1. First modified example) Next, a first modified example of the second embodiment will be described. The first modified example of the second embodiment is an example in which the terminal device 11 determines whether or not to transmit acquired sensor information to the server system 2 based on the immediacy of the sensor information. Note that the first modified example of the second embodiment can apply the configuration of the driving assistance system according to the first embodiment described with reference to FIG. 13.

[0121] A more detailed explanation will be given. The sensor information acquired by the camera 12 may simultaneously include information with significantly different orders of time constants. As an example, the time from when the information on lens aging correction included in the metadata of the sensor information is generated until when a response is required is significantly different from the time when the information is generated until when a response is required for pedestrian recognition processing performed based on image data. Therefore, in the first modified example of the second embodiment, the necessity of immediate transmission for each purpose of use is determined for the target sensor information, and the sensor information required immediately is transmitted to the server system 2 with priority. The sensor information with low immediacy is stored and not transmitted at the time of acquisition.

[0122] Fig. 16 is a functional block diagram of an example for explaining functions of a terminal device 11 according to a first modified example of the second embodiment. In Fig. 16, the terminal device 11 has a learning data immediacy calculation unit 152 and a learning data immediacy determination unit 153 inserted between the learning data application determination unit 108a and the transmission determination unit 107a in the configuration of the terminal device 11 according to the second embodiment described using Fig. 14.

[0123] The learning data immediacy calculation unit 151 calculates the immediacy of the sensor information specified by the learning data determination unit 251 to be applied to the recognizer 210. For example, the learning data immediacy calculation unit 151 calculates the immediacy of the sensor information according to the intended use of the recognizer 210 specified to be applied to the sensor information by the learning data determination unit 251. The learning data immediacy determination unit 153 determines the immediacy of the sensor information according to the calculated immediacy.

[0124] In the above example, when the recognizer 210 performs recognition related to lens aging correction and the span of the change in the recognition content is long, the learning data immediacy calculation unit 152 obtains the timing of the next correction based on, for example, the metadata of the sensor information, and calculates the difference between this timing and the current time. When the calculated difference is longer than a threshold, the learning data immediacy determination unit 153 determines that the sensor information has low immediacy. In another example described above, when the recognizer 210 performs pedestrian recognition, the learning data immediacy determination unit 153 determines that the sensor information has high immediacy.

[0125] The transmission determination unit 107b determines whether or not to transmit the sensor information to the server system 2, depending on the immediacy determined by the learning data immediacy determination unit 153. That is, the transmission determination unit 107b determines that the sensor information determined to have high immediacy should be transmitted to the server system 2 at that time. On the other hand, the transmission determination unit 107b determines that the sensor information determined to have low immediacy does not need to be transmitted at that time, and stores the sensor information in the storage unit 105 by the storage unit 104, for example.

[0126] The communication unit 110 immediately transmits to the server system 2 the sensor information that is determined to be transmitted by the transmission determination unit 107b.

[0127] 17 is a flowchart of an example of processing in the terminal device 11 according to the first modified example of the second embodiment. In a first step S500, the terminal device 11 performs imaging processing using the imaging unit 300, and acquires sensor information including image data of the captured image from the camera 12 and metadata related to the imaging by the camera 12.

[0128] In the next step S501, the terminal device 11 performs object recognition processing based on the sensor information acquired in the imaging processing in step S500 by the recognition unit 101. In the next step S502, the terminal device 11 performs processing using the recognition result by the object recognition processing of the recognition unit 101 by the recognition result utilization unit 301.

[0129] In the next step S503, the metadata analysis unit 250 acquires metadata from the sensor information acquired in the imaging process in step S500, and executes an analysis process of the acquired metadata.

[0130] The next step S504st indicates the start of a loop process between step S504ed and all application destinations of the sensor information, that is, each recognizer 210 included in the server system 2.

[0131] In step S505, the terminal device 11 executes a metadata application determination process by the learning data determination unit 251. That is, in step S505, the learning data determination unit 251 checks whether the sensor information acquired in the imaging process in step S500 is applicable to the target recognizer 210 among the recognizers 210 based on the metadata. Here, the learning data determination unit 251 can apply the determination method described in the first embodiment. That is, the learning data determination unit 251 determines the scene at the time of capturing the image data included in the sensor information based on a predetermined rule or learning, and, for example, for each determined scene, specifies the recognizer 210 to which the sensor information is applied as learning data.

[0132] In the next step S506, if the learning data determination unit 251 determines that the sensor information is applicable to the target recognizer 210 (step S506, "Yes"), the process proceeds to step S507. In step S507, the accumulation determination unit 103 of the terminal device 11 executes accumulation processing of the learning data to which the sensor information is to be applied. That is, in step S507, the accumulation determination unit 103 determines to accumulate the sensor information as learning data for the target recognizer 210 determined to be applicable in step S506. The accumulation unit 104 stores and accumulates the sensor information in the storage unit 105 according to this determination.

[0133] After the process of step S507, the terminal device 11 returns the process from step S504ed to step S504st, and the processes of steps S505 to S507 are executed with the next recognizer 210 of the recognizers 210 as the target recognizer 210.

[0134] On the other hand, if it is determined in step S506 that the target sensor information is not applicable to the target recognizer 210 (step S506, "No"), the process returns from step S504ed to step S504st.

[0135] When the processes of steps S504st to S504ed for each recognizer 210 for the sensor information are completed, the terminal device 11 shifts the process to step S508. In step S508, the terminal device 11 calculates the immediacy of the sensor information by the learning data immediacy calculation unit 152, and in the next step S509, the learning data immediacy determination unit 153 determines the immediacy of the sensor information.

[0136] In the next step S510, the terminal device 11 determines, based on the immediacy determined in step S509, by the transmission determination unit 107b, whether or not to transmit the sensor information to the server system 2. If the immediacy of the sensor information is determined to be low, the transmission determination unit 107b determines not to transmit the sensor information to the server system 2 (step S510, "No"), and ends the series of processes in the flowchart of FIG.

[0137] On the other hand, in step S510, if it is determined that the sensor information has high immediacy, the transmission determination unit 107b determines that the sensor information is to be transmitted to the server system 2 (step S510, "Yes"), and moves the process to step S511. In step S511, the transmission determination unit 107b performs a transmission process of the sensor information. For example, the transmission determination unit 107b adds, to the sensor information, information indicating the recognizer 210 determined to be applicable by the processes of steps S504st to S504ed.

[0138] In the next step S512, the sensor information to which the information indicating each recognizer 210 is added is transmitted by the communication unit 110 to the server system 2 via the network 1. The server system 2 stores the sensor information transmitted from the terminal device 11 in the storage unit 20 in association with the recognizer 210 as learning data for re-learning the recognizer 210 based on the information indicating the recognizer 210 added to the sensor information.

[0139] In this way, in the first modified example of the second embodiment, based on the immediacy of the sensor information, it is determined whether or not to transmit the sensor information to the server system 2. Therefore, it is possible to reduce the communication cost for transmitting the sensor information.

[0140] (3-2. Second modified example) Next, a second modified example of the second embodiment will be described. The second modified example of the second embodiment is an example in which the terminal device 11 determines whether or not to transmit acquired sensor information to the server system 2 based on the rarity of the sensor information. Note that the configuration of the driving assistance system according to the first embodiment described with reference to FIG. 13 can be applied to the second modified example of the second embodiment.

[0141] A more specific description will be given. Some sensor information acquired by the camera 12 is scarce and needs to be collected early. For example, it is preferable to reflect sensor information related to newly introduced roads and traffic systems in the learning data as early as possible. In order to collect such sensor information, an area where a situation that needs to be collected early occurs is set in advance or at any time by communication such as V2X, and the sensor information acquired within the area is determined to be information that should be transmitted to the server system 2.

[0142] Fig. 18 is a functional block diagram of an example for explaining the function of the terminal device 11 according to the second modified example of the second embodiment. In Fig. 18, the terminal device 11 omits the learning data application determination unit 108a from the configuration of the terminal device 11 according to the second embodiment described using Fig. 14, and adds a rarity analysis unit 310 and a rarity determination unit 311.

[0143] The sensor information acquired by the imaging unit 300 is input to the transmission determination unit 107e via the recognition unit 101. On the other hand, the rarity analysis unit 310 analyzes the rarity of the event in the range imaged by the camera 12. The rarity determination unit 311 determines the rarity of the event analyzed by the rarity analysis unit 310.

[0144] For example, the rarity analysis unit 310 performs a process of setting an area including a rare event by communication with the outside, user setting, etc. Furthermore, the rarity analysis unit 310 acquires location information indicating the current position from, for example, the GNSS receiver 1007 provided in the terminal device 11. The rarity determination unit 311 determines whether or not the imaging range captured by the camera 12 includes the set area based on the analysis result of the rarity analysis unit 310. For example, based on the location information indicating the current position acquired by the rarity analysis unit 310 and the set area, the rarity determination unit 311 determines that the sensor information acquired from the camera 12 has rarity when the acquired location information corresponds to the set area.

[0145] The transmission determination unit 107c determines whether or not to transmit the sensor information acquired from the camera 12 to the server system 2 according to the determination result of the rarity determination unit 311. The communication unit 110 transmits the sensor information determined to be transmitted by the transmission determination unit 107c to the server system 2 via the network 1.

[0146] 19 is a flowchart of an example of processing in the terminal device 11 according to the second modified example of the second embodiment. In a first step S600, the terminal device 11 performs imaging processing using the imaging unit 300, and receives sensor information including image data of the captured image and metadata related to imaging by the camera 12 from the camera 12.

[0147] In the next step S601, the terminal device 11 performs object recognition processing based on the sensor information acquired in the imaging processing in step S600 by the recognition unit 101. In the next step S602, the terminal device 11 performs processing using the recognition result by the object recognition processing of the recognition unit 101 by the recognition result utilization unit 301.

[0148] In the next step S603, the rarity analysis unit 310 acquires metadata from the sensor information acquired in the imaging process of the set step S600, and executes an analysis process of the acquired metadata. In the next step S604, the rarity analysis unit 310 performs rarity analysis. For example, the rarity analysis unit 310 acquires location information indicating the current location and information on the set area.

[0149] In the next step S605, the rarity determination unit 311 determines whether or not the sensor information is applicable to the recognizer 210 that has learned using rare sensor information, based on the analysis result in step S604. For example, based on position information indicating the current position and information indicating a set area, the rarity determination unit 311 can determine that the sensor information acquired from the camera 12 is rare sensor information when the current position is a position corresponding to the area.

[0150] In the next step S606, if the rarity determination unit 311 determines that the sensor information is applicable to the recognizer 210 that learned using rare sensor information (step S606, "Yes"), the process proceeds to step S607. In step S607, the terminal device 11 performs a transmission process of the sensor information by the transmission determination unit 107c. For example, the transmission determination unit 107c adds information indicating that the sensor information is rare to the sensor information. The sensor information to which the information has been added is transmitted to the server system 2 by the communication unit 110 via the network 1.

[0151] On the other hand, if it is determined in step S606 that the sensor information is not applicable to the recognizer 210 that has been trained using rare sensor information, the sensor information is not transmitted to the server system 2, and the series of processes according to the flowchart in FIG. 19 is terminated.

[0152] In this way, in the second modified example of the second embodiment, based on the rarity of the sensor information, it is determined whether or not to transmit the sensor information to the server system 2. Therefore, it is possible to reduce the communication cost for transmitting the sensor information.

[0153] (3-3. Third modified example) Next, a third modified example of the second embodiment will be described. The third modified example of the second embodiment is an example that makes it possible to prevent duplicated sensor information from being transmitted from a plurality of vehicles 10. Note that, in the third modified example of the second embodiment, the configuration of the driving assistance system according to the first embodiment described with reference to FIG. 13 can be applied.

[0154] That is, when multiple vehicles 10 having a function of transmitting sensor information exist, for example, in relatively close proximity, there is a risk that the multiple vehicles 10 transmit similar sensor information to the server system 2. For this reason, in the third modified example of the second embodiment, multiple vehicles 10 existing in close proximity are allowed to transmit different data to the server system 2. More specifically, in the third modified example of the second embodiment, a vehicle 10 of interest (hereinafter, a vehicle of interest) transmits and receives metadata of the sensor information to be transmitted between the vehicles 10 around it, and prevents the vehicle of interest from transmitting sensor information similar to sensor information already transmitted by other vehicles 10.

[0155] Fig. 20 is a functional block diagram of an example for explaining functions of the terminal device 11 according to the third modified example of the second embodiment. In Fig. 20, the terminal device 11 of the attention vehicle includes an imaging unit 300, a recognition unit 101, a recognition result utilization unit 301, a transmission determination unit 107d, a communication unit 110a, a different vehicle data accumulation unit 320, and a storage unit 321. In addition, the other vehicles 101, 102, 103, ... with respect to the attention vehicle each include an imaging unit 300, a recognition unit 101, a recognition result utilization unit 301, a transmission determination unit 107, and a communication unit 110. Not limited to this, each of the other vehicles 101, 102, 103, ... may have the same configuration as the attention vehicle.

[0156] The imaging unit 300, the recognition unit 101, and the recognition result utilization unit 301 have functions equivalent to those of the imaging unit 300, the recognition unit 101, and the recognition result utilization unit 301 shown in FIG. 14, for example, and therefore will not be described here.

[0157] 20, the communication unit 110a communicates with the communication units 110 of the other vehicles 101, 102, 103, . . . and obtains information indicating the sensor information that has already been transmitted to the server system 2 by the other vehicles 101, 102, 103, .

[0158] For example, in the target vehicle, the recognition unit 101 performs object recognition processing on image data included in the sensor information acquired from the camera 12, and passes the recognition result to the transmission determination unit 107d. Similarly, in each of the vehicles 101, 102, 103, ..., the recognition unit 101 performs object recognition processing on image data included in the sensor information acquired from the camera 12, and passes the recognition result to each transmission determination unit 107d.

[0159] The communication unit 110a communicates with each of the vehicles 101, 102, 103, ..., and requests information indicating sensor information to be transmitted to each of the vehicles 101, 102, 103, .... In response to this request, the transmission determination unit 107 of each of the vehicles 101, 102, 103, ... transmits, for example, metadata included in the sensor information and the recognition result to the vehicle of interest via the network 1.

[0160] In the terminal device 11 mounted on the target vehicle, the communication unit 110a passes the metadata and the recognition results received from each of the vehicles 101, 102, 103, ... to the other vehicle data accumulation unit 320. The other vehicle data accumulation unit 320 accumulates the passed metadata and the recognition results in the storage unit 321.

[0161] Based on the recognition result passed from the recognition unit 101 and the metadata included in the sensor information, the transmission determination unit 107d determines whether or not data corresponding to the recognition result and metadata that has already been transmitted is stored in the storage unit 321. If the recognition result and metadata have not been stored, the transmission determination unit 107d transmits the recognition result and metadata to the server system 2 via the communication unit 110a. On the other hand, if the recognition result and metadata have been stored, the transmission determination unit 107d does not transmit the recognition result and metadata.

[0162] In this way, by avoiding the transmission of duplicated sensor information between a plurality of vehicles 10, communication costs can be reduced.

[0163] Fig. 21 is a flowchart of an example showing the processing in the terminal device 11 according to the third modified example of the second embodiment. In Fig. 21, the processing in steps S700 to S709 shown in section (a) and the processing in steps S720 to S721 shown in section (b) are processing that operate independently of each other by, for example, different threads. Moreover, the processing in sections (a) and (b) is processing that is executed by the terminal device 11 mounted on the vehicle of interest.

[0164] First, the process of section (b) will be described. In step S720, the communication unit 110a performs a receiving process for receiving other vehicle transmission information. For example, the communication unit 110a requests metadata of the transmitted sensor information and a recognition result based on the sensor information from each of the other vehicles 101, 102, 103, . . . In the next step S721, the communication unit 110a performs a storage process for storing the other vehicle transmission information in response to the request in step S720 in the storage unit 321 by the other vehicle data storage unit 320. For example, the communication unit 110a receives a set of metadata and a recognition result transmitted from the other vehicles 101, 102, 103, . . . in response to the request in step S720, and passes the received set of metadata and a recognition result to the other vehicle data storage unit 320. The other vehicle data storage unit 320 stores and accumulates the set of metadata and a recognition result passed from the communication unit 110a in the storage unit 321.

[0165] The processes in steps S720 and S721 in this section (b) are executed as needed.

[0166] Next, the processing of section (a) will be described. In the first step S700, the terminal device 11 performs imaging processing using the imaging unit 300, and acquires sensor information including image data of the captured image from the camera 12 and metadata related to the imaging by the camera 12.

[0167] In the next step S701, the terminal device 11 performs object recognition processing based on the sensor information acquired in the imaging processing in step S700 using the recognition unit 101. In the next step S702, the terminal device 11 performs processing using the recognition result from the object recognition processing by the recognition unit 101 using the recognition result utilization unit 301. In addition, the recognition unit 101 passes the recognition result and metadata contained in the sensor information to the transmission determination unit 107d.

[0168] The next step S703st indicates the start of a loop process between step S703st and step S703ed, which is executed for all other vehicle data (recognition results and metadata) accumulated in the storage unit 321 in step S721 of section (b).

[0169] In step S704, the transmission determination unit 107d requests the other vehicle data accumulation unit 320 to read one pair of metadata and recognition results stored in the memory unit 321, and obtains the pair of metadata and recognition results read from the memory unit 321 in response to this request.

[0170] In the next step S705, the transmission determination unit 107d determines whether the set of metadata and recognition result read from the storage unit 321 in step S704 is similar data to the set of metadata and recognition result passed from the recognition unit 101 in step S702. For example, the transmission determination unit 107d calculates the similarity between each recognition result, and if the calculated similarity is equal to or less than a threshold, it can determine that the set of metadata and recognition result read from the storage unit 321 is similar data to the set of metadata and recognition result passed from the recognition unit 101. This is not limited to this, and this determination may be made by comparing each metadata.

[0171] If the transmission determination unit 107d determines that these are similar data (step S706, “Yes”), it transitions the process to step S706 and sets a not-yet-transmitted flag to the pair of metadata and recognition result read from the storage unit 321. After the process of step S706, the transmission determination unit 107d returns the process from step S703ed to step S703st and executes the processes from step S704 for the next pair of metadata and recognition result stored in the storage unit 321.

[0172] On the other hand, if the transmission determination unit 107d determines that these are not similar data (step S705, ``No''), the process returns from step S703ed to step S703st, and executes the process from step S704 on the next pair of metadata and recognition results stored in the memory unit 321.

[0173] When the processes of steps S704 and S705 are completed for all pairs of metadata and recognition results stored in the storage unit 321, the transmission determination unit 107d shifts the process to step S707.

[0174] In step S707, the transmission determination unit 107d determines whether or not there is any pair of metadata and recognition results for which the untransmitted flag has been set in step S706, among all pairs of metadata and recognition results stored in the storage unit 321. When the transmission determination unit 107d determines that the untransmitted flag has not been set for any pair of metadata and recognition results stored in the storage unit 321 (step S707, "Yes"), the transmission determination unit 107d transitions to step S708.

[0175] In step S708, the transmission determination unit 107d passes sensor information corresponding to the set of metadata and the recognition result passed from the recognition unit 101 in step S702 to the communication unit 110a, and performs a transmission process for transmitting the sensor information to the server system 2. In the next step S709, the communication unit 110a transmits the sensor information passed from the transmission determination unit 107d to the server system 2 via the network 1.

[0176] On the other hand, in step S707, if the transmission determination unit 107d determines that, among all the pairs of metadata and recognition results stored in the memory unit 321, there is at least one pair of metadata and recognition results for which the untransmitted flag has been set in step S706 (step S707, "No"), the transmission determination unit 107d terminates the series of processes according to the flowchart of section (a) in Figure 21 without performing the sensor information transmission process.

[0177] In this way, in the third variant of the second embodiment, sensor information close to sensor information already transmitted by other vehicles 10 is not transmitted, thereby reducing communication costs for transmitting sensor information.

[0178] [4. Third embodiment] Next, a third embodiment will be described. The third embodiment relates to sensor information that was not transmitted to the server system 2 in the second embodiment described above, and is applicable to both the second embodiment and each of its modified examples. For the sake of explanation, the process according to the third embodiment will be described as being applied to the second embodiment described above.

[0179] Fig. 22 is a block diagram showing an example of the configuration of the driving support system according to the third embodiment. The configuration shown in Fig. 22 is equivalent to the configuration described in the second embodiment using Fig. 13, but as shown by solid lines in Fig. 22, the accumulation determination unit 103, the accumulation unit 104, and the accumulation information output unit 106 play major roles.

[0180] That is, in the above-mentioned second embodiment and the like, there may be cases where sensor information that was not transmitted is necessary. Therefore, in the third embodiment, the sensor information that was not transmitted is stored and accumulated in the storage unit 105 by the accumulation unit 104. Thereafter, at any timing, the sensor information is output from the accumulated information output unit 106 by a direct connection that does not go through the Internet, such as a cable connection or a LAN (Local Area Network).

[0181] Fig. 23 is a functional block diagram of an example for explaining the function of the terminal device 11 according to the third embodiment. The basic configuration is the same as the configuration of the terminal device 11 according to the second embodiment described with reference to Fig. 14, so detailed description will be omitted. In the terminal device 11 according to the third embodiment, the transmission determination unit 107e passes the sensor information determined not to be transmitted to the accumulation unit 104, and the accumulation unit 104 stores and accumulates the sensor information in the storage unit 105. When an external device capable of collecting sensor information is connected, the accumulated information output unit 106 requests the accumulation unit 104 to provide the sensor information accumulated in the storage unit 105, and outputs the sensor information read from the storage unit 105 by the accumulation unit 104 to the external device.

[0182] Fig. 24 is a flowchart of an example showing processing in the terminal device 11 according to the third embodiment. In Fig. 24, the processing in steps S820 to S821 shown in section (b) is processing that is started when an external device is connected to the accumulated information output unit 106, for example.

[0183] First, the processing of section (a) will be described. The processing of section (a) is substantially the same as the processing according to the second embodiment described with reference to FIG. 15. That is, in the first step S800, the terminal device 11 performs imaging processing using the imaging unit 300, and receives sensor information including image data of the captured image and metadata related to imaging by the camera 12 from the camera 12. In the next step S801, the terminal device 11 performs object recognition processing using the recognition unit 101 based on the sensor information acquired in the imaging processing in step S800. In the next step S802, the terminal device 11 performs processing using the recognition result obtained by the object recognition processing of the recognition unit 101 using the recognition result utilization unit 301.

[0184] In the next step S803, the metadata analysis unit 250 acquires metadata from the sensor information acquired in the imaging process in step S800, and executes an analysis process of the acquired metadata.

[0185] The next step S804st indicates the start of a loop process between step S804ed and all application destinations of the sensor information, that is, each recognizer 210 included in the server system 2.

[0186] In step S805, in the terminal device 11, metadata application determination processing is executed by the learning data determination unit 251. That is, in step S805, the learning data determination unit 251 checks whether the sensor information acquired in the imaging processing in step S800 is applicable to the target recognizer 210 among the recognizers 210 based on the metadata, for example, by using the determination method described in the first embodiment.

[0187] In the next step S806, if the learning data determination unit 251 determines that the sensor information is applicable to the target recognizer 210 (step S806, "Yes"), the process proceeds to step S807. In step S807, the accumulation determination unit 103 of the terminal device 11 executes accumulation processing of the learning data to which the sensor information is to be applied. That is, in step S807, the accumulation determination unit 103 determines to accumulate the sensor information as learning data for the target recognizer 210 determined to be applicable in step S806. The accumulation unit 104 stores and accumulates the sensor information in the storage unit 105 in accordance with this determination.

[0188] After the process of step S807, the terminal device 11 returns the process from step S804ed to step S804st, and the processes of steps S805 to S807 are executed with the next recognizer 210 among the recognizers 210 as the target recognizer 210.

[0189] On the other hand, if it is determined in step S806 that the target sensor information is not applicable to the target recognizer 210 (step S806, "No"), the process returns from step S804ed to step S804st.

[0190] When the processing of steps S804st to S804ed for each recognizer 210 for the sensor information is completed, the terminal device 11 shifts the processing to step S808. In step S808, the terminal device 11 determines by the transmission determination unit 107e whether the number of recognizers 210 determined as application destinations of the sensor information in the processing of steps S804st to S804ed is sufficient (for example, whether it exceeds a threshold).

[0191] When the transmission determination unit 107e determines that the number of recognizers 210 determined as application destinations of the sensor information is insufficient, the transmission determination unit 107e shifts the process to step S811. In step S811, the transmission determination unit 107e performs a process of storing the sensor information. For example, the transmission determination unit 107e instructs the storage unit 104 to hold the sensor information stored in the storage unit 105 in step S807 as it is. When the storage process is completed, the series of processes according to the flowchart in section (a) of FIG. 24 is completed.

[0192] On the other hand, if the transmission determination unit 107e determines in step S808 that the number of recognizers 210 determined as application destinations of the sensor information is sufficient, the transmission determination unit 107e shifts the process to step S809. In step S809, the transmission determination unit 107e performs a transmission process of the sensor information. For example, the transmission determination unit 107e adds information indicating each recognizer 210 determined to be applicable by the processes of steps S804st to S804ed to the sensor information accumulated in the storage unit 105 in step S807.

[0193] In the next step S810, the sensor information to which the information indicating each recognizer 210 is added is read from the storage unit 105, and is transmitted by the communication unit 110 to the server system 2 via the network 1. The accumulation unit 104 deletes the transmitted sensor information from the storage unit 105. The server system 2 stores the sensor information transmitted from the terminal device 11 in the storage unit 20 in association with each recognizer 210 as learning data for re-learning each recognizer 210 based on the information indicating each recognizer 210 added to the sensor information.

[0194] The processing of section (b) will now be described. When an external device is directly connected via a cable or the like, the accumulated information output unit 106 executes a connection process with the external device in step S820. When a connection with the external device is established, the accumulated information output unit 106 issues an instruction to the accumulation unit 104 to read the accumulated sensor information from the storage unit 105. The accumulated information output unit 106 transmits each piece of sensor information read from the storage unit 105 by the accumulation unit 104 to the external device.

[0195] In this way, in the third embodiment, the sensor information that has not been transmitted to the server system 2 by the transmission determination unit 107e can be output to an external device at any timing. This enables efficient use of the sensor information without incurring communication costs via the network 1.

[0196] (4-1. First modified example) Next, a first modified example of the third embodiment will be described. The first modified example of the third embodiment relates to sensor information that was not transmitted to the server system 2 in the above-mentioned second embodiment and each of its modified examples, and is applicable to any of the second embodiment and each of its modified examples. For the sake of explanation, the process according to the third embodiment will be described as being applied to the above-mentioned second embodiment.

[0197] The communication cost when transmitting sensor information from the terminal device 11 to the server system 2 via the network 1 varies depending on, for example, when a Wi-Fi (registered trademark) hotspot is used to connect to the network 1, when communication is performed at a timing when the communication load is low, and the like. Therefore, in a first modified example of the third embodiment, a network communication fee plan (communication fee information for each hour) and a communication plan (communication volume plan for each hour) are set in advance or by communication such as V2X. Then, the accumulated sensor information is transmitted at a timing when the communication cost is low, during a time period when the communication load is not high, when the communication volume in the vehicle is low, and the like.

[0198] FIG. 25 is a functional block diagram of an example for explaining the function of the terminal device 11 according to the first modified example of the third embodiment. The basic configuration is the same as the configuration of the terminal device 11 according to the second embodiment described with reference to FIG. 14, so detailed description will be omitted. In the terminal device 11 according to the first modified example of the third embodiment, the transmission determination unit 107e passes the sensor information determined not to be transmitted to the accumulation unit 104, and the accumulation unit 104 stores and accumulates the sensor information in the storage unit 105. The transmission determination unit 107e reads out the accumulated sensor information from the storage unit 105 and transmits it to the server system 2 via the network 1 at the above-mentioned timing when the communication cost is low, the time period when the communication load is not high, and the communication volume in the vehicle is low.

[0199] Fig. 26 is a flowchart of an example showing the processing in the terminal device 11 according to the first modified example of the third embodiment. In Fig. 26, the processing in steps S900 to S911 shown in section (a) and the processing in steps S920 to S924 shown in section (b) are processing that operate independently of each other by, for example, different threads.

[0200] First, the processing of section (a) will be described. The processing of section (a) is substantially the same as the processing according to the second embodiment described with reference to FIG. 15. That is, in the first step S900, the terminal device 11 performs imaging processing by the imaging unit 300, and receives sensor information including image data of the captured image and metadata related to imaging by the camera 12 from the camera 12. In the next step S901, the terminal device 11 performs object recognition processing by the recognition unit 101 based on the sensor information acquired in the imaging processing of step S900. In the next step S902, the terminal device 11 performs processing using the recognition result by the object recognition processing of the recognition unit 101 by the recognition result utilization unit 301.

[0201] In the next step S903, the metadata analysis unit 250 acquires metadata from the sensor information acquired in the imaging process in step S900, and executes an analysis process of the acquired metadata.

[0202] The next step S904st indicates the start of a loop process between step S904ed and all application destinations of the sensor information, that is, each recognizer 210 included in the server system 2.

[0203] In step S905, in the terminal device 11, metadata application determination processing is executed by the learning data determination unit 251. That is, in step S905, the learning data determination unit 251 checks whether the sensor information acquired in the imaging processing in step S900 is applicable to the target recognizer 210 among the recognizers 210 based on the metadata, for example, by using the determination method described in the first embodiment.

[0204] In the next step S906, if the learning data determination unit 251 determines that the sensor information is applicable to the target recognizer 210 (step S906, "Yes"), the process proceeds to step S907. In step S907, the accumulation determination unit 103 of the terminal device 11 executes accumulation processing of the learning data to which the sensor information is to be applied. That is, in step S907, the accumulation determination unit 103 determines to accumulate the sensor information as learning data for the target recognizer 210 determined to be applicable in step S906. The accumulation unit 104 stores and accumulates the sensor information in the storage unit 105 in accordance with this determination.

[0205] After the process of step S907, the terminal device 11 returns the process from step S904ed to step S904st, and the processes of steps S905 to S907 are executed with the next recognizer 210 of the recognizers 210 as the target recognizer 210.

[0206] On the other hand, if it is determined in step S906 that the target sensor information is not applicable to the target recognizer 210 (step S906, "No"), the process returns from step S904ed to step S904st.

[0207] When the processing of steps S904st to S904ed for each recognizer 210 for the sensor information is completed, the terminal device 11 shifts the processing to step S908. In step S908, the terminal device 11 determines by the transmission determination unit 107e whether the number of recognizers 210 determined as application destinations of the sensor information in the processing of steps S904st to S904ed is sufficient (for example, whether it exceeds a threshold value).

[0208] If the transmission determination unit 107e determines in step S908 that the number of recognizers 210 determined as application destinations of the sensor information is insufficient (step S908, "No"), the transmission determination unit 107e shifts the process to step S911. In step S911, the transmission determination unit 107e performs a process of storing the sensor information. For example, the transmission determination unit 107e instructs the storage unit 104 to hold the sensor information stored in the storage unit 105 in step S907 as it is. When the storage process is completed, the series of processes according to the flowchart in section (a) of FIG. 26 is completed.

[0209] On the other hand, when the transmission determination unit 107e determines in step S908 that the number of recognizers 210 determined as the application destination of the sensor information is sufficient (step S908, "Yes"), the transmission determination unit 107e shifts the process to step S909. In step S909, the transmission determination unit 107e performs a transmission process of the sensor information. For example, the transmission determination unit 107e adds information indicating each recognizer 210 determined to be applicable by the processes of steps S904st to S904ed to the sensor information accumulated in the storage unit 105 in step S907.

[0210] In the next step S910, the sensor information to which the information indicating each recognizer 210 is added is read from the storage unit 105, and is transmitted by the communication unit 110 to the server system 2 via the network 1. The accumulation unit 104 deletes the transmitted sensor information from the storage unit 105. The server system 2 stores the sensor information transmitted from the terminal device 11 in the storage unit 20 in association with each recognizer 210 as learning data for re-learning each recognizer 210 based on the information indicating each recognizer 210 added to the sensor information.

[0211] Next, the processing of section (b) will be described. In step S920, the transmission determination unit 107e performs a connection process to connect to the network 1 via the communication unit 110. In the next step S921, the transmission determination unit 107e performs a process to estimate the communication cost in the network 1. For example, the transmission determination unit 107e estimates the communication cost based on the communication plan related to the connection to the network 1, the rate plan, the time period, the traffic volume of the network 1, and the like.

[0212] In the next step S922, the transmission determination unit 107e determines whether the communication cost estimated in step S921 is low, for example, whether the communication cost is less than a predetermined value. If the transmission determination unit 107e determines that the estimated communication cost is high (step S922, “No”), the sensor information accumulated in the storage unit 105 is not transmitted, and the series of processes according to the flowchart in this section (b) is terminated.

[0213] On the other hand, in step S922, if the transmission determination unit 107e determines that the communication cost estimated in step S921 is low (step S922, “Yes”), the process proceeds to step S923. In step S923, the transmission determination unit 107e performs a transmission process of the sensor information stored in the storage unit 105. For example, the transmission determination unit 107e instructs the storage unit 104 to read out the sensor information stored in the storage unit 105.

[0214] In the next step S924, the transmission determination unit 107e transmits the sensor information read out from the storage unit 105 by the accumulation unit 104 in response to the instruction in step S923 to the server system 2 via the network 1 by the communication unit 110. The accumulation unit 104 deletes the transmitted sensor information from the storage unit 105.

[0215] In this way, in the first modified example of the third embodiment, the sensor information that is not transmitted to the server system 2 when acquired and is stored in the storage unit 105 is transmitted to the server system 2 via the network 1 according to the communication cost. Therefore, the communication cost related to the transmission of the sensor information can be reduced.

[0216] (4-2. Second modified example) Next, a second modified example of the third embodiment will be described. The second modified example of the third embodiment relates to sensor information that was not transmitted to the server system 2 in the above-mentioned second embodiment and each of its modified examples, and is applicable to any of the second embodiment and each of its modified examples. For the sake of explanation, the process according to the second modified example of the third embodiment will be described as being applied to the above-mentioned second embodiment.

[0217] In a second modified example of the third embodiment, a priority is assigned to sensor information that was not transmitted to the server system 2 at the time of acquisition, and accumulation of the sensor information is controlled according to the priority. That is, the priority of the sensor information may differ depending on the contents contained in the sensor information and the situation at the time of acquisition. Therefore, when accumulating sensor information that was not transmitted to the server system 2, the accumulation determination unit 103 assigns a priority to the sensor information and accumulates it in the storage unit 105. When accumulating an amount of sensor information that exceeds the amount of information that can be accumulated in the accumulation unit 104 (storage unit 105), the accumulation determination unit 103 sequentially deletes sensor information with the lowest priority from among the accumulated sensor information, and accumulates sensor information with a higher priority.

[0218] FIG. 27 is a functional block diagram of an example for explaining the function of the terminal device 11 according to the second modified example of the third embodiment. The basic configuration is the same as the configuration of the terminal device 11 according to the second embodiment described using FIG. 14, so detailed description will be omitted. In the terminal device 11 according to the second modified example of the third embodiment, the sensor information determined not to be transmitted is passed to the accumulation determination unit 103, and the accumulation determination unit 103 assigns a priority to the sensor information. The accumulation unit 104 stores the sensor information to which the priority has been assigned in the storage unit 105 and accumulates it. At this time, when there is no free space in the storage unit 105, the accumulation determination unit 103 deletes the sensor information to which the lowest priority has been assigned among the sensor information stored in the storage unit 105.

[0219] In addition, the transmission determination unit 107e reads out the accumulated sensor information from the memory unit 105 and transmits it to the server system 2 via the network 1 at times when communication costs are low, during times when the communication load is not high, and when the amount of communication within the vehicle is low, as described above.

[0220] Fig. 28 is a flowchart of an example showing the processing in the terminal device 11 according to the second modified example of the third embodiment. In Fig. 28, the processing in steps S1000 to S1017 shown in section (a) and the processing in steps S1030 to S1034 shown in section (b) are processing that operate independently of each other by different threads.

[0221] First, the processing of section (a) will be described. In the processing of section (a), the processing of steps S1000 to S1010 is substantially the same as the processing of steps S400 to S410 in the flowchart of Fig. 15. That is, in the first step S1000, the terminal device 11 performs imaging processing by the imaging unit 300, and receives sensor information including image data of the captured image and metadata related to imaging by the camera 12 from the camera 12. In the next step S1001, the terminal device 11 performs object recognition processing by the recognition unit 101 based on the sensor information acquired in the imaging processing of step S1000. In the next step S1002, the terminal device 11 performs processing using the recognition result by the object recognition processing of the recognition unit 101 by the recognition result utilization unit 301.

[0222] In the next step S1003, the metadata analysis unit 250 acquires metadata from the sensor information acquired in the imaging process in step S1000, and executes an analysis process of the acquired metadata.

[0223] The next step S1004st indicates the start of a loop process between step S1004ed and all application destinations of the sensor information, that is, each recognizer 210 included in the server system 2.

[0224] In step S1005, in the terminal device 11, metadata application determination processing is executed by the learning data determination unit 251. That is, in step S1005, the learning data determination unit 251 checks whether the sensor information acquired in the imaging processing in step S1000 is applicable to the target recognizer 210 among the recognizers 210 based on the metadata, for example, by using the determination method described in the first embodiment.

[0225] In the next step S1006, if the learning data determination unit 251 determines that the sensor information is applicable to the target recognizer 210 (step S1006, "Yes"), the process proceeds to step S1007. In step S1007, the accumulation determination unit 103 of the terminal device 11 executes accumulation processing of the learning data to which the sensor information is to be applied. That is, in step S1007, the accumulation determination unit 103 determines to accumulate the sensor information as learning data for the target recognizer 210 determined to be applicable in step S1006. The accumulation unit 104 stores and accumulates the sensor information in the storage unit 105 according to this determination.

[0226] After the process of step S1007, the terminal apparatus 11 returns the process from step S1004ed to step S1004st, and the processes of steps S1005 to S1007 are executed with the next recognizer 210 among the recognizers 210 as the target recognizer 210.

[0227] On the other hand, if it is determined in step S1006 that the target sensor information is not applicable to the target recognizer 210 (step S1006, "No"), the process returns from step S1004ed to step S1004st.

[0228] When the processing of steps S1004st to S1004ed for each recognizer 210 for the sensor information is completed, the terminal device 11 shifts the processing to step S1008. In step S1008, the terminal device 11 determines by the transmission determination unit 107e whether the number of recognizers 210 determined as application destinations of the sensor information in the processing of steps S1004st to S1004ed is sufficient (for example, whether it exceeds a threshold).

[0229] On the other hand, when the transmission determination unit 107e determines in step S1008 that the number of recognizers 210 determined as the application destination of the sensor information is sufficient (step S1008, "Yes"), the transmission determination unit 107e shifts the process to step S1009. In step S1009, the transmission determination unit 107e performs a transmission process of the sensor information. For example, the transmission determination unit 107e adds information indicating each recognizer 210 determined to be applicable by the processes of steps S1004st to S1004ed to the sensor information accumulated in the storage unit 105 in step S1007.

[0230] In the next step S1010, the sensor information to which the information indicating each recognizer 210 is added is read from the storage unit 105, and is transmitted by the communication unit 110 to the server system 2 via the network 1. The accumulation unit 104 deletes the transmitted sensor information from the storage unit 105. The server system 2 stores the sensor information transmitted from the terminal device 11 in the storage unit 20 in association with each recognizer 210 as learning data for re-learning each recognizer 210 based on the information indicating each recognizer 210 added to the sensor information.

[0231] If the transmission determination unit 107e determines in step S1008 that the number of recognizers 210 determined as application destinations of the sensor information is not sufficient (step S1008, "No"), the transmission determination unit 107e shifts the process to step S1011.

[0232] In step S1011, the accumulation determination unit 103 sets a priority for the sensor information (which will be referred to as new sensor information). For example, the accumulation determination unit 103 can set a priority based on metadata contained in the new sensor information. The accumulation determination unit 103 can also set a priority for the new sensor information based on the recognition result of the recognition unit 101. For example, the accumulation determination unit 103 can set a higher priority for the new sensor information as the recognition result indicates a higher recognition rate.

[0233] In the next step S1012, the accumulation determination unit 103 determines whether or not there is enough free space in the storage unit 105 to store the new sensor information. For example, the accumulation determination unit 103 inquires of the accumulation unit 104 about the free space in the storage unit 105.

[0234] If it is determined in step S1012 that the memory unit 105 has enough free space to store the new sensor information (step S1012, "No"), the accumulation determination unit 103 transitions the process to step S1017. In step S1017, the accumulation determination unit 103 instructs the accumulation unit 104 to store the new sensor information in the memory unit 105. In response to this instruction, the memory unit 105 stores and accumulates the new sensor information in the memory unit 105. When the storage of the sensor information in the memory unit 105 is completed, the series of processes according to the flowchart in this section (a) is terminated.

[0235] On the other hand, if it is determined in step S1012 that the memory unit 105 does not have enough free space to store new sensor information (step S1012, "Yes"), the accumulation determination unit 103 shifts the process to step S1013. In step S1013, the accumulation determination unit 103 instructs the accumulation unit 104 to search for sensor information with the lowest priority among the pieces of sensor information stored in the memory unit 105.

[0236] In the next step S1014, the accumulation determination unit 103 compares the priority of the sensor information searched for in step S1013 with the priority of the new sensor information. If it is determined as a result of the comparison that the priority of the sensor information searched for in step S1013 (assumed to be the lowest priority sensor information) is higher than the priority of the new sensor information (step S1014, "the priority of the lowest priority data is higher"), the series of processes according to the flowchart in this section (a) is terminated. In this case, the new sensor information is, for example, discarded.

[0237] On the other hand, if the accumulation determination unit 103 determines in step S1014 that the priority of the new sensor information is higher than the priority of the lowest priority sensor information (step S1014, "priority of new priority data is higher"), the accumulation determination unit 103 shifts the process to step S1015. In step S1015, the accumulation determination unit 103 instructs the accumulation unit 104 to delete the lowest priority sensor information from the storage unit 105. In response to this instruction, the accumulation unit 104 deletes the lowest priority sensor information from the storage unit 105.

[0238] In the next step S1016, accumulation determination unit 103 instructs accumulation unit 104 to store the new sensor information in memory unit 105. In response to this instruction, memory unit 105 stores and accumulates the new sensor information in memory unit 105. When storage of the sensor information in memory unit 105 is completed, the series of processes according to the flowchart in this section (a) is terminated.

[0239] The processing of section (b) will be described. The processing of section (b) in FIG. 28 is substantially the same as the processing of section (b) in FIG. 26. That is, in step S1030, the transmission determination unit 107e performs a connection process to connect to the network 1 by the communication unit 110. In the next step S1031, the transmission determination unit 107e performs a process to estimate the communication cost in the network 1. For example, the transmission determination unit 107e estimates the communication cost based on the communication plan, fee plan, time period, traffic volume of the network 1, etc., related to the connection to the network 1.

[0240] In the next step S1032, the transmission determination unit 107e determines whether the communication cost estimated in step S1031 is low, for example, whether the communication cost is less than a predetermined value. If the transmission determination unit 107e determines that the estimated communication cost is high (step S1032, “No”), the sensor information accumulated in the storage unit 105 is not transmitted, and the series of processes according to the flowchart in this section (b) is terminated.

[0241] On the other hand, in step S1032, if the transmission determination unit 107e determines that the communication cost estimated in step S1031 is low (step S1032, “Yes”), the process proceeds to step S1033. In step S1033, the transmission determination unit 107e performs a transmission process of the sensor information stored in the storage unit 105. For example, the transmission determination unit 107e instructs the storage unit 104 to read out the sensor information stored in the storage unit 105.

[0242] In the next step S1034, the transmission determination unit 107e transmits the sensor information read out from the storage unit 105 by the accumulation unit 104 in response to the instruction in step S1033 to the server system 2 via the network 1 by the communication unit 110. The accumulation unit 104 deletes the transmitted sensor information from the storage unit 105.

[0243] In this way, in the second modified example of the third embodiment, a priority is set for sensor information that has not been transmitted to the server system 2, and when there is not enough free space in the storage unit 105 to store the sensor information, sensor information with a low priority is deleted. This makes it possible to save the storage capacity of the storage unit 105. In addition, since the sensor information with a high priority is accumulated in the storage unit 105, the sensor information stored in the storage unit 105 can be read out at a predetermined timing and transmitted to the server system 2, thereby making it possible to efficiently use the sensor information and also reducing communication costs.

[0244] [5. Fourth embodiment] The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, a robot, etc.

[0245] FIG. 29 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a moving object control system to which the technology according to the present disclosure can be applied.

[0246] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 29, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also, as functional configurations of the integrated control unit 12050, a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (interface) 12053 are illustrated.

[0247] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0248] The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as head lamps, back lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves or signals of various switches transmitted from a portable device that replaces a key may be input to the body system control unit 12020. The body system control unit 12020 receives the input of these radio waves or signals and controls the door lock device, power window device, lamps, and the like of the vehicle.

[0249] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle and receives the captured image. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for a person, a car, an obstacle, a sign, or characters on a road surface, based on the received image.

[0250] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0251] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing.

[0252] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the outside-of-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc.

[0253] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on the driver's operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0254] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside-vehicle information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detection unit 12030, and perform cooperative control for the purpose of preventing glare, such as switching from high beams to low beams.

[0255] The audio / video output unit 12052 transmits at least one output signal of audio and video to an output device capable of visually or audibly notifying information to passengers in the vehicle or the outside of the vehicle. In the example of Fig. 29, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as the output device. The display unit 12062 may include at least one of an on-board display and a head-up display, for example.

[0256] FIG. 30 is a diagram showing an example of the installation position of the imaging unit 12031.

[0257] In FIG. 30, a vehicle 12100 has imaging units 12101, 12102, 12103, 12104, and 12105 as an imaging unit 12031.

[0258] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided at positions such as the front nose, side mirrors, rear bumper, back door, and the upper part of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided at the front nose and the imaging unit 12105 provided at the upper part of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided at the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided at the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The images of the front acquired by the imaging units 12101 and 12105 are mainly used to detect a preceding vehicle, a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

[0259] 30 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, an overhead image of the vehicle 12100 viewed from above is obtained by superimposing the image data captured by the imaging units 12101 to 12104.

[0260] At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera made up of a plurality of imaging elements, or may be an imaging element having pixels for detecting a phase difference.

[0261] For example, the microcomputer 12051 can extract, as a preceding vehicle, a three-dimensional object that is the closest three-dimensional object on the travel path of the vehicle 12100 and travels at a predetermined speed (for example, 0 km / h or more) in approximately the same direction as the vehicle 12100, by calculating the distance to each three-dimensional object in the imaging ranges 12111 to 12114 and the change over time of this distance (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104. Furthermore, the microcomputer 12051 can set a vehicle distance to be secured in advance in front of the preceding vehicle, and perform automatic brake control (including follow-up stop control) and automatic acceleration control (including follow-up start control). In this way, cooperative control can be performed for the purpose of automatic driving that travels autonomously without relying on the driver's operation.

[0262] For example, the microcomputer 12051 classifies and extracts three-dimensional object data on three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines a collision risk indicating the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0263] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured images of the imaging units 12101 to 12104. The recognition of such a pedestrian is performed, for example, by a procedure of extracting feature points in the captured images of the imaging units 12101 to 12104 as infrared cameras, and a procedure of performing pattern matching processing on a series of feature points that indicate the contour of an object to determine whether or not the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular contour line for emphasis on the recognized pedestrian. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.

[0264] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. In the above-described configuration, the technology according to the present disclosure can apply the camera 12 to the imaging unit 12031, and the terminal device 11 to the vehicle outside information detection unit 12030. By applying the technology according to the present disclosure to the vehicle outside information detection unit 12030, the sensor information acquired by the camera 12 can be used more efficiently, and as a result, it is possible to increase the recognition rate of pedestrians, etc. In addition, by applying the second embodiment and each of its modified examples, or the third embodiment and each of its modified examples to the vehicle outside information detection unit 12030, it is also possible to reduce communication costs for transmitting sensor information.

[0265] The present technology can also be configured as follows. (1) a recognition unit that performs object recognition processing using sensor information acquired by a sensor by a first recognizer that has been trained in advance; a learning data application determination unit that determines whether the sensor information can be applied as learning data to a second recognizer different from the first recognizer; An information processing device comprising: (2) a transmission determination unit that determines whether or not to transmit the sensor information to the server so as to apply the sensor information to the second recognizer as the learning data, based on the sensor information; The information processing device according to (1) above, further comprising: (3) The transmission determination unit transmits to the server, among the sensor information, sensor information in which a number of recognizers capable of performing the recognition process designated by the learning data application determination unit from the second recognizer is equal to or greater than a predetermined number; The information processing device according to (2). (4) The transmission determination unit transmitting the sensor information, the time until the next update of which is less than a predetermined time, to the server; The information processing device according to (2) or (3). (5) The transmission determination unit transmitting the sensor information whose occurrence frequency is less than a predetermined frequency to the server; An information processing device according to any one of (2) to (4). (6) Further comprising a communication unit capable of communicating with other information processing devices, The transmission determination unit communicating with the other information processing device by the communication unit to acquire information indicating sensor information to be transmitted by the other information processing device to the server, and determining whether or not to transmit the sensor information acquired by the sensor to the server based on the acquired information; The information processing device according to any one of (2) to (5). (7) An accumulation determination unit that determines whether or not to accumulate the sensor information in an accumulation unit, The accumulation determination unit is The sensor information determined not to be transmitted to the server by the transmission determination unit is stored in the storage unit. An information processing device according to any one of (2) to (6). (8) The transmission determination unit The sensor information stored in the storage unit is transmitted to the server at a timing when communication costs are low. The information processing device according to (7) above. (9) The accumulation determination unit is determining a priority of the sensor information determined not to be transmitted to the server, and determining whether or not to store the sensor information in the storage unit based on the priority; The information processing device according to (7) or (8). (10) A server; An information processing device capable of communicating with the server; Including, The server, A learning unit that generates a first recognition unit and a second recognition unit different from the first recognition unit by machine learning based on learning data, The information processing device includes: a recognition unit that performs object recognition processing using sensor information acquired by a sensor by the first recognizer; a learning data application determination unit that determines whether the sensor information can be applied as learning data to a second recognizer different from the first recognizer; Equipped with Information processing system. (11) Executed by a processor, A recognition step of performing an object recognition process using sensor information acquired by a sensor by a first recognizer that has been trained in advance; a recognizer determination step of selecting a recognizer capable of performing recognition processing using the sensor information from a second recognizer different from the first recognizer based on the sensor information; a learning data application determination step of determining whether the sensor information can be applied as learning data to a second recognizer different from the first recognizer; An information processing method comprising: (12) A recognition step of performing an object recognition process using sensor information acquired by a sensor by a first recognizer that has been trained in advance; a recognizer determination step of selecting a recognizer capable of performing recognition processing using the sensor information from a second recognizer different from the first recognizer based on the sensor information; a learning data application determination step of determining whether the sensor information can be applied as learning data to a second recognizer different from the first recognizer; An information processing program for causing a computer to execute the above. (13) a learning unit that generates a first recognition unit and a second recognition unit different from the first recognition unit by machine learning based on learning data; a learning data application determination unit that determines whether or not the sensor information can be applied as learning data to the second recognizer based on sensor information transmitted from a terminal device including the first recognition unit; Equipped with The learning unit is when it is determined by the learning data application determination unit that the sensor information is applicable to the second recognizer as learning data, the second recognizer is re-learned based on the sensor information to update the second recognizer. Information processing device. (14) The learning data application determination unit is determining whether the sensor information is applicable to the second recognizer as learning data based on the scene indicated by the sensor information; The information processing device according to (13). (15) The learning data application determination unit is determining whether or not the sensor information can be applied as learning data to the second recognizer by converting the sensor information; The information processing device according to (13) or (14). (16) The learning data application determination unit is determining whether the sensor information is applicable to the second recognizer as learning data according to the object detected by the sensor, the sensor information indicating the object; 16. The information processing device according to any one of (13) to (15). (17) An information processing device; A terminal device having a first recognition unit that performs object recognition processing using sensor information acquired by a sensor and is capable of communicating with the information processing device; Including, The information processing device includes: a learning unit that generates the first recognition unit and a second recognition unit different from the first recognition unit by machine learning based on learning data; a learning data application determination unit that determines whether or not the sensor information can be applied as learning data to the second recognizer based on sensor information transmitted from a terminal device including the first recognition unit; Equipped with The learning unit is when it is determined by the learning data application determination unit that the sensor information is applicable to the second recognizer as learning data, the second recognizer is re-learned based on the sensor information to update the second recognizer. Information processing system. (18) Executed by a processor, A learning step of generating a first recognition unit and a second recognition unit different from the first recognition unit by machine learning based on learning data; a learning data application determination step of determining whether or not the sensor information can be applied as learning data to the second recognizer based on the sensor information transmitted from the terminal device including the first recognition unit; Including, The learning step includes: When it is determined that the sensor information can be applied to the second recognizer as learning data in the learning data application determination step, the second recognizer is re-trained based on the sensor information to update the second recognizer. Information processing methods. (19) A learning step of generating a first recognition unit and a second recognition unit different from the first recognition unit by machine learning based on learning data; a learning data application determination step of determining whether or not the sensor information can be applied as learning data to the second recognizer based on the sensor information transmitted from the terminal device including the first recognition unit; on the computer, The learning step includes: When it is determined that the sensor information can be applied to the second recognizer as learning data in the learning data application determination step, the second recognizer is re-trained based on the sensor information to update the second recognizer. Information processing program for. [Explanation of symbols]

[0266] 1 Network 2. Server System 3,202 Study Department 10,101,102,103 vehicles 11 Terminal Equipment 12 Camera 20,105 Storage section 22, 22a, 22b, 22c, 22d, 108, 108a, Learning data application judgment unit 101 Recognition part 103 Accumulation determination section 104 Storage Unit 106 Accumulated information output unit 107, 107a, 107b, 107c, 107d, 107e Transmission decision unit 110,110a,200 Communication Department 152 Learning Data Immediacy Calculation Unit 153 Learning Data Immediacy Judgment Unit 201 Learning Data Storage Unit 210,2101,2102,2103,2104 recognizer 221,250 Metadata Analysis Department 230 Domain Analysis Department 231 Domain Evaluation Department 232,241,251 Learning data judgment unit 240 Sensing Analysis Unit 300 Imaging unit 310 Scarcity Analysis Department 311 Rarity Judgment Department 320 Other vehicle data storage unit

Claims

1. a recognition unit that performs object recognition processing using sensor information acquired by a sensor by a first recognizer that has been trained in advance; a learning data application determination unit that determines whether the sensor information can be applied as learning data to a second recognizer different from the first recognizer; a transmission determination unit that determines whether or not to transmit the sensor information to a server capable of generating the first recognizer so as to apply the sensor information to the second recognizer as the learning data, based on the sensor information; Equipped with The transmission determination unit transmits to the server, among the sensor information, sensor information in which a number of recognizers capable of performing the recognition process designated by the learning data application determination unit from the second recognizer is equal to or greater than a predetermined number; Information processing device.

2. The transmission determination unit transmitting the sensor information, the time until the next update of which is less than a predetermined time, to the server; The information processing device according to claim 1 .

3. The transmission determination unit transmitting the sensor information whose occurrence frequency is less than a predetermined frequency to the server; The information processing device according to claim 1 .

4. Further comprising a communication unit capable of communicating with other information processing devices, The transmission determination unit communicating with the other information processing device by the communication unit to acquire information indicating sensor information to be transmitted by the other information processing device to the server, and determining whether or not to transmit the sensor information acquired by the sensor to the server based on the acquired information; The information processing device according to claim 1 .

5. An accumulation determination unit that determines whether or not to accumulate the sensor information in an accumulation unit, The accumulation determination unit is The sensor information determined not to be transmitted to the server by the transmission determination unit is stored in the storage unit. The information processing device according to claim 1 .

6. The transmission determination unit The sensor information stored in the storage unit is transmitted to the server at a timing when communication costs are low. The information processing device according to claim 5 .

7. The accumulation determination unit is determining a priority of the sensor information determined not to be transmitted to the server, and determining whether or not to store the sensor information in the storage unit based on the priority; The information processing device according to claim 5 .

8. A server; An information processing device capable of communicating with the server; Including, The server, A learning unit that generates a first recognizer and a second recognizer different from the first recognizer by machine learning based on learning data, The information processing device includes: a recognition unit that performs object recognition processing using sensor information acquired by a sensor by the first recognizer; a learning data application determination unit that determines whether the sensor information can be applied as learning data to a second recognizer different from the first recognizer; a transmission determination unit that determines whether or not to transmit the sensor information to a server capable of generating the first recognizer so as to apply the sensor information to the second recognizer as the learning data, based on the sensor information; Equipped with The transmission determination unit transmits to the server, among the sensor information, sensor information in which a number of recognizers capable of performing the recognition process designated by the learning data application determination unit from the second recognizer is equal to or greater than a predetermined number; Information processing system.

9. Executed by a processor, A recognition step of performing an object recognition process using sensor information acquired by a sensor by a first recognizer that has been trained in advance; a recognizer determination step of specifying a recognizer capable of performing recognition processing using the sensor information from a second recognizer different from the first recognizer based on the sensor information; a learning data application determination step of determining whether the sensor information can be applied as learning data to a second recognizer different from the first recognizer; a transmission determination step of determining, based on the sensor information, whether or not to transmit the sensor information to a server capable of generating the first recognizer so as to apply the sensor information to the second recognizer as the learning data; Including, The transmission determination step includes: transmits to the server, among the sensor information, sensor information for which a number of recognizers capable of performing the recognition process designated by the second recognizer in the learning data application determination step is equal to or greater than a predetermined number; Information processing methods.

10. A recognition step of performing an object recognition process using sensor information acquired by a sensor by a first recognizer that has been trained in advance; a recognizer determination step of specifying a recognizer capable of performing recognition processing using the sensor information from a second recognizer different from the first recognizer based on the sensor information; a learning data application determination step of determining whether the sensor information can be applied as learning data to a second recognizer different from the first recognizer; a transmission determination step of determining, based on the sensor information, whether or not to transmit the sensor information to a server capable of generating the first recognizer so as to apply the sensor information to the second recognizer as the learning data; on the computer, The transmission determination step includes: transmits to the server, among the sensor information, sensor information for which a number of recognizers capable of performing the recognition process designated by the second recognizer in the learning data application determination step is equal to or greater than a predetermined number; Information processing program for.

11. a learning unit that generates a first recognizer and a second recognizer different from the first recognizer by machine learning based on learning data; a learning data application determination unit that determines whether or not the sensor information can be applied as learning data to the second recognizer based on sensor information transmitted from a terminal device including the first recognizer; a transmission determination unit that determines whether or not to transmit the sensor information to a server capable of generating the first recognizer so as to apply the sensor information to the second recognizer as the learning data, based on the sensor information; Equipped with The learning unit is when it is determined by the learning data application determination unit that the sensor information is applicable to the second recognizer as learning data, re-learning the second recognizer based on the sensor information to update the second recognizer; The transmission determination unit transmits to the server, among the sensor information, sensor information in which a number of recognizers capable of performing the recognition process designated by the learning data application determination unit from the second recognizer is equal to or greater than a predetermined number; Information processing device.

12. The learning data application determination unit is determining whether the sensor information is applicable to the second recognizer as learning data based on the scene indicated by the sensor information; The information processing device according to claim 11.

13. The learning data application determination unit is determining whether or not the sensor information can be applied as learning data to the second recognizer by converting the sensor information; The information processing device according to claim 11.

14. The learning data application determination unit is determining whether the sensor information is applicable to the second recognizer as learning data according to an object detected by a sensor, the object being indicated by the sensor information; The information processing device according to claim 11.

15. An information processing device; a terminal device having a first recognizer that performs object recognition processing using sensor information acquired by a sensor and capable of communicating with the information processing device; Including, The information processing device includes: a learning unit that generates the first recognizer and a second recognizer different from the first recognizer by machine learning based on learning data; a learning data application determination unit that determines whether or not the sensor information can be applied as learning data to the second recognizer based on sensor information transmitted from a terminal device including the first recognizer; a transmission determination unit that determines whether or not to transmit the sensor information to a server capable of generating the first recognizer so as to apply the sensor information to the second recognizer as the learning data, based on the sensor information; Equipped with The learning unit is when it is determined by the learning data application determination unit that the sensor information is applicable to the second recognizer as learning data, re-learning the second recognizer based on the sensor information to update the second recognizer; The transmission determination unit transmits to the server, among the sensor information, sensor information in which a number of recognizers capable of performing the recognition process designated by the learning data application determination unit from the second recognizer is equal to or greater than a predetermined number; Information processing system.

16. Executed by a processor, A learning step of generating a first recognizer and a second recognizer different from the first recognizer by machine learning based on training data; a learning data application determination step of determining whether or not the sensor information can be applied as learning data to the second recognizer based on the sensor information transmitted from a terminal device including the first recognizer; a transmission determination step of determining, based on the sensor information, whether or not to transmit the sensor information to a server capable of generating the first recognizer so as to apply the sensor information to the second recognizer as the learning data; Including, The learning step includes: when it is determined that the sensor information can be applied to the second recognizer as learning data in the learning data application determination step, the second recognizer is re-learned based on the sensor information to update the second recognizer; The transmission determination step includes: transmits to the server, among the sensor information, sensor information for which a number of recognizers capable of performing the recognition process designated by the second recognizer in the learning data application determination step is equal to or greater than a predetermined number; Information processing methods.

17. A learning step of generating a first recognizer and a second recognizer different from the first recognizer by machine learning based on training data; a learning data application determination step of determining whether or not the sensor information can be applied as learning data to the second recognizer based on the sensor information transmitted from a terminal device including the first recognizer; a transmission determination step of determining, based on the sensor information, whether or not to transmit the sensor information to a server capable of generating the first recognizer so as to apply the sensor information to the second recognizer as the learning data; on the computer, The learning step includes: when it is determined that the sensor information can be applied to the second recognizer as learning data in the learning data application determination step, the second recognizer is re-learned based on the sensor information to update the second recognizer; The transmission determination step includes: transmits to the server, among the sensor information, sensor information for which a number of recognizers capable of performing the recognition process designated by the second recognizer in the learning data application determination step is equal to or greater than a predetermined number; Information processing program for.

Citation Information

Patent Citations

  • Learning data management device, learning data management method and air-conditioner for vehicle, and control device of apparatus

    JP2010055303A

  • Learning system, on-vehicle device, and server

    JP2015135552A

  • Service providing system and database

    JP2018202595A

  • Information processing device

    JP2019159659A

  • Filtering training data for models in a data center

    US20190034829A1