Driving support system, image recognition device, image recognition program, update information providing method, update information providing program, update data providing method, and vehicle

Abstract

To provide a technique for efficiently realizing a learned model according to a region.SOLUTION: A driving support system includes a vehicle, a server device, and a broadcast device. The server device uses image data collected from a target image collection area corresponding to a broadcast area of a broadcast wave as input data, generates learning data obtained by adding correct answer data to the image data, generates a learned model by learning the learned model using the generated learning data, generates update data for model update based on the generated learned model, and outputs the generated update data. The broadcast device transmits the update data by superimposing the update data on a broadcast wave of a digital broadcast in which a broadcast area is defined. The vehicle includes: an image recognition device having the learned model for performing object detection; and a driving assistance device that performs driving assistance based on object detection information obtained by the image recognition device. The image recognition device extracts update data from the received broadcast wave of the digital broadcast, and updates the learned model based on the extracted update data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a driving assistance system, an image recognition device, an image recognition program, a method for providing update information, an update information providing program, a method for providing update data, and a vehicle.

Background Art

[0002] Conventionally, a monitoring system has been proposed that generates a dedicated AI model suitable for video analysis by a camera by adjusting a general-purpose AI model for each camera (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a so-called object detection method using AI (Artificial Intelligence), the detection accuracy for data acquired in an environment different from the learning data may decrease. This is likely to occur when the distribution of the data features is different between the learning data set and the data input during operation. For example, in an object detection task using a DNN (Deep Neural Network), the detection accuracy also decreases depending on differences in the imaging equipment and the scene (such as illuminance and color tone) at the imaging location. In the case of an in-vehicle image recognition device, it is necessary to learn images around the vehicle (around the road). The images around the vehicle have regional characteristics. For example, the images around the vehicle in winter in a snowy region are different in characteristics from the images around the vehicle in a southern region. However, in order to store models corresponding to various regions including models with low usage frequency in the image recognition device in advance, it is necessary to secure storage capacity, which is wasteful and not reasonable.

[0005] ​

[0006] This technology aims to provide a technology for efficiently realizing a learned model according to a region.

Means for Solving the Problem

[0007] The driving support system according to the present disclosure is as follows. A driving support system including a vehicle, a server device, and a broadcasting device, The server device, uses, as input data, image data collected from a target image collection area corresponding to the broadcast area of a broadcast wave, generates learning data with correct answer data added to the image data, learns a learning model using the generated learning data to generate a learned model, generates update data for model update based on the generated learned model, outputs the generated update data, The broadcasting device, acquires the update data, superimposes the acquired update data on a broadcast wave of digital broadcast whose broadcast area is defined and transmits it, The vehicle, has an image recognition device having a learned model for performing object detection, and a driving support device that performs driving support based on object detection information by the image recognition device, and is provided with The image recognition device, extracts update data of the learned model that the image recognition device has from the received broadcast wave of the digital broadcast, and updates the learned model that the image recognition device has based on the extracted update data, A driving support system.

Advantages of the Invention

[0008] According to this technology, it is possible to provide a technology for efficiently realizing a learned model according to a region.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Mode for Carrying Out the Invention

[0010] <Embodiment> Hereinafter, the embodiments will be described with reference to the drawings. Figure 1 is a diagram for explaining the outline of the driving support system. System 1 includes a server 2 that performs machine learning and creates update data for the learned model, a transmission station 3 (3A, 3B) that transmits a broadcast wave including the update data of the learned model, and vehicles 4 (4A to 4F) equipped with an imaging device, an image recognition device, and a driving support device.

[0011] The driving support device of the vehicle 4 performs driving support for the driver of the vehicle 4 based on the object detection information output by the image recognition device. Note that the driving support device may be integrally configured with the image recognition device. The image recognition device of the vehicle 4 stores a learned model for performing object detection. The learned model is constructed using, for example, a DNN (Deep Neural Network). It is a machine learning model for detecting an object from image data (recognizing an object and an object type in this embodiment). Also, it is assumed that a general-purpose model is stored in the vehicle 4 in advance for the learned model, and it can be customized by applying differential update data. Further, the update data may be various parameters of a machine learning model for detecting an object and an object type, and may be information including a combined structure of the machine learning model for detecting an object and an object type and weights between nodes, or may be weight information for causing a machine learning model with a predetermined structure to detect an object and an object type. Also, among a plurality of layers in a neural network, for example, weight information between some of the output-side layers may be used as update information.

[0012] At least a part of the update data is created for each region. That is, an image is captured for each area (within the region and its neighboring regions (for example, an area where it can be determined that the scenery in each season is similar)) where an image having regional characteristics can be captured, and update data for the region corresponding to the captured area is created from the captured image. For the sake of easy understanding, the area where the machine learning model to be updated exists is referred to as the update target model area, and the area where an image suitable for creating learning data for the learning model existing in the update target model area is captured (collected) is referred to as the target image collection area. In other words, the update data is data for changing the learned model used by the image recognition device of the vehicle 4 to a model that reflects the regional characteristics of the object image captured in the corresponding target image collection area in order to improve the detection accuracy in the update target model area. That is, the update data may be data for updating the entire learned model (all weight information, or combined structure and all weight information), or may be data for updating a part of the learned model (for example, all or part of the weight information (the combined structure is not updated)).

[0013] In addition, in snowy regions, the scenery is completely different between winter and seasons other than winter, and the images of each object are also affected by snow and the like. Therefore, it is preferable to create update data based on the photographed images for each season (winter and seasons other than winter), for example, based on image data such as a snow-covered road, a traffic signal, a road sign, and other vehicles. That is, it is also effective to create update data not only for each update target model area but also for each period (season), each time zone, and each weather condition.

[0014] The created update data (for example, the one suitable for the area, season, time zone, and weather is selected) is transmitted from the server 2 to the transmission station 3 in the area corresponding to the update data via a predetermined transmission path (Figure 1: solid arrow). Then, a broadcast wave including the update data for each region is transmitted to the radio wave reach range 30 (30A, 30B) of the transmission station 3 (Figure 1: dashed arrow).

[0015] The transmission station 3 is, for example, a broadcast transmission station or a relay station that transmits terrestrial digital broadcasts. The transmission station 3 encodes the update data, for example, as part of the data broadcast of a broadcast station (performance venue), and repeatedly transmits it in a carousel manner. The image recognition device of the vehicle 4 receives the broadcast wave and updates the learned model using the update data read from the broadcast wave. In this way, the system 1 can provide region-specific data as update data multiplexed on the broadcast wave to the vehicle 4 (image recognition device) existing in the radio wave reach range of the broadcast wave.

[0016] Figure 2 is a functional block diagram of the server 2 and the vehicle 4. The server 2 includes a processor 21, a storage device 22, a communication interface (IF) 23, and a user interface (UI) 24. These components are connected via a bus. The processor 21 is an arithmetic processing device such as a CPU (Central Processing Unit). Pro The sensor 21 executes the processing according to a predetermined program. For example, the processor 21 performs machine learning on the relationship between an image and teacher data indicating an object captured in the image, and creates a learned model for object detection. The storage device 22 is a main storage device such as a RAM (Random Access Memory) or a ROM (Read Only Memory), and an auxiliary storage device such as an HDD (Hard-Disk Drive) and an SSD (Solid State Drive) or a flash memory. The main storage device temporarily stores a program read by the processor 21 and information transmitted and received to and from other computers, and secures a working area for the processor 21. The auxiliary storage device stores a program executed by the processor 21 and information transmitted and received to and from other computers. Note that the storage device 22 may include a cache memory provided in the processor 21. The communication I / F 23 is, for example, a network card or a communication module, and communicates with other computers via the network 5 based on a predetermined protocol. The UI 24 is an input / output device such as a touch panel, a keyboard, or a pointing device, for example. The UI 24 receives a user's operation and outputs information to the user. Note that the server 2 does not necessarily include the UI 24.

[0017] The vehicle 4 is a passenger car or the like and may perform automatic driving at any level. The vehicle 4 includes an in-vehicle device 40 that performs the processing according to the embodiment, an imaging device 44, a position sensor 45, and a receiving device 46. The in-vehicle device 40 is a computer and includes a processor 41, a storage device 42, and a communication interface (IF) 43. The processor 41 of the in-vehicle device 40 is an arithmetic processing device such as a CPU. The processor 41 performs each process according to the embodiment by executing a program. For example, the processor 41 transmits the image data output by the imaging device 44 to the server 2 at a predetermined timing via the communication IF 43. Also, the processor 41 reads out update data of the learned model from the data related to the data broadcast stored in the storage device 42 by the receiving device 46, and updates the learned model. The storage device 42 is a main storage device such as a RAM or a ROM, and an auxiliary storage device such as an HDD, an SSD, or a flash memory. The storage device 42 temporarily stores the program read by the processor 41 and the information to be processed, and secures the working area of the processor 41. The storage device 42 stores the learned model for performing image recognition. The communication IF 43 is, for example, a communication module, and communicates with other computers via the network 5 based on a predetermined protocol. Also, the imaging device 44 is a camera that converts light into an electrical signal by an image sensor using, for example, a CCD or a CMOS, creates image data, and outputs it. The imaging device 44 functions as a digital still camera or a digital video camera, continuously outputs image data, and stores it in, for example, the storage device 42 of the in-vehicle device 40. The position sensor 45 is a receiving device that receives, for example, a signal of a satellite positioning system (GNSS: Global Navigation Satellite System), calculates coordinates indicating the position of the position sensor 45, and outputs it as position information. The receiving device 46 includes an antenna for terrestrial digital broadcast, and receives, for example, a broadcast wave and stores data (carousel) related to the data broadcast in the storage device 42. Note that the receiving device 46 may be able to decode and reproduce the stream data constituting the video and audio.

[0018] The network 5 includes, for example, an IP (Internet Protocol) network, and devices connected to the network 5 can communicate based on a predetermined communication protocol. A part of the network 5 may be a telephone network (fixed telephone network or mobile communication network), an ad hoc network, an intranet, a VPN (Virtual Private Network), LAN (Local Area Network), a wireless LAN (Wireless LAN), a WAN (Wide Area Network ), or the Internet. )

[0019] FIG. 3 is a functional block diagram showing an example of the transmitter 3. The transmitter 3 includes a communication IF (Interface) 31, a storage device 32, an encoding unit 33, a multiplexing unit 34, and a transmission path encoding unit 35 and a terrestrial transmission antenna 36. These can be realized as an electric circuit, as software operating on a computer, or as a combination thereof. The transmitter 3 receives video, audio, and other data for data broadcasting, etc. from one or more broadcasting stations (not shown) and transmits them as a broadcast wave. Note that the network 5 may include an STL (Studio to Transmitter Link) line using microwaves or the like. Also, in the present embodiment, the transmitter 3 superimposes and transmits the update data transmitted from the server 2 via the network 5 on the broadcast wave. The communication IF 31 communicates with other computers via the network 5 based on a predetermined protocol. The storage device 32 includes a main storage device or an auxiliary storage device, and the storage device 32 stores video, audio, and other data received from the broadcasting station and the update data received from the server 2. The encoding unit 33 compresses video and audio. Also, the encoding unit 33 encodes the data for data broadcasting, for example, in the BML (Broadcast Markup Language) format . The multiplexing unit 34 multiplexes the compressed video, audio, and data and converts them into, for example, a TS (Transport Stream) signal. The transmission path encoding unit 35 performs modulation by a predetermined error correction encoding, OFDM (Orthogonal Frequency Division Multiplex), or the like. The terrestrial transmission antenna 36 is connected to the transmission path encoding unit 35 via an amplifier, for example. Also, the terrestrial transmission antenna 36 transmits the multiplexed broadcast signal, which is modulated by the transmission path encoding unit 35 and appropriately amplified so as to form a prescribed broadcast area, as radio waves. Note that the format of the broadcast wave is not particularly limited and may be ISDB-T, DVT-T, DAB / DMB, or the like.

[0020] <Machine Learning> FIG. 4 is a diagram for explaining an overview of machine learning. Note that the vehicle 4 in FIG. 4 may be any of the vehicles 4 shown in FIG. 1, or may be a vehicle different from these. As shown in FIG. 2, the vehicle 4 includes an imaging device 44 and images the periphery (for example, the front) of the vehicle 4. Note that position information indicating the position where the image data was imaged is recorded in association with the captured image data. Further, imaging date and time information indicating the date and time when the image data was imaged may be further recorded in association with the captured image data. For example, the position information and the like can be recorded in metadata such as Exif (Exchangeable image file format). Further, the vehicle 4 transmits transmission data 401 including the image data and the position information to the server 2 via the network 5. Note that the transmission data 401 may further include date and time information. Further, the vehicle 4 may repeat imaging and continuously transmit the transmission data 401 to the server 2. Note that the vehicle 4 may transmit the image data to, for example, a charging facility of an electric vehicle (EV), and the facility may transmit the image data collected from a plurality of vehicles 4 to the server 2.

[0021] Server 2 collects image data from one or more vehicles 4 and stores the image data captured at various locations and times in the storage device 22. Further, the server 2 extracts images captured within the target image collection area using the position information from the stored data 201, and performs annotation on the extracted data 202 based on, for example, a user's operation. For example, the target image collection area may be the same as any of the radio wave reach ranges 30 shown in FIG. 1 (which becomes the update target model area), or may be an area slightly different from the radio wave reach range 30 (for example, wider) within a range where the environment is somewhat similar. In addition, when date and time information is associated with the image data, the image may be further extracted using the date and time information. For example, based on the date and time information, it may be further segmented by season or by time zone associated with the outdoor brightness, and then annotation and machine learning may be performed. Also, weather information at that time in the area may be obtained based on the position information and the date and time information from a database that stores information on past weather, and annotation and machine learning may be performed for each weather condition.

[0022] When performing object detection from the image data, one or more correct labels representing the objects captured in the image data are input by the user and stored in association with the image data (and the object positions in the image). Specifically, the correct level is stored for each object image in each captured image. Specifically, for example, the identification information of each captured image, the position (area) information of each object in each captured image, and the object type (object name) of each object are stored in association. Further, the server 2 performs machine learning using the data 203 after annotation as learning data (input data: object image, correct data (label): object type), and creates a learned model 204 that has learned the relationship between the features of the object image data and the correct label.

[0023] When performing machine learning by DNN, the learned model 204 includes information representing the connection structure between layers including one or more nodes and the weights applied to each connection. Further, when detecting a plurality of types of objects, one learned model 204 may perform multi-label classification (input data is various object images, and correct data (labels) are learned by learned data of various object types in the learned model 204), or a plurality of learned models 204 that perform single-label classification may be created (input data is various object images, and correct data (labels) are learned by learned data of discrimination data as to whether or not it is the target object in the learned model 204). Further, by combining a plurality of learned models 204 that perform multi-label classification or single-label classification, it may be possible to detect a plurality of objects to be detected.

[0024] As described above, it is possible to create a learned model 204 that reflects the bias specific to the area in the characteristics of the image data captured in the target image collection area. Then, the created learned model 204 (connection structure and weights between nodes), or the weight information created to update the learned model 204 mounted on the vehicle 4 in the area is transmitted from the server 2 to the transmitter 3 as update data. Thereafter, these update data are transmitted from the transmitter 3 to each vehicle 4 in the broadcast area by a broadcast wave on which the update data are superimposed.

[0025] FIG. 5 is a processing flowchart showing an example of the learning process performed by the server 2. The learning process starts at an arbitrary timing based on, for example, a learning start operation by the model developer in a state where the image data collected from the vehicle 4 is stored in the storage device 22. It is started at an arbitrary timing based on a learning start operation by the developer or the like.

[0026] The processor 21 of the server 2 reads out the image data captured in the target image collection area from the storage device 22 (FIG. 5: Step S1). In this step, the processor 21 reads out one piece of image data whose associated position information is included in the target image collection area. For example, one record of the extraction data 202 captured in the target image collection area is read out from the accumulation data 201 in FIG. 4.

[0027] After step S1, the processor 21 annotates the object image data read in step S1 (FIG. 5: step S2). In this step, the processor 21, for example, displays the object image of the target in the extraction data 202, and based on the user's operation (correct label instruction operation), associates the correct label with the object image data being displayed to generate learning data and stores it in the storage device 22. For example, for the extraction data 202 in FIG. 4, one record of the data 203 after annotation is created. Note that the correct label is, for example, information representing the object type (object name) for the object image.

[0028] After step S2, the processor 21 determines whether there is unprocessed object image data (FIG. 5: step S3). If there is unprocessed object image data (object image data for which annotation is not completed) (step S3: YES), the process returns to step S1 and the processing is repeated. The processing of steps S1 to S2 is repeated, and for each object image in each captured image collected in the target image collection area, data 203 after annotation (that is, learning data) is created. Note that the learning data is appropriately divided into a training data set (training data) used to create a model and an evaluation data set (test data) used to evaluate the model.

[0029] On the other hand, if it is determined that there is no unprocessed image data (step S3: NO), the processor 21 performs machine learning on the relationship between the image data and the correct labels using the generated training data to create a trained model (FIG. 5: step S4). Machine learning can be performed using existing techniques such as backpropagation used in neural networks, such as so-called DNN. The trained model, for example, has one or more nodes in the output layer that correspond one-to-one to correct labels and outputs the probability of each correct label. In this step, a training data set is used to perform the training process, and the created model is evaluated using an evaluation data set. The evaluation can be performed by calculating an existing evaluation index such as the accuracy rate. If the accuracy rate does not exceed a desired standard (threshold), machine learning may be performed using additional training data or by changing the training data.

[0030] After step S4, the processor 21 determines whether there is any unprocessed update data corresponding to each model area to be updated (FIG. 5: step S5). The model area to be updated may be determined as appropriate, such as an area where a sufficient number of object images for learning data have been collected, or an area where a predetermined update period has elapsed since the previous update data was created. If it is determined that there is any unprocessed update data (step S5: YES), the processor 21 performs the process of creating one of the unprocessed update data. In order to create updated data, the image data read conditions and the learning target model are switched, and the process returns to step S1, where the processing for the unprocessed updated data is repeated.

[0031] On the other hand, if it is determined that there is no unprocessed update data (step S5: NO), the processor 21 transmits the update data for each trained model to the transmission station 3 corresponding to the area corresponding to each update data (FIG. 5: step S6), and ends the process. Each update data is transmitted to a transmission station whose radio wave reach includes the area to which the location information of the image data used to create it belongs. Note that the update data contains information about the model area to be updated, and a timestamp indicating the application time based on the shooting date and time of the captured image, in order to determine whether the update data is new or old. It may include information such as etc. Further, considering the uncertainty of the radio wave reach from the transmitter 3 (the terrain, buildings, weather, etc. within the area, and the radio wave reach is not stable due to the influence of the receiving performance and driving state of the in-vehicle device), it may include information indicating the area to which the update data should be applied (position information defining the area).

[0032] <Broadcast> The transmitter 3 encodes the update data received from the server 2 as part of the data broadcast of a broadcasting station with the update target model area as the broadcast area, for example, and repeatedly transmits it in a carousel manner. In this way, the vehicle 4 can receive the update data when the power of its in-vehicle device (broadcast receiver) is turned on and it is located within the radio wave reach of the transmitter 3.

[0033] <Update Process> FIG. 6 is a processing flowchart showing an example of the update process performed by the vehicle 4. The update process is continuously and repeatedly executed when the power of the in-vehicle device 40 is turned on. The processor 41 of the in-vehicle device 40 mounted on the vehicle 4 receives the broadcast wave from the transmitter 3 via the receiving device 46 (FIG. 6: step S11). In the system (in-vehicle device 40) of this example, the broadcast wave of terrestrial digital broadcast is received.

[0034] After step S11, the processor 41 reads out the update data from the received broadcast wave (FIG. 6: step S12). Specifically, the processor 41 restores, for example, the update data included in a predetermined data carousel (set to include the update data) as data broadcast content from the broadcast wave.

[0035] After step S12, the processor 41 determines whether there is updated data to be applied (FIG. 6: step S13). Specifically, the processor 41 determines whether the received broadcast wave contains updated data, whether the host vehicle 4 is located in the model area to be updated included in the updated data, and whether the current date and time is included in the application time included in the updated data. When the broadcast wave contains updated data, the host vehicle 4 is located in the model area to be updated, and the current date and time is included in the application time, it is determined that there is updated data to be applied. Since the updated data that can be received (acquired) is guaranteed to a certain extent to be the corresponding updated data by the radio wave reach range from the transmitter 3, it is also possible to perform the process of applying the received (acquired) updated data. When other conditions such as season, time zone, and weather are set in the updated data, the updated data for which the environmental conditions of the host vehicle 4 and other conditions are satisfied will be applied.

[0036] When it is determined that there is data to be applied (step S13: YES), the processor 41 determines whether the vehicle 4 is parked or stopped (FIG. 6: step S14). For example, when the speed of the vehicle 4 obtained via the vehicle speed sensor is zero, it is determined that the vehicle 4 is parked or stopped. The vehicle speed sensor is, for example, a vehicle speed pulse generator, and generates a pulse signal (vehicle speed signal) according to the rotation of the axle. The vehicle speed signal is transmitted to the in-vehicle device 40 via the ECU (Electronic Control Unit). When it is determined that the vehicle 4 is neither parked nor stopped (step S14: NO), the processor 41 repeats the determination in step S14 and waits for the vehicle 4 to park or stop. On the other hand, when it is determined that the vehicle 4 is parked or stopped (step S14: YES), the processor 41 applies the updated data to the learned model stored in the storage device 42 (FIG. 6: step S15). That is, it is preferable to perform the update process at the timing when the vehicle 4 stops so that the driving support can be continued during driving. When it is determined that there is no data to be applied (step S13: NO), or after step S15, the update process in FIG. 6 is terminated.

[0037] <Effect> As described above, the vehicle 4 (in-vehicle device 40) can acquire update data for the relevant area from the broadcast waves that can be received at the position where the host vehicle is located and apply it to the learned model. The reach of the broadcast waves is substantially limited to the broadcast areas of the broadcast stations in each area. Therefore, the vehicle 4 (in-vehicle device 40) can selectively obtain update data (learning data) for the AI model corresponding to the position during travel and update the AI model. And, for example, in the case of distributing update data via IP (Internet Protocol) network when traffic increases and the bandwidth is compressed when distributing update data through a network such as a work, such inconvenience does not occur if broadcast waves are used to transmit the update data as in this embodiment. Thus, according to this embodiment, it becomes possible to efficiently realize AI according to the area. Also, by performing machine learning using learning data including image data captured within the radio wave reach of the broadcast waves and learning data including image data showing the characteristics of the climate within the radio wave reach of the broadcast waves, update data matching the characteristics of the area corresponding to the radio wave reach from the transmitter 3 can be created, and the update data can be efficiently transmitted to the vehicle 4.

[0038] <Modification example> The updated data may be for changing the learned model according to application condition data such as time (season), time zone (ambient brightness), weather, vehicle type, etc. That is, the updated data may include, as application condition data indicating the application conditions of the data, in addition to the model area to be updated, time, time zone, weather, vehicle type, etc. In this case, the training data used for machine learning includes date and time information for determining time, time information for determining time zone, information indicating the weather of the area at the time of imaging, and information representing the type of vehicle that captured the image, and a learned model is created for each predetermined time, time zone, weather, or vehicle type. For example, the transmission data 401 shown in FIG. 4 includes at least any one of the date and time information, time information, and vehicle type information recorded at the time of imaging. Further, the accumulated data 201 shown in FIG. 4 may further include the date and time information, time information, and vehicle type information recorded in the transmission data 401, and weather information obtained from a predetermined database for the weather of the location indicated by the location information at the date and time. Also, in step S1 of FIG. 5, in addition to the location information, the image data extracted using at least any one of time, time zone, weather, and vehicle type is filtered. In this way, machine learning can be performed to create a learned model for proper use according to time, time zone, weather, vehicle type, etc.

[0039] Also, when the application condition data includes information on the time period to be applied, the transmission station 3 may broadcast the updated data only during the corresponding period. Further, when the application condition data includes time, time, or weather, the processor 41 of the vehicle 4 may apply the updated data in a timely manner and may also switch the learned model according to conditions such as the period, time zone, weather, etc. to be applied.

[0040] Also, the update data applied to the vehicle 4 may be changed according to the type of the vehicle 4. In this case, information indicating the type of the vehicle to which the update data should be applied is included in the update data and broadcasted. Based on this information, the processor 41 of the vehicle 4 determines whether to apply the update data to its own vehicle. For example, for a special vehicle such as a snowplow, the update data of the learned model created using the image data of a road with a relatively large amount of snow accumulation before snow removal may be applied.

[0041] Also, the update data transmitted by the transmission station 3 includes information indicating a storage location on the network, such as a URI (Uniform Resource Identifier), and the in-vehicle device 40 of the vehicle 4 may download the necessary update data via the network work 5. For example, for update data with a low usage frequency, such as update data for special vehicles, such a mode may be particularly adopted. In this way, the amount of data multiplexed on the broadcast wave can be reduced.

[0042] The update data is not only transmitted as data broadcasting by the broadcasting station, but also transmitted using the vacant band (white space) of the broadcast wave or the segment set for the update data, or transmitted at an appropriate timing in the engineering service using the engineering service. Also, the transmission station 3 includes a relay station in addition to the broadcast transmission station, and the relay station may transmit update data corresponding to the radio wave reach range from the relay station.

[0043] Further, the learned model may be created by federated learning. FIG. 7 is a diagram for explaining federated learning. In the example of FIG. 7, between the vehicle 4 (4G to 4J) and the server 2, there are computers 20 (20A, 20B) provided at a base such as an electric vehicle charging facility. And the vehicle 4 copies the image data to the computer 20 by predetermined communication, either wired or wireless, for example while charging. Also, the computer 20 performs machine learning as an edge terminal in federated learning. Note that the computer 20 may aggregate the image data into one computer 20 in the target image collection area or the like and perform machine learning. Also, the computer 20 may perform machine learning using the image data captured in the target image collection area as the learning target. The computer 20 creates a local model by machine learning, and the server 2 collects the parameters of the local model and creates a global model. Also, the created global model is transmitted from the transmission station 3 to the vehicle 4. And the vehicle 4 updates the learned model stored in the storage device 42. Also in this case, a part of the learned model may be updated, or the whole may be updated.

[0044] <Others> The configurations of the illustrated systems and devices are examples and are not limited to the above examples. For example, at least a part of the processing performed by the server 2, the transmission station 3, and the vehicle 4 may be shared and executed by a plurality of devices, or may be executed in parallel by a plurality of devices. Similarly, within a range where the results do not change, the order may be changed and executed, or may be executed in parallel. For example, in the processing shown in FIGS. 4 and 5, first, annotation may be performed on all the image data, and the image data captured in a predetermined area may be extracted therefrom. Also, the data structure shown in FIG. 4 is an example, and appropriately, the information may be normalized and stored separately in a plurality of tables, or may be non-normalized and stored together in one table. Also, the contents of the above-described embodiments and modifications can be implemented in combination.

[0045] Furthermore, the present invention includes a computer program that executes the above-described processing method, and a computer-readable recording medium on which the program is recorded. By causing the computer to read the recording medium and execute the recorded program, the above-described processing becomes possible. A computer-readable recording medium refers to a recording medium that accumulates information such as data and programs by an electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer. Examples of removable recording media from a computer include flexible disks, magneto-optical disks, optical disks, magnetic tapes, memory cards, and the like. Examples of recording media fixed to a computer include hard disk drives and ROMs.

Explanation of Signs

[0046] 1: Broadcasting system 2: Server, 21: Processor, 22: Storage device, 23: Communication interface (IF), 24: User interface (UI) 3: Transmission station, communication interface (IF), 32: Storage device, 33: Encoding unit, 34: Multiplexing unit, 35: Channel encoding unit, 36: Antenna 30: Radio wave reach range 4: Vehicle, 40: On-vehicle device, 41: Processor, 42: Storage device, 43: Communication interface (IF), 44: Imaging device, 45: Position sensor, 46: Receiving device 5: Network

Claims

1. A driving assistance system including a vehicle, a server device, and a broadcasting device, The server device image data collected from a target image collection area corresponding to the broadcast area of the broadcast wave is used as input data, and learning data is generated by adding correct answer data to the image data; training a learning model using the generated learning data to generate a trained model; Generate update data for model updating based on the generated trained model, outputting the generated update data; The broadcasting device Obtaining the update data; The acquired update data is superimposed on a broadcast wave of a digital broadcast having a specified broadcast area and transmitted; The vehicle is an image recognition device having a trained model for object detection; a driving assistance device that performs driving assistance based on object detection information by the image recognition device; Equipped with The image recognition device Extracting update data for a trained model of the image recognition device from the received digital broadcast wave; updating a trained model of the image recognition device based on the extracted update data; Driver assistance system.

2. An image recognition device that has a trained model for object detection and is mounted on a vehicle, Extracting update data for the trained model from a digital broadcast wave having a specified broadcast area; Updating the trained model based on the extracted update data Image recognition device.

3. The update data includes application condition data indicating application conditions for the update; The trained model is updated based on the update data in which the environment of the vehicle satisfies the application conditions. The image recognition device according to claim 2 .

4. The application condition is an application area, updating the trained model based on the update data for the application area that includes the position of the vehicle; The image recognition device according to claim 3 .

5. extracting, from the broadcast wave, storage location information indicating a storage location where the update data can be acquired via a network; acquiring the update data from the storage location via the network based on the storage location information; Update the trained model based on the update data acquired from the storage location The image recognition device according to claim 2 .

6. Determining whether to acquire the update data from the storage location according to the type of the vehicle. The image recognition device according to claim 5 .

7. An image recognition program having a trained model for object detection and executed by a computer mounted on a vehicle, the computer comprising: Extracting update data for the trained model from broadcast waves of a digital broadcast having a specified broadcast area; The trained model is updated based on the extracted update data. Image recognition program.

8. Image data collected from a target image collection area corresponding to the broadcast area of the broadcast wave is used as input data, and learning data is generated as correct answer data added to the image data; training a learning model using the generated learning data to generate a trained model; Generate update data for model updating based on the generated trained model, The generated update data is included in the broadcast data and transmitted. How to provide updated information.

9. On the computer, image data collected from a target image collection area corresponding to the broadcast area of the broadcast wave is used as input data, and learning data is generated as correct answer data added to the image data; training a learning model using the generated learning data to generate a trained model; Generate update data for updating the model based on the generated trained model, the update data being to be included in broadcast data and transmitted. Update information program for.

10. A method for providing update data for a trained model for object detection to an image recognition device having the trained model, comprising: An update data providing method for providing update data by superimposing the update data on broadcast waves of digital broadcasting having a specified broadcast area.

11. A vehicle equipped with an image recognition device having a trained model for object detection, and a driving assistance device that provides driving assistance based on object detection information by the image recognition device, The image recognition device Extracting update data for the trained model from a digital broadcast wave having a specified broadcast area; Updating the trained model based on the extracted update data vehicle.

Images