Data acquisition system, imaging device, and program

Abstract

The system collects information on target objects from images of the surrounding area captured by a moving object, while preventing the leakage of personal information. [Solution] In the data collection system, the imaging device 10 includes a camera, an image determination unit that determines the objects to be collected and privacy objects including personal information contained in the image, an image processing unit that processes the image, and an information output unit that can transmit information to a server. The image determination unit determines the objects to be collected and privacy objects contained in the image based on an image determination model learned using training images containing the objects to be collected and privacy objects. The image processing unit generates an image in which personal information cannot be identified by processing at least a portion of the privacy objects in the image so that the personal information of the privacy objects cannot be identified. The information output unit transmits the image in which personal information cannot be identified generated by the image processing unit and the information of the determined objects to be collected to the server, but does not transmit the image itself, which contains both the objects to be collected and privacy objects, to the server.

Description

Technical Field

[0001] The present invention relates to a data collection system using a moving body, and particularly to a data collection system, an imaging device, and a program for collecting information on a collection target object, such as road infrastructure information, from an image around the moving body captured by the moving body including the imaging device.

Background Art

[0002] Regarding aging, damage, particularly cracks, depressions, and hole problems on the road surface of roads in Japan, multiple issues and technical countermeasures have been taken. If damage to the road surface is left unattended, damage such as cracks may progress to the roadbed that serves as the base, and ultimately may lead to damage to the entire pavement structure, so early repair is required. However, repair work is costly and time-consuming, and it is particularly difficult for local governments with limited budgets to conduct appropriate inspections and repairs. For such inspections of road infrastructure, there are visual inspections and advanced inspection methods using laser equipment. For example, since July 2014, road administrators have been required to conduct close visual inspections of all bridges, tunnels, etc. once every five years, and a legal inspection system has been established to diagnose the soundness in four levels as inspection results. However, it is stipulated that the inspections should be carried out by those with the necessary knowledge and skills. However, the number of skilled technicians who can conduct inspections is limited, and the number of bridges and tunnels to be inspected is enormous, so the burden on road administrators has been significant. In addition, it is a subjective judgment by the inspector, and variations may occur in the results due to differences in individual standards. There is also a possibility that fatigue accumulates due to long inspection work, resulting in a scattered attention and affecting the results. On the other hand, inspections using a "pavement condition measurement vehicle" equipped with laser equipment can provide detailed and accurate data and enable the grasping of the deterioration state of the road. However, the introduction cost is high, maintenance costs are also required, and there are cases where measurements cannot be made because other vehicles and pedestrians become obstacles.

[0003] Patent Document 1 discloses a road information distribution system that can collect and distribute information on road conditions such as road surface conditions, fallen objects, and road construction. In the road information distribution system of Patent Document 1, a vehicle-side device mounted on a vehicle transmits video information captured by a camera that images the area around the vehicle and location information detected by a vehicle position detection device to a road information distribution server. The road information distribution server compares the received video information with reference video information in the video database that corresponds to the location information to determine whether there is a region of difference (change region) between the two images. If a change region exists, it distributes the location information and video information contained in the received information to the vehicle-side device as distributed road information. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2006-318356 [Overview of the project] [Problems that the invention aims to solve]

[0005] The road information distribution system described in Patent Document 1 can collect in real time the differences between video information captured by a camera on a vehicle-mounted device and reference video information such as road surface conditions, fallen objects, and road construction. By combining this with a large number of vehicles, it is possible to easily obtain road information over a wide area. However, collecting images over a wide area in public places may raise privacy issues, and it is necessary to properly protect the collected data and safeguard it from unauthorized access. The video database in Patent Document 1 stores reference video information that is an average of multiple videos captured at the same location, and moving objects such as people and vehicles are removed from the reference video information. Even with video information transmitted from vehicles, areas where changes are expected, such as vehicles ahead and people on the sidewalk, are masked and compared with the reference video information in the database, so as a result, personal information such as human faces and license plates are masked. However, if the data is illegally stolen during transmission from the vehicle-mounted device to the road information distribution server, personal information is not protected, and there is a possibility of privacy leakage. Furthermore, there is also a risk of information leakage from the road information distribution server.

[0006] Furthermore, in the road information distribution system described in Patent Document 1, the vehicle-side equipment constantly transmits video information and location information to the road information distribution server. However, this increases the amount of communication between the vehicle-side equipment and the road information distribution server, and at least a portion of the communication bandwidth is constantly occupied. Moreover, when combined with a large number of vehicles, the road information distribution server must receive and process a massive amount of video information, leading to overload due to the large amount of data processing and a decrease in response speed. Furthermore, exceeding the processing capacity may cause the server and system to crash.

[0007] Therefore, the present invention aims to provide a data collection system, an imaging device, and a program that can collect information on objects to be collected, such as road infrastructure information, from images of the surroundings of a mobile object captured by a mobile object including an imaging device, while preventing the leakage of personal information. [Means for solving the problem]

[0008] To solve the above problems, the data collection system of the present invention includes a mobile body including an imaging device and a data collection server connected to the imaging device via a network, and is a data collection system that collects information on objects to be collected in images captured by the imaging device, wherein the imaging device includes a camera that images the surroundings of the mobile body, an image determination unit that determines objects to be collected and privacy objects including personal information contained in the images captured by the camera, an image processing unit that processes the images captured by the camera, and an information output unit that can transmit information to the server via the network, and the image determination unit includes an image determination model learned using training images including objects to be collected and training images including privacy objects, and the camera captures The system determines the objects to be collected and the objects to be private contained in the image captured by the camera from the output result obtained by inputting the captured image into the image determination model, the image processing unit generates an image in which personal information cannot be identified by processing at least a portion of the objects to be private contained in the image captured by the camera so that personal information of the determined objects to be private cannot be identified, and the information output unit transmits the image in which personal information cannot be identified generated by the image processing unit and the information of the determined objects to be collected to the server, provided that the image captured by the camera contains both objects to be collected and objects to be private, but does not transmit the image containing both objects to be collected and objects to the server.

[0009] Furthermore, in the above data collection system, if the image captured by the camera includes objects to be collected but does not include objects subject to privacy, the information output unit may transmit the image captured by the camera and the information of the determined objects to be collected to the server.

[0010] Furthermore, in the above data collection system, the information output unit may temporarily store the image determination unit and the images processed by the image determination unit in storage and later transmit them to the server. Alternatively, the image determination unit may temporarily store all images processed by the image determination unit in storage, but transmit only the images processed by the image determination unit and the image processing unit that contain the objects to be collected to the server.

[0011] Furthermore, in the above data collection system, the image processing unit may process the region of the object or organism containing the determined privacy object, or it may process only the region of the privacy object in the image captured by the camera. In addition, the image processing unit may process the image in a way that makes it possible to distinguish the type or characteristics of the object or organism containing the determined privacy object.

[0012] Furthermore, the imaging device of the present invention is a device that can be held on a mobile body and connected to a data collection server via a network, the imaging device includes a camera that images the area around the mobile body, an image determination unit that determines the objects to be collected and privacy objects including personal information contained in the image captured by the camera, an image processing unit that processes the image captured by the camera, and an information output unit that can transmit information to the server via the network, the image determination unit includes an image determination model that has been trained using training images including objects to be collected and training images including privacy objects, and obtained by inputting the image captured by the camera into the image determination model. The system determines the objects to be collected and the objects to be privately collected in the image captured by the camera based on the output results. The image processing unit generates an image in which personal information cannot be identified by processing at least a portion of the objects to be privately collected in the image captured by the camera so that personal information of the determined objects to be private cannot be identified. The information output unit transmits the image in which personal information cannot be identified generated by the image processing unit and the information of the determined objects to be collected to the server, provided that the image captured by the camera contains both objects to be collected and objects to be privately collected, but does not transmit the image containing both objects to be collected and objects to the server.

[0013] Furthermore, the present invention includes a computer for an imaging device that can be held in a mobile body and connected to a data collection server via a network, and which includes a camera that images the area around the mobile body and an information output unit that can transmit information to the server via the network, an image determination means for determining objects to be collected and privacy objects including personal information contained in the image captured by the camera, and an image processing means for processing the image captured by the camera. The program functions as follows: The image determination means includes an image determination model trained using training images containing objects to be collected and training images containing private objects; the image determination means determines objects to be collected and private objects contained in the image captured by the camera from the output result obtained by inputting the image captured by the camera into the image determination model; the image processing means generates an image in which personal information cannot be identified by processing at least a portion of the private objects in the image captured by the camera so that personal information of the determined private objects cannot be identified; and the information output unit is operated such that, if the image captured by the camera contains both objects to be collected and private objects, it transmits the image in which personal information cannot be identified generated by the image processing unit and the information of the determined objects to be collected to the server, but does not transmit the image containing both objects to be collected and private objects itself to the server.

[0014] Furthermore, the road infrastructure information collection system of the present invention includes an imaging device, a mobile body that moves along a road, and a data collection server connected to the imaging device via a network, and is a road infrastructure information collection system that collects road infrastructure information in an image captured by the imaging device, wherein the imaging device includes a camera that images the area around the mobile body, an image determination unit that determines road infrastructure information and privacy-related objects including personal information contained in the image captured by the camera, an image processing unit that processes the image captured by the camera, and an information output unit that can transmit information to the server via the network, and the image determination unit includes an image determination model that has been trained using training images containing road infrastructure information and training images containing privacy-related objects, and the camera The system determines road infrastructure information and privacy-sensitive objects contained in the image captured by the camera from the output results obtained by inputting the image captured by the camera into the image determination model, the image processing unit generates an image in which personal information cannot be identified by processing at least a portion of the privacy-sensitive objects in the image captured by the camera so that personal information of the determined privacy-sensitive objects cannot be identified, and the information output unit transmits the image in which personal information cannot be identified generated by the image processing unit and the determined road infrastructure information to the server if the image captured by the camera contains both road infrastructure information and privacy-sensitive objects, but does not transmit the image containing both the road infrastructure information and privacy-sensitive objects itself to the server.

[0015] Furthermore, the imaging device of the present invention includes a camera, an image determination unit that determines objects to be collected and privacy objects including personal information contained in an image captured by the camera, an image processing unit that processes the image captured by the camera, and an information output unit. The image determination unit includes an image determination model that has been trained using training images containing objects to be collected and training images containing privacy objects. The image determination unit determines objects to be collected and privacy objects contained in the image captured by the camera from the output result obtained by inputting the image captured by the camera into the image determination model. The image processing unit generates an image in which personal information cannot be identified by processing at least a portion of the privacy objects in the image captured by the camera so that personal information of the determined privacy objects cannot be identified. The information output unit outputs the image in which personal information cannot be identified generated by the image processing unit and information on the determined objects to be collected, at least when the image captured by the camera contains both objects to be collected and privacy objects. [Effects of the Invention]

[0016] The data acquisition system, imaging device, and program of the present invention include an imaging device included in a mobile body, an image determination unit that determines objects to be collected and privacy objects including personal information contained in an image captured by a camera, and an image processing unit that processes the image captured by the camera. The image determination unit determines objects to be collected and privacy objects contained in an image captured by the camera, and generates an image in which personal information cannot be identified by processing at least a portion of the privacy objects in the image captured by the camera so that personal information of the determined privacy objects cannot be identified. If the image captured by the camera contains both objects to be collected and privacy objects, the image in which personal information cannot be identified generated by the image processing unit and information on the determined objects to be collected are transmitted to the server, but the image containing both objects to be collected and privacy objects itself is not transmitted to the server. Furthermore, in the present invention, the image determination unit includes an image determination model that has been trained using training images containing objects to be collected and training images containing privacy objects including personal information. Therefore, the image determination model can determine both objects to be collected and privacy objects in an image captured by a camera, thereby increasing the processing speed. Furthermore, since the server outputs images that do not allow for the identification of personal information, even if the server is illegally accessed or if information is stolen during transmission, personal information will not be leaked. In addition, since the imaging device determines the information of the objects to be collected, rather than the server determining the objects, the server load can be reduced. Moreover, the images that do not allow for the identification of personal information acquired by the server include a real-time background image of the moving object at the time of image acquisition, excluding the processed parts of the privacy object. Therefore, information that is not the object to be collected and is included in the images that do not allow for the identification of personal information (background image) can be collected later by image analysis, making the collected information suitable for secondary use. Other effects will be explained in the embodiments. [Brief explanation of the drawing]

[0017] [Figure 1] Schematic diagram of the data acquisition system of the present invention [Figure 2] An example of an image captured by a camera [Figure 3] An example of an image showing the state where an object in an image is detected [Figure 4] An example of an image in which personal information has been anonymized and cannot be identified [Figure 5] An example of an image in which personal information has been anonymized and cannot be identified [Figure 6] An example of a flowchart of the image processing of the present invention [Figure 7] Another example of a flowchart of the image processing of the present invention [Figure 8] Schematic configuration diagram of another embodiment of the imaging device of the present invention

Mode for Carrying Out the Invention

[0018] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a schematic configuration diagram of a data collection system 1 of the present invention. The data collection system 1 includes a moving body 2 (2a, 2b, 2c) including imaging devices 10 (10a, 10b, 10c), and a data collection server 4 connected to the imaging devices 10 via a network 3, and may further include a non-volatile storage 5. Each imaging device 10 includes a camera 11, an image determination unit 12, an image processing unit 13, and an information output unit 14, and may further include a non-volatile storage 15. The image determination unit 12 includes an image determination model 16.

[0019] The data collection system 1 is a system that collects image information captured by the imaging device 10 and information on the collection target objects in the image. For example, it can be used in a road infrastructure information collection system that collects road infrastructure information as the collection target object. Road infrastructure information includes, for example, road condition information (such as road paving condition, necessity for repair (cracks, depressions, etc.), accident information, construction information, etc.), traffic volume data (such as peak-hour traffic conditions, congestion information, etc.), traffic regulation information (such as road closures, speed limits, traffic signal information, signs, etc.), safety information (such as accident occurrence locations, dangerous locations, guardrails, etc.). Road infrastructure information may be used for road management by road administrators (such as the national government, local governments, etc.), may be used to provide road information to drivers, or may be used as road information for autonomous driving. In addition to road infrastructure information, it can also be used to collect various information included in the movement route of the moving body 2. For example, as the collection target object, there are gasoline prices, parking lot information (such as prices, congestion levels, etc.), store information (such as restaurants, convenience stores, etc.), advertisements, crowd information, etc., and these may be collected using the data collection system of the present invention. Also, these information may be collected by secondarily using the image information including the personal information non-identifiable images collected by the data collection system of the present invention. For example, information such as gasoline prices, parking lot information (such as prices, congestion levels, etc.), store information (such as restaurants, convenience stores, etc.), advertisements, crowd information, etc. included in the image information collected for road infrastructure information for road management may be collected by performing image analysis later.

[0020] An image that makes personal information unidentifiable is an image that has undergone personal information de-identification processing, in which at least a portion of the private object in the image captured by the camera is processed so that personal information of the private object containing personal information cannot be identified. In the Personal Information Protection Act, personal information is defined as information relating to a living individual that can identify a specific individual by name, date of birth, or other descriptions contained in the information (including information that can be easily collated with other information to identify a specific individual) or a "personal identification code." However, in the present invention, the definition is further expanded to include information that can identify a specific individual regardless of whether they are alive or dead, information that can identify a specific individual by collation with other information, and even license plate numbers that can identify a specific vehicle, and among these, those predetermined as protected items. For example, personal information includes a person's face, name, and license plate number. Private objects include, for example, a person's face, name tag, sign, and license plate.

[0021] The mobile body 2, which includes the imaging device 10, may be, for example, a vehicle 2a equipped with a drive recorder 10a, a drone 2b equipped with a digital camera 10b, or a human 2c with a smartphone 10c, as shown in Figure 1. A vehicle is a wheeled vehicle and includes automobiles, motorcycles, bicycles, and robots with autonomous driving functions. In addition, the mobile body 2 may be an airplane, a ship, or a living organism other than a human (dog, cat, bird, fish), as long as it is equipped with an imaging device 10 such as a drive recorder, smartphone, or digital camera. In particular, when collecting road infrastructure information, it is preferable that the mobile body (vehicle, human, etc.) is moving on the road.

[0022] Network 3 can utilize various networks, such as the Internet, wired LAN (Local Area Network) or wireless LAN, mobile communication networks, and Bluetooth®.

[0023] The data collection server 4 is a computer connected to the network 3 that collects image information and information on objects to be collected within the images captured by the imaging device 10. Examples of objects to be collected include road infrastructure information, gasoline prices, parking information, store information, advertisements, and crowd information. The information on objects to be collected and the image information stored in the data collection server 4 may be further reused. Although the information on objects to be collected is identified in the imaging device 10, the data collection server 4 may have image analysis means for analyzing the image information (for example, an image analysis model trained on information to be reused). The information on objects to be collected and the image information collected by the data collection server 4 are stored in the storage 5.

[0024] Storage 5 is a non-volatile storage medium connected to network 3 for recording and storing data, and can be a hard disk drive (HDD), flash memory, optical disc drive, etc. Storage 5 is the internal storage medium of data collection server 4 and may be connected to the network via the server, or it may be online storage on the cloud.

[0025] The imaging device 10 includes a camera 11 that captures images of the surroundings, an image determination unit 12 that analyzes images captured by the camera, an image processing unit 13 that processes images captured by the camera, and an information output unit 14 that can be connected to a network. It can be held on a mobile body 2 and connected to a data collection server via a network. Examples of imaging devices 10 include digital cameras, drive recorders, smartphones, personal information terminals, and notebook personal computers. The imaging device 10 includes a control computer and, as hardware, a CPU that performs calculations to realize various control functions, volatile RAM (Random Access Memory: e.g., SRAM, DRAM, etc.) that serves as a work area for calculations, and non-volatile storage 15 (e.g., hard disk, flash memory, etc.) that stores various data. Programs pre-stored in the storage 15 are read into the RAM, and various processes (e.g., image determination, image processing) are executed by the CPU performing calculations according to the program. The non-volatile storage 15 may be built-in or it may be a detachable, portable device (e.g., SD memory card, USB memory). The imaging device 10 may also include other hardware, such as sensors for acquiring location information (e.g., GPS), sensors for collecting information other than images (e.g., microphone, accelerometer, thermometer, hygrometer, speedometer), and input means (e.g., touch panel, switch).

[0026] Camera 11 consists of an optical system (lens) that collects and focuses light, an image sensor (such as a CCD (Charge-Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor)) that converts light into electrical signals, and a shutter that controls the time it takes for light to reach the image sensor. Camera 11 is used to image the area around a moving object; for example, if the moving object is a vehicle, it is positioned facing outwards. The number and arrangement of cameras 11 may be changed as appropriate. The image captured by camera 11 is called the "original image".

[0027] Figure 2 is an example of the original image 20, which includes a road collapse 21, a sign 22, a man 23, a woman 24, and a vehicle 25. In this data collection system, where the collected object is road infrastructure information and the privacy objects are human faces and license plates, the image 20 in Figure 2 includes the road collapse 21 and sign 22 as road infrastructure information, and the privacy objects include the face 23a of the man 23, the face 24a of the woman 24, and the license plate 25a of the vehicle 25.

[0028] Furthermore, the frame rate (fps: frames per second) of the camera 11 is preferably such that it can capture the surrounding scenery without excess or deficiency, in accordance with the speed of the moving object. If the moving object is a vehicle, it is preferable that the frame rate be 16fps or higher, and more preferably 30fps or higher. When the moving object is traveling at 60 km / h, the distance traveled in one second is approximately 16.7 m, and with 16fps, images can be captured at intervals of approximately 1 m. Furthermore, when the moving object is traveling at 100 km / h, the distance traveled in one second is approximately 27.8 m, and with 30fps, images can be captured at intervals of approximately 0.9 m. By increasing the frame rate, the imaging interval can be shortened, making it possible to more reliably identify the object to be collected. However, increasing the frame rate may require a high-performance image sensor and CPU, and the image quality of each frame may deteriorate, potentially resulting in misidentification or omission of the object to be collected. For this reason, a configuration in which the frame rate is variable according to the speed of the moving object is also acceptable. For example, the frame rate may be changed according to the speed of the moving object so that images can be captured at intervals of 0.5 m to 5 m.

[0029] The image determination unit 12 determines the objects to be collected and the objects to be privately shared contained in the original image captured by the camera 11 from the output results obtained by inputting the original image captured by the camera 11 into the image determination model 16, by having the CPU execute a program stored in the storage 15.

[0030] The image recognition model 16 can be constructed by training it using machine learning (deep learning is particularly preferred) with a dataset that includes training images containing objects to be collected, training images containing private objects, and the correct labels of the objects to be collected and private objects contained in the training images. The machine learning may be supervised or semi-supervised. The image recognition model 16 of the present invention, having been trained in this way, can detect the regions of objects (objects to be collected and private objects) contained in the image when an image is input, and output them along with the type of object detected, thereby being able to determine both objects to be collected and private objects at once. The type of object only needs to be distinguishable as to whether it is an object to be collected or a private object, but it may also be a label of a specific object.

[0031] Figure 3 is an example of the output result obtained by inputting the original image 20 from Figure 2 into the image recognition model 16, and shows image 30 in which objects (objects to be collected and privacy objects) in the original image 20 have been detected. In image 30, the image recognition model 16 has detected the road collapse 21, which is an object to be collected, in frame 31, and the sign 22 in frame 32, while the face 23a of the man 23, which is a privacy object, is detected in frame 33, the face 24a of the woman 24, in frame 34, and the license plate 25a of the vehicle 25 in frame 35. In Figure 3, the area of ​​the object is identified by enclosing the area of ​​the object in rectangular frames 31-35, but it may also be identified by other representations (for example, enclosing it in a circular frame, displaying a combination of coordinates in the image, displaying a type (label), etc.), or a combination of these may be used.

[0032] Depending on the speed of the moving object 2 and the frame rate of the camera 11, the same object (object to be collected and object to be kept private) may be captured in multiple frame images. If the object is determined to be different in each frame image, a discrepancy will arise between the actual object and the determined object. In particular, for objects to be collected (e.g., sinkholes), if the objects to be collected determined in each frame image are detected as different objects, the object will be detected as many times as there are frame images in which it is captured, resulting in the storage and / or transmission of information for multiple objects to be collected. For this reason, it is preferable for the image determination unit 12 to perform tracking processing to track the same object by referring to the preceding and succeeding frame images. When tracking processing is used, even if the same object is captured in multiple frame images, it can be determined as a single object. Furthermore, when tracking processing is used, once an object is found, the search can be conducted from its surroundings, thus reducing false detections and failures to detect the object.

[0033] The image processing unit 13 generates an image in which personal information cannot be identified by processing at least a portion of the private object in the original image captured by the camera, by having the CPU execute a program stored in the storage 15, so that the personal information of the private object determined by the image determination unit 12 cannot be identified. The personal information de-identification process can be performed by a combination of one or more processes, such as mosaic processing in which the color and brightness change in a mosaic pattern at regular pixel intervals, blurring processing which averages with the color of the surrounding pixels, masking processing which fills with a single color or multiple colors, wireframe processing which creates an image indicated by outlines, or processing which replaces with an abstract illustration.

[0034] The image processing unit 13 may process the area of ​​the object or living organism that contains the determined privacy object. If only the area of ​​the privacy object is processed, it may still be possible to identify an individual or vehicle based on a combination of features in the unprocessed parts. For example, it may be possible to identify an individual based on a combination of a person's gender, body shape, hairstyle, clothing, luggage, etc., or to identify the owner of a moving object based on a combination of its shape, type, color, pattern, decoration, etc. For this reason, it is preferable from the standpoint of protecting personal information to process almost the entire area of ​​the object or living organism that contains the privacy object.

[0035] Figure 4 is an example of an image 40 processed by the image processing unit 13, which processes the area of ​​an object or living being containing a privacy object as determined in image 30 of Figure 3. In image 40, the area 43 surrounding the entire face 23a of the man 23, the area 44 surrounding the entire face 24a of the woman 24, and the area 45 surrounding almost the entire license plate 25a of the vehicle 25 are masked. The masking in Figure 4 has a transparency of 10%, so personal information cannot be identified, but the type of object or living being (human, automobile) that is masked can be determined to some extent. Furthermore, in image 40, the label 41 "sinkhole" is attached to the frame 31 of the road sinkhole 21, which is the object to be collected, and the label 42 "sign" is attached to the frame 32 of the sign 22.

[0036] Furthermore, the image processing unit 13 may process only the area of ​​the private object in the image captured by the camera. This could be the entire area of ​​the private object, or only a part of the area that identifies personal information. Examples of areas that identify personal information include parts of a human face (eyes, eyes and nose, mouth, etc.) and parts of a car's license plate (part of the numbers). In these cases, because the processed area is narrow, the amount of information that can be grasped from the actual image information increases. For this reason, from the perspective of information gathering, it is preferable to process only the area of ​​the private object.

[0037] Figure 5 is an example of an image 50 obtained by processing only the areas of privacy-sensitive objects identified in image 30 of Figure 3 in the image processing unit 13. In image 50, the area 53 surrounding the face 23a of the man 23 identified in image 30 is masked, the area 54 surrounding the face 24a of the woman 24 is masked, and the area 55 surrounding the license plate 25a of the vehicle 25 is masked. Furthermore, in image 50, the label 41 "sinkhole" is attached to the road sinkhole 21, which is an object to be collected, and the label 42 "sign" is attached to the sign 22.

[0038] The image processing unit 13 may process the image to make it discernible the type or characteristics of the object or organism containing the determined privacy object (for example, the gender, body shape, hairstyle, clothing, luggage, etc. of a person; the type, shape, make, color, pattern, decoration, etc. of a vehicle). As shown in Figure 5, discernibility is possible when only the area of ​​the privacy object is processed, but even when processing the entire area of ​​the object or organism containing the privacy object, it can be processed to be discernible by applying semi-transparent masking as shown in Figure 4, replacing the object or organism containing the privacy object with an abstract illustration, or adding text by verbalizing it. In this case, the type or characteristics of the privacy object can also be used as the object to be collected, and for example, traffic congestion information, crowd information, etc. can also be collected.

[0039] The information output unit 14 can be connected directly or indirectly to the network 3 and transmits information to the data collection server 4 via the network 3. The information output unit 14, which has a communication function, may be directly connected to the network, or the information output unit 14 may be connected to another device with a communication function and indirectly connected to the network via the communication function of the other device. The information output unit 14 transmits to the server 4 the image that does not allow identification of personal information, generated by the image processing unit 13, and the information of the determined object to be collected, if the original image captured by the camera 11 includes both the object to be collected and the object to be private. However, it does not transmit the original image itself, which includes both the object to be collected and the object to be private, to the server 4. The information output unit 14 transmits to the server 4 the original image and the information of the determined object to be collected, if the original image includes the object to be collected but does not include the object to be private. The information output unit 14 may or may not transmit the image to the server 4 if the original image does not include the object to be collected. By configuring the system to send images to server 4 only if the source image contains the object to be collected, and not send images if the source image does not contain the object to be collected, the amount of data transmitted to network 3 can be reduced. When transmitting an image that does not contain the object to be collected, if the source image contains a private object, an image that does not allow for the identification of personal information is transmitted, and the source image itself is not transmitted; if the source image does not contain a private object, the source image is transmitted. The information output unit 14 has multiple output destinations (server and storage, server and display device), and the output content may be changed depending on the output destination.

[0040] It is preferable that the information output unit 14 can output and store images and information in a non-volatile storage 15 inside the imaging device 10 and / or in an external non-volatile storage 5. In particular, it is preferable to output and store images and information in a non-volatile storage 15 inside the imaging device 10. The information output unit 14 may store the results of image processing in the non-volatile storage 5 and later transmit them to the server 4. If the original image captured by the camera 11 is an image that includes both the object to be collected and the object to be kept private, the information output unit 14 stores the image that does not allow for the identification of personal information generated by the image processing unit 13 and the information of the object to be collected that has been determined in storage, but it is preferable that the original image itself, which includes both the object to be collected and the object to be kept private, in storage. If the original image is an image that includes the object to be collected but does not include the object to be kept private, the information output unit 14 stores the original image and the information of the object to be collected that has been determined in storage. If the original image includes the object to be kept private, it is preferable that the information output unit 14 stores the image that does not allow for the identification of personal information without storing the original image itself. However, in cases such as when recording as evidence for accidents or crime prevention, like with a dashcam, the original image containing private information may be saved. The imaging device 10 temporarily records the original image captured by the camera 11 in its internal volatile memory and processes it with the image determination unit 12 and the image processing unit 13. If only the processed image (an image where personal information cannot be identified if the original image contains private information, or the original image if the original image does not contain private information) is saved, the information temporarily recorded in the volatile memory will be erased or overwritten with other information, and the original image will not be saved. In addition, the information output unit 14 may display information about the collected objects that are saved and / or transmitted on a screen output to a display device.

[0041] The information on collected objects stored and / or transmitted includes the coordinates of the collected object within the image (for example, the coordinates of the diagonal vertices of a rectangle enclosing the collected object), and may also include the type of collected object determined (including a label). In addition, the position information (latitude, longitude) of the mobile body 2 at the time the collected object was imaged may be obtained from a sensor (such as GPS) and included in the information on the collected object. The image information stored and / or transmitted is an image that does not allow for the identification of personal information if the original image contains privately owned objects, and the original image if it does not contain privately owned objects.

[0042] [Image processing] Figure 6 is an example of a flowchart of the image processing of the present invention. As shown in Figure 6, the computer of the imaging device inputs the original image captured by the camera 11 to the image judgment model 16 (S61), and determines whether or not an object has been detected in the output result of the image judgment model 16 (S62). If no object is detected (No in S62), the original image does not contain any objects to be collected or any privacy objects, so the image processing is terminated, and the original image is sent to the server 4 or saved to non-volatile storage and terminated (S67). If an object is detected (Yes in S62), the type of the detected object (or one of them if multiple objects are detected) is determined (S63). If the object is an object to be collected (bottom of S63), information about the object to be collected is acquired (S64). If the object is a privacy object (right of S63), the image processing unit 13 performs personal information despecification processing (S65). Then, it is determined whether or not other objects have been detected in the original image (S66). If an object is detected (Yes in S66), the type of object detected is determined again (S63). If there are no other objects (No in S66), the image processing is terminated, the image processing results are obtained, and the process is terminated by sending them to Server 4 or saving them to non-volatile storage (S67). The image processing results are obtained as follows: if the original image contains both the object to be collected and the object to be protected from privacy, an image with personal information anonymized and information on the object to be collected is obtained; if the original image contains only the object to be collected, the original image and information on the object to be collected are obtained; and if the original image contains only the object to be protected from privacy, an image with personal information anonymized and information on the object to be protected from privacy is obtained. In addition, in S67, the image processing results may be saved to non-volatile storage and sent to Server 4 later.

[0043] Figure 7 shows another example of the image processing flowchart of the present invention. Steps S61 to S66 in Figure 7 are the same as steps S61 to S66 in the flowchart of Figure 6. After completing the image processing, the original image obtained in step S62 and the image processing result obtained in step S66 are stored in non-volatile storage (S71). It is determined whether or not the image processing result contains information about the object to be collected (S72). If it contains information about the object to be collected (Yes in S72), the information about the object to be collected and its image are sent to the server 4 (S73). If it does not contain information about the object to be collected (No in S72), the process is terminated (S74). In the image processing flowchart of Figure 7, the information output unit 14 stores all images and information about the object to be collected in storage after processing by the image determination unit 12 and the image determination unit 13. However, the server is only sent images and information about the object to be collected if they contain the object to be collected. This configuration not only reduces the amount of data communication on the network 3, but also allows for secondary use of the images stored in storage.

[0044] [program] The program of the present invention causes the computer of an imaging device 10, which includes a camera 11 and an information output unit 14, to function as an image determination means for determining objects to be collected and privacy objects including personal information contained in the image captured by the camera 11, and as an image processing means for processing the image captured by the camera 11, and causes the information output unit to perform predetermined operations. The image determination means of the program of the present invention has the function of an image determination unit 12, and the image processing means has the function of an image processing unit 13, and can perform the above-described image processing.

[0045] [Other embodiments of imaging devices] Figure 8 is a schematic diagram of another embodiment of the imaging device of the present invention. The imaging device 80 includes a camera 81 that images the surroundings, an image determination unit 82 that determines the objects to be collected and the privacy objects, including personal information, contained in the image captured by the camera, an image processing unit 83 that processes the image captured by the camera, and an information output unit 84. The image determination unit 82 includes an image determination model 86. This embodiment includes an image determination model trained using training images containing objects to be collected and training images containing privacy objects, and focuses on the function of determining the objects to be collected and the privacy objects contained in the image captured by the camera from the output results obtained by inputting the image captured by the camera into the image determination model, thereby providing an imaging device 80 that can be applied not only to the above data collection system but also to more general applications. For example, the imaging device 80 does not have to be installed on a mobile device, it may be fixed like a security camera, or it may be portable. Also, the information output unit 84 may not be connected to a network, and information may be provided via storage (e.g., hard disk, flash memory, etc.). Of course, by installing the imaging device 80 on a mobile device and connecting the information output unit 84 to a data acquisition server via a network, it can also be used as an imaging device for a data acquisition system. Examples of imaging devices 80 include digital cameras, drive recorders, smartphones, personal digital assistants, and notebook computers. The imaging device 80 includes a control computer and, as hardware, a CPU that performs calculations to realize various control functions, volatile RAM (Random Access Memory: e.g., SRAM, DRAM, etc.) which serves as a workspace for calculations, and non-volatile storage (e.g., hard disk, flash memory, etc.) for storing various data. Programs pre-stored in the storage are read into the RAM, and various processes (e.g., image judgment, image processing) are performed by the CPU executing calculations according to the program. The non-volatile storage may be built-in or it may be a removable, portable device (e.g., SD memory card, USB memory).The imaging device 80 may also include other hardware, such as sensors for acquiring location information (e.g., GPS), sensors for collecting information other than images (e.g., microphone, accelerometer, thermometer, hygrometer, speedometer), and input means (e.g., touch panel, switch).

[0046] The camera 81 consists of an optical system (lens) that collects and focuses light, an image sensor (such as a CCD (Charge-Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor)) that converts light into electrical signals, and a shutter that controls the time it takes for light to reach the image sensor.

[0047] The image determination unit 82 determines the objects to be collected and the objects to be privately shared contained in the original image captured by the camera 81 based on the output results obtained by inputting the original image captured by the camera 81 to the image determination model 86 when the CPU executes a program.

[0048] The image recognition model 86 can be constructed by training it using machine learning (deep learning is particularly preferred) with a dataset that includes training images containing objects to be collected, training images containing private objects, and the correct labels of the objects to be collected and private objects contained in the training images. The machine learning may be supervised or semi-supervised. The image recognition model 86 of the present invention, having been trained in this way, can detect the regions of objects (objects to be collected and private objects) contained in the image when an image is input, and output them along with the type of object detected, thereby being able to determine both objects to be collected and private objects at once. The type of object only needs to be distinguishable as to whether it is an object to be collected or a private object, but it may also be a label of a specific object.

[0049] The image processing unit 83 generates an image in which personal information cannot be identified by processing at least a portion of the private object in the original image captured by the camera, using a program stored in storage via the CPU. Personal information de-identification processing can be performed by a combination of one or more processes, such as mosaic processing where the color and brightness change in a mosaic pattern at regular pixel intervals, blurring processing where the color is averaged with the color of the surrounding pixels, masking processing where the image is filled with a single color or multiple colors, wireframe processing where the image is shown with outlines, or processing where the image is replaced with an abstracted illustration. The image processing unit 83 may process the region of an object or living organism that includes the determined private object, or it may process only the region of the private object in the image captured by the camera. The image processing unit 83 may also process the image in a way that makes it possible to identify the type or characteristics of the object or living organism that includes the determined private object (for example, the gender, body shape, hairstyle, clothing, luggage, etc. of a human, or the type, shape, model, color, pattern, decoration, etc. of a vehicle).

[0050] The information output unit 84 outputs information to other devices, and may output to other devices (display devices, storage) inside or outside the imaging device, or it may be directly or indirectly connected to a network and output to other devices (servers, online storage, display devices) via the network. The information output unit 84 outputs the image in which personal information cannot be identified and the information of the determined object to be collected, if the original image captured by the camera 81 is an image that includes both the object to be collected and the object to be private. The information output unit 84 outputs the original image and the information of the determined object to be collected, if the original image includes the object to be collected but does not include the object to be private. The information output unit 84 does not have to output if the original image does not include the object to be collected but includes the object to be private, but if it does output, it outputs the image in which personal information cannot be identified, generated by the image processing unit 83. The information output unit 84 does not have to output if the original image does not include the object to be collected or the object to be private, but if it does output, it outputs the original image.

[0051] The information output unit 84 may have multiple output destinations, and the output content may be changed depending on the output destination. For example, it may have a first output destination (such as a drive recorder storage or in-car monitor) that outputs the original image as is, and a second output destination (such as an information collection storage or data collection server) that outputs an image in which personal information cannot be identified if the image contains a private object. In addition, if the original image contains a private object, the information output unit 84 may save an image in which personal information cannot be identified instead of saving the original image itself. The imaging device 80 temporarily records the original image captured by the camera 81 in volatile memory inside the imaging device 80, and processes it with the image determination unit 82 and the image processing unit 83. However, if only the processed image (an image in which personal information cannot be identified if the original image contains a private object, or the original image if the original image does not contain a private object) is saved, the information temporarily recorded in the volatile memory is erased or overwritten with other information, and the original image is not saved. [Explanation of symbols]

[0052] 1. Data Acquisition System 2 Mobile Units 3 Network 4 Data Collection Servers 5.15 Storage 10 Imaging device

Claims

1. A mobile body including an imaging device, The imaging device and a data acquisition server connected via a network are included. A data acquisition system for collecting information on objects to be collected in an image captured by the aforementioned imaging device, The imaging device includes a camera that images the area around the moving object, an image determination unit that determines the objects to be collected and the objects to be private, including personal information, that are included in the image captured by the camera, an image processing unit that processes the image captured by the camera, and an information output unit that can transmit information to the server via the network. The image determination unit includes an image determination model trained using training images containing objects to be collected and training images containing privacy objects, and determines the objects to be collected and privacy objects contained in the image captured by the camera from the output result obtained by inputting the image captured by the camera into the image determination model. The image processing unit generates an image in which personal information cannot be identified by processing at least a portion of the privacy object in the image captured by the camera so that the personal information of the determined privacy object cannot be identified. The information output unit transmits to the server, at least when the image captured by the camera includes both the object to be collected and the object to be protected from privacy, the image that does not allow for the identification of personal information generated by the image processing unit and the information of the object to be collected that has been determined, but does not transmit the image itself that includes both the object to be collected and the object to be protected from privacy to the server. Data collection system.

2. The data collection system according to claim 1, characterized in that, if the information output unit is an image captured by the camera that includes objects to be collected but does not include objects subject to privacy, it transmits the image captured by the camera and the information of the determined objects to be collected to the server.

3. The data collection system according to claim 1, characterized in that the information output unit stores the image determination unit and the image processed by the image determination unit in storage and then transmits them to the server.

4. The data collection system according to claim 3, characterized in that the information output unit stores all images processed by the image determination unit in storage, but transmits only the images processed by the image determination unit and the image processing unit to the server if they contain the object to be collected.

5. The data collection system according to claim 1, wherein the image processing unit processes the region of an object or living organism that includes the determined privacy object.

6. The data collection system according to claim 1, wherein the image processing unit processes only the area of ​​the private object in the image captured by the camera.

7. The data collection system according to claim 1, wherein the image processing unit processes the image to make it possible to identify the type or characteristics of an object or living organism that includes the determined privacy object.

8. An imaging device that can be held by a mobile body and connected to a data acquisition server via a network, The imaging device includes a camera that images the area around the moving object, an image determination unit that determines the objects to be collected and the objects to be private, including personal information, that are included in the image captured by the camera, an image processing unit that processes the image captured by the camera, and an information output unit that can transmit information to the server via the network. The image determination unit includes an image determination model trained using training images containing objects to be collected and training images containing privacy objects, and determines the objects to be collected and privacy objects contained in the image captured by the camera from the output result obtained by inputting the image captured by the camera into the image determination model. The image processing unit generates an image in which personal information cannot be identified by processing at least a portion of the privacy object in the image captured by the camera so that the personal information of the determined privacy object cannot be identified. The information output unit transmits to the server, at least when the image captured by the camera includes both the object to be collected and the object to be protected from privacy, the image that does not allow for the identification of personal information generated by the image processing unit and the information of the object to be collected that has been determined, but does not transmit the image itself that includes both the object to be collected and the object to be protected from privacy to the server. Imaging device.

9. A computer for an imaging device that can be held by a mobile body, can be connected to a data collection server via a network, and includes a camera that images the area around the mobile body, and an information output unit that can transmit information to the server via the network, Image determination means for determining objects to be collected and objects containing personal information included in the image captured by the camera, and Image processing means for processing images captured by the aforementioned camera It is a program that makes it function as such. The image determination means includes an image determination model trained using training images containing objects to be collected and training images containing privacy objects, and determines the objects to be collected and privacy objects contained in the images captured by the camera from the output results obtained by inputting the images captured by the camera into the image determination model. The image processing means generates an image in which personal information cannot be identified by processing at least a portion of the privacy object in the image captured by the camera so that the personal information of the determined privacy object cannot be identified. The information output unit is configured to transmit to the server, at least when the image captured by the camera includes both the object to be collected and the object to be protected from privacy, the image that does not allow the identification of personal information generated by the image processing unit and the information of the object to be collected that has been determined, but not to transmit the image itself that includes both the object to be collected and the object to be protected from privacy to the server. program.

10. A mobile body that moves along a road, including an imaging device, The imaging device and a data acquisition server connected via a network are included. A road infrastructure information collection system that collects road infrastructure information in an image captured by the aforementioned imaging device, The imaging device includes a camera that images the area around the moving object, an image determination unit that determines road infrastructure information and privacy-sensitive objects including personal information contained in the image captured by the camera, an image processing unit that processes the image captured by the camera, and an information output unit that can transmit information to the server via the network. The image determination unit includes an image determination model trained using training images containing road infrastructure information and training images containing privacy objects, and determines the road infrastructure information and privacy objects contained in the image captured by the camera from the output results obtained by inputting the image captured by the camera into the image determination model. The image processing unit generates an image in which personal information cannot be identified by processing at least a portion of the privacy object in the image captured by the camera so that the personal information of the determined privacy object cannot be identified. The information output unit transmits to the server, at least when the image captured by the camera includes both road infrastructure information and objects subject to privacy, the image that does not allow for the identification of personal information generated by the image processing unit and the determined road infrastructure information, but does not transmit the image itself, which includes both the road infrastructure information and objects subject to privacy, to the server. Road infrastructure information collection system.

11. The system includes a camera, an image determination unit that determines objects to be collected and privacy objects including personal information contained in the image captured by the camera, an image processing unit that processes the image captured by the camera, and an information output unit. The image determination unit includes an image determination model trained using training images containing objects to be collected and training images containing privacy objects, and determines the objects to be collected and privacy objects contained in the image captured by the camera from the output result obtained by inputting the image captured by the camera into the image determination model. The image processing unit generates an image in which personal information cannot be identified by processing at least a portion of the privacy object in the image captured by the camera so that the personal information of the determined privacy object cannot be identified. The information output unit is characterized in that, at least when the image captured by the camera includes both the object to be collected and the object to be protected from privacy, it outputs the image generated by the image processing unit that does not allow for the identification of personal information and the information of the object to be collected that has been determined. Imaging device.

Images