Systems and programs

Abstract

To use an image obtained by a camera attached to a vehicle to identify the cause of an accident more effectively than before, and to utilize the image more effectively without being limited to the identification of the cause of the accident.SOLUTION: Processing means of a system has a function to record an image captured by a camera attached to a vehicle, perform image analysis of the images captured by the camera, and a function that executes processing corresponding to the analysis results on the basis of the analysis results.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a system for processing images captured by a camera or the like attached to a vehicle or the like. Systems and programs, etc. [Background technology]

[0002] The following Patent Document 1 discloses a system that is applied to a drive recorder, etc. The system repeats the cycle of recording video data in the recording area. When the cycle end criteria is met, the currently ongoing cycle ends and the recorded This starts a new cycle of overwriting the old video data with new video data. The amount of video data recorded in the current cycle and the amount of video data recorded in the previous cycle are displayed. based on at least one of the capacity of the video data that has not been overwritten and the capacity of the video data that has been overwritten, Information about the overwriting position is output to the output means. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-6969 A Summary of the Invention [Problem to be solved by the invention]

[0004] Images recorded by drive recorders are useful for identifying the cause of accidents, etc. The images captured by cameras installed in vehicles will be used to detect accidents more accurately than ever before. To utilize the information more effectively, not only to identify the cause of the accident, but also to identify the cause of the accident. is desired.

[0005] The object of the present invention is not limited to this, and the present invention is not limited to the above. The right to a configuration that aims to obtain the same effect as the above may be obtained by divisional application, amendment, etc. For example, in this specification, the phrase "can..." should be changed to "the problem is..." The present specification discloses a problem in which the problem is replaced with "to provide a method for producing a semiconductor device using a semiconductor manufacturing method." The structure for solving this problem is also described in the divisional application. The intention is to obtain rights by amendment, etc. Although the problem is implicitly understood from the description of the specification, However, the applicant may amend or file a divisional application to include a part of the configuration described in this specification. The scope of the claim is intended to be a combination of these independent issues. do. [Means for solving the problem]

[0006] (1) A function to record images captured by a camera mounted on the vehicle; and The image captured by the camera is analyzed, and a response is made based on the analysis results. Function to perform the process The system may preferably have a processing means having:

[0007] Not only does it record images captured by cameras installed in vehicles, but it also This can be utilized more effectively.

[0008] Examples of the vehicle include a passenger car, a freight vehicle, a special vehicle, etc. For example, it is preferable to capture at least one of the interior and exterior of the vehicle. "Outside the vehicle" can be interpreted as follows, for example:

[0009] When the vehicle body clearly separates the interior and exterior spaces, the body The inside of the vehicle is called the "interior" and the outside of the body is called the "outside". For example, the passenger space In the case of a van, where the passenger space and cargo space are connected, It is better to call it the inside of the car. In the case of a sedan, where the passenger space and the cargo space are separated, Both the passenger space and the cargo space may be referred to as the interior of the vehicle. In an automobile, the passenger space and the interior of the enclosed cargo area are called the interior of the vehicle. In the case of an open-top freight vehicle (flatbed vehicle) with a raised loading platform, the empty space that accommodates the cargo on the loading platform is It is difficult to clearly distinguish between the space inside and the space around it, but it is possible to accommodate open cargo. The passenger space of an open freight car can also be called the interior of the car. In some special vehicles, such as forklifts, both the passenger space and the cargo space are open. The passenger space and cargo space are clearly separated from the surrounding space. Even in such a case, it is possible to maintain both passenger space and cargo space. It is better to call it the interior of the vehicle. For example, the space above the seats where the passengers sit, and the space above the seats that hold the cargo and allow it to go up and down. The space above the forks that are moving can be called the interior of the vehicle. The space that can be thought of as moving with the vehicle is called the inside of the vehicle, and the space around it is called the outside of the vehicle. It is good.

[0010] As a "camera mounted on a vehicle," the relative position and It is recommended to use a camera with a fixed orientation. Also, "a camera mounted on a vehicle" is For this purpose, it is advisable to use a 360-degree camera of either a hemispherical or full sphere type. Using a 360-degree camera allows you to capture a wider range of images than using a regular camera. For example, one camera can capture both the inside and outside of the vehicle. In addition, instead of the 360-degree camera, multiple normal cameras can be installed on the vehicle. It is okay.

[0011] The image captured by the camera may be a video. The image captured by the camera may be a video. The processing means performs image analysis to detect the movement of the subject, the movement of the subject before and after a certain point in time, This makes it possible to easily determine the presence or absence of a subject.

[0012] The function of recording images captured by the camera communicates with the on-board device of the vehicle in which the camera is installed. This can be realized by a server that stores the data, but it is better to do it in a vehicle-mounted device. It is recommended to use a removable media such as a USB stick. When using this function, the removable media on which the images are recorded can be removed from the in-vehicle device and used on a PC. The images can be displayed on other terminals such as the iPod nano.

[0013] The function to analyze images captured by the camera is a sub-system that communicates with the in-vehicle device equipped with the camera. This can be achieved by the server, but it is even better to have it done by an in-vehicle device. Therefore, processing corresponding to the analysis results can be quickly performed in the vehicle.

[0014] The processing means may be configured as one device or as multiple devices. The live recorder has the function of recording images captured by the camera. The function of the processing means is realized by a microcomputer and a memory built in the drive recorder. The processing may be realized by an application program stored in the memory. The system is composed of two devices, an in-vehicle device and a server. The in-vehicle device records images and the server interprets them. It is advisable to have the server perform the analysis.

[0015] In addition, the function of recording images captured by cameras attached to objects other than the vehicle and the function of recording images captured by cameras attached to the front The image captured by the camera is analyzed, and based on the analysis results, The system may have a processing means having a function of executing the process. It is advisable to do so.

[0016] (2) The processing means detects the movement of a person in an image as a function of the image analysis, and stores the movement of the person in the image in advance. It has the function of determining whether a person's movement corresponds to a specific movement that has been recorded, and When the analysis results are processed, the image capturing a specific movement of a person is Recording the person's movements, sending images of the person's movements to a server, and reporting that the person has made a certain movement. The system according to claim 1 performs at least one of the following processes: issuing a notification and issuing an alarm. It is good.

[0017] For example, the images recorded or sent to the server may show a person making a certain movement. For example, a person can be shown to have made a certain movement. By reporting the incident, outsiders can be informed that someone is behaving in a specific way. For example, if a person is engaged in a certain movement due to an issued warning, the person may not be able to detect the warning. Being aware of what is being done motivates you to stop making certain movements.

[0018] The person performing a specific movement may be, for example, a driver of a vehicle. For example, it may be an action other than driving. For example, looking at a mobile device such as a smartphone Examples of actions that may be taken include actions of eating, looking away, or falling asleep. The presence or absence of the driver's inattentive behavior may be determined based on the detection result of the driver's eye position. The captured image can be used to improve driver awareness, analyze the cause of accidents, etc. When it detects that the driver is operating a mobile device such as a smartphone, the system automatically notifies the driver not to use the mobile device while driving. It is advisable to have a warning issued to warn drivers not to touch the terminal. By stopping certain movements, danger can be avoided in advance. The server of the vehicle manager or the police should be used as the server. Based on the transmitted data, the driver's driving behavior can be evaluated. Based on the captured images, it is possible to evaluate the driver's driving behavior and analyze the causes of accidents.

[0019] A person who moves in a specific way, for example, a suspicious person outside a car, and a specific movement, for example, a car Attempting to break into a building, such as taking a hammer and trying to break a glass window It is advisable to have the driver peer into the vehicle, etc., in order to detect trespassers by sounding an alarm. This can deter suspicious people from entering the vehicle. The images sent to the server provide valuable information for arresting thieves after a car break-in has occurred. It can be used to notify the vehicle owner or driver that a person has made a certain movement. By notifying the owner or driver, the owner or driver can immediately return to the vehicle and prevent damage from vehicle theft. In addition, the driver or owner who receives the notice may also report the matter to the police. In this way, this function can be used to help prevent crime.

[0020] A person who behaves in a specific way, for example, a suspicious person in a car, and various vehicles are detected as specific movements. High-level tactics, such as illegal entry into a parked car, suspicious behavior in a parked car, The alarm should be set to the action of removing the device. By scattering the images, the damage caused by car theft can be reduced. The images sent to the server provide useful information for arresting thieves after a car break-in has occurred. This can be used to inform the vehicle owner or driver that a person has made a certain movement. This notification allows the owner or driver to immediately return to the vehicle or call the police. This will help reduce the damage caused by car theft. Examples of such emails include sending emails with a photo of the examiner attached. This may be the vehicle manager, driver, owner, police, etc.

[0021] As a person who moves in a specific way, for example, a passenger in a business vehicle with unspecified people riding in it is assumed that As a specific movement, various robbery methods, such as the movement of brandishing a knife or the movement of holding a gun, The processing means executes a process for issuing an alarm. If the robber is deterred from committing the crime, the damage can be prevented. The images sent to the server can be used as useful information for arresting robbers. By notifying the vehicle manager that a person has made a certain movement, the manager can In addition, the processing means can use the communication function to notify the police of the occurrence of the incident. It would be good for the system to have the ability to report to the relevant authority.

[0022] For example, a person who moves in a specific direction may be a baby or infant in a car. For example, it would be good to use image analysis to detect the movement of an infant waking up from sleep. Then, the processing means may notify the guardian by e-mail or the like. I fell asleep and left the engine running while I went shopping or ran some errands. In this case, if you forcefully wake up the baby, he or she will become upset, so However, parents are requested to keep their children asleep while away from the vehicle. I am worried that my baby might be awake. By using this system, I can When the vehicle is stopped, an email is sent to the parent's mobile device, so the parent can immediately rush to the vehicle. This allows the mother to deal with the awakening of her infant.

[0023] For example, a person who needs care or rescue outside the vehicle may be considered as a person who makes a specific movement. For example, if we define the actions specific to a person requiring care, or the actions corresponding to situations where rescue is likely to be required, For example, examples of the specific movements of people who require assistance include using a cane, bending the waist, and Examples of movements that may indicate rescue is required include falling. Examples of actions that correspond to a situation in which a person is shaken and collapses, foaming at the mouth, etc.

[0024] The function to detect people making specific movements is based on images of the front of the vehicle, the rear of the vehicle, and the interior of the vehicle. Select at least one of the images, and then select the person from the selected images. It is recommended that the selection of at least one image be specified by the user. This prevents the processing means from analyzing unnecessary or less important images. Therefore, there is no need to provide the processing means with excessive processing power.

[0025] It would be better to add a function to detect driver drowsiness to the function to detect people making specific movements. For example, the driver's pulse rate can be measured from the degree of change in the redness of the skin, and based on the change in pulse rate, It is recommended to determine whether the driver is drowsy or not. It is preferable to improve the accuracy of drowsiness detection by linking the measurement of the pulse rate, etc., which is performed by the processing means of the present system. If drowsiness is detected, the air conditioner will lower the temperature setting and issue a voice or sound warning. good.

[0026] (3) The processing means detects whether a person in an image is a person according to a preregistered feature as a function of the image analysis. The function of the system is to determine whether or not a certain feature is present in the image, and to process the results of the analysis. It is preferable that the system executes processing according to specific characteristics possessed by a person.

[0027] Appropriate processing can be performed depending on the characteristics of the person appearing in the image.

[0028] Functions that determine whether a person in an image has certain characteristics, such as facial recognition functions For example, it is advisable to perform face recognition on the person sitting in the driver's seat and to determine whether the person sitting in the driver's seat is in advance. If the person does not correspond to the registered person, the processing means is not able to access the vehicle and start the engine. For example, the processing means may not be able to read the engine control unit (ECU) ) to prohibit the engine from starting. It is possible to register people who are able to drive a vehicle and prohibit unregistered people from driving the vehicle. As a result, the vehicle's crime prevention function can be improved. In addition to prohibiting the engine from starting, it also prevents the accelerator from being pressed. Prevents the handbrake (parking brake) from being released. Prevent the person from entering the Eve position, sound an alarm, and send a picture to the registered email address Processing such as sending an e-mail may also be performed.

[0029] In addition, for example, face recognition of people outside the vehicle is performed, and if the person outside the vehicle matches a person registered in advance, If so, the processing means may control the vehicle to unlock the doors. For example, If the vehicle is provided to a car-sharing service, the car-sharing service It is advisable to register the members of the service in advance. The registered members can approach the vehicle and take pictures of the vehicle's camera. By moving your face within the corner, you can use the vehicle even if you do not have a card key. This will make it easier to use the vehicles of car sharing services. It becomes.

[0030] The vehicle is a commercial vehicle in which unspecified persons ride, and the processing method is to determine whether the passenger is a wanted criminal. It is advisable to use face recognition to determine whether a person is a criminal or not. The data is transmitted from the database to the processing means of this system via LTE, wireless LAN, etc. The passenger may be registered in advance in the processing means. The processing means determines that the passenger is a wanted criminal. In such a case, the vehicle manager and the police will be notified through the communication device and communication network. In addition, the processing means may notify the driver in a manner that is not noticeable to the passengers. The vehicle manager, police, driver, etc. should be aware that a wanted criminal has boarded the vehicle. be able to recognize and take appropriate action promptly.

[0031] For example, in a taxi, the driver and passengers are the only people in the vehicle, so the driver's safety is a concern. There seems to be a high demand for this system in terms of securing criminals. When the processing means detects this, it is preferable to notify the taxi company or the police that a criminal has been in the vehicle. In addition, the system detects whether a criminal is in a vehicle before he or she actually commits a crime in the vehicle. It is possible to detect criminals as soon as they get in the car. This allows the system to prevent crimes from occurring. It will also be possible to do this.

[0032] The function of determining whether a person in an image has a specific feature is, for example, A function that compares the appearance of a person with a specific appearance registered in advance and determines whether the two match (hereinafter referred to as It is recommended to include a function for identifying the appearance of a wanderer. It is a good idea to register the appearances of criminals in advance. Examples of criminal appearances are as follows: For example, carrying a knife on the street or carrying a dangerous object. Combining images from external cameras, ultraviolet cameras, etc. with images from regular cameras to detect illegal drugs The processing means may detect a person who is carrying a particular appearance in the image. When a person carrying the device is detected, the device notifies the police or the vehicle manager via the communication device and communication network. It is good to know.

[0033] If you want to find a specific missing person or wanderer, you can look at their appearance at the time they went missing. It is advisable to register these as search information in the processing means individually. For example, gender, height, clothing, etc. Type of clothing, color of clothing, whether or not a hat is worn, whether or not a walking stick is held, whether or not the person is bent over It is a good idea to register individual search information such as the following. The processing means may register the individual vehicle location information (search target area) when the vehicle is located in the search target area. It is advisable to carry out a process to detect people who match the search information. Such individual search information can be used as a warning. The information is sent from the database (server) of the local support center to the processing means of this system. The vehicle-mounted device may perform a function of determining whether a person in an image has a specific characteristic, When the vehicle enters the search area, search information is downloaded from the server to the vehicle's processing means. It is advisable to do so.

[0034] The function to determine whether a person in an image has certain characteristics, for example, whether they are in the passenger seat or the back seat A function to determine whether a person sitting in a car is wearing a seat belt (hereinafter referred to as appearance recognition) The processing means of the present system may include a function to detect a person who is not wearing a seat belt. When this is detected, a message may be issued to prompt the driver to fasten his or her seat belt.

[0035] Especially when the person in the back seat is not wearing their seat belt and they are approaching the entrance to the highway. It is advisable to issue a message encouraging the driver to fasten his / her seat belt. For example, When the distance to the entrance of the expressway falls below the standard distance, the vehicle is approaching the entrance of the expressway. In addition, if you receive a signal to check if your ETC card has been inserted, It is advisable to determine that the vehicle is approaching the entrance to a road. Also, the image in front of the vehicle is analyzed, and When an entrance to a highway is detected, it may be determined that the vehicle is approaching the entrance to the highway.

[0036] (4) The processing means has a function of counting the number of people in an image as a function of the image analysis. As a process corresponding to the analysis result, the number of people in the image is recorded or transmitted. It would be a good idea to have a system that does this.

[0037] This system counts people without using any special equipment for counting people. It is possible.

[0038] For example, a camera could be installed on a bus to capture images of the interior of the bus as the basis for counting people. The processing means may display the counting result and inform the flight attendant. This function allows you to easily know the number of passengers on board the vehicle. This can be used in place of a tour roll call. It is used to call out passengers when all passengers should disembark. When the count doesn't reach zero, the crew knows there is someone left in the car. For example, people who cannot express their own will, such as babies, or people with disabilities, It is possible to detect people who are unable to get off by themselves and remain behind, such as:

[0039] For example, if the vehicle on which the camera is installed is a type of vehicle with a predefined number of passengers, e.g. For example, tourist buses, long-distance express buses, etc., the number of passengers is determined in advance and processed by this system. The processing means compares the result of the counting of the number of people with the number of people registered. If there is a shortage or excess, the driver should be warned. This prevents trains from departing with insufficient passengers. It is easy to determine when someone has boarded the vehicle and encourages them to get off.

[0040] For example, a camera (e.g., a hemispherical camera on the ceiling) is installed inside a route bus, and the processing procedure is Dan performs image analysis to count the number of passengers on board, standing passengers, and seated passengers. In addition, it would be good to have a function to provide information on the current location of the bus, the opening and closing signals of the bus doors, and when passengers should get off the bus. The processing means receives signals from the buzzer, touching the IC card when getting on and off the bus, and the fare box. It would be good to have a function to record these as triggers. This information should be accumulated in a data lake, Big data analysis of where passengers got on and off, and the seating situation in the car during that time This data will be available to consulting companies that handle urban planning, etc. This information can be valuable for the vehicle. It is advisable to collect information on the vehicle's status and the driver's vital signs. When an abnormality occurs, it is necessary to determine whether the abnormality is due to driving factors, passenger factors, or the driver himself / herself, and It can be used to identify causal relationships between two factors.

[0041] For example, it is advisable to use an image of the outside of the vehicle as the image on which the number of people is counted. The processing means may record the counting result as number of people data. Based on this, it is easy to extract locations with a large number of people. For example, if the number of people is less than a predetermined threshold, the image on which the count was made can be recorded. It is a good idea to record images when the above occurs.

[0042] (5) The processing means, as a function of the image analysis, performs the following operations on a display device of the vehicle from an image of the interior of the vehicle. It has the function of detecting the display contents or the operation status of the vehicle's operating devices, and takes action according to the analysis results. As a process, a process is executed according to the display contents of the display device or the operation status of the operation device. It would be a good idea to have a system like this.

[0043] The display device is connected to the device that acquires the data that is the basis for the display content, or to the operating status of the operating device. This system can obtain the data according to the needs of the user without the need to install any additional equipment. For example, these can be done without connecting any additional equipment to the OBD connector. In particular, you can connect a third-party product to the OBD connector to obtain data. Even if the vehicle manufacturer prohibits the collection of such data, this system can This display device can display various gauges, warning lights in the car, front lights, and other information. It is advisable to include a head-up display that projects images onto the front glass, etc.

[0044] For example, the display content of the display device to be detected is the vehicle speed displayed on the speedometer. The processing means performs pattern recognition on the position of the needle of the analog speedometer and character recognition on the numbers on the dial. It is also possible to obtain vehicle speed data by recognizing the numbers displayed on the digital speedometer using character recognition. The acquired vehicle speed data can be used for various processing. An example of how the vehicle speed data is used will be described below.

[0045] The processing means may have a function of automatically detecting the speedometer by image recognition. Thus, the processing means can detect the speedometer in various vehicles, regardless of the vehicle type. In addition, the processing means may include a processing unit for processing the speed meter of the vehicle in which the processing means is mounted. It is preferable to register the position and shape of the object in advance. This allows the processing means to easily calculate the speed. The meter can be detected.

[0046] Standalone car navigation devices use GPS to obtain the current location. In tunnels and other places where GPS signals cannot be received, other methods are used. The vehicle's current position must be determined by the speed just before the GPS signal is lost. The vehicle's current position is moved within the map at a constant speed of approximately 80% of the normal speed for five minutes. The mode in which such processing is performed is called tunnel mode. This is effective not only when entering a tunnel, but also when driving under an elevated road. If you encounter a traffic jam while processing in the normal mode, If we make this assumption, the error in the current position will be too large and it will be practically useless. If an acceleration sensor is installed in the vehicle and the acceleration data is used, This would improve the accuracy of the current location information in tunnel mode. However, it is difficult and expensive for users to install an acceleration sensor on the vehicle, making it impractical.

[0047] The car's speed is read from the speedometer display, and the car navigation system automatically calculates the speed based on the read speed. The current location information of the vehicle is acquired using the map matching function implemented in the application system. This can improve the accuracy of the current position information of the vehicle.

[0048] Where the road inside the tunnel and the ground road run parallel to each other, where the elevated road and the ground road under the elevated road run parallel to each other. In locations where tunnel mode is enabled, the system will match the road to which it is connected based on whether tunnel mode is enabled or disabled. For example, when tunnel mode is activated, the vehicle position is tracked. Map it to the road in the tunnel or the ground road under the overpass and make sure the tunnel mode is activated. If not, the vehicle's position is mapped to the ground road or elevated road that runs parallel to the road in the tunnel. It is a good idea to have them do this.

[0049] The display content of the display device to be detected may be, for example, the remaining fuel amount displayed on a fuel gauge. The processing means may detect the remaining amount of fuel by performing image recognition of the fuel gauge. When the processing means detects that the remaining amount of fuel falls below a predetermined reference value, the processing means notifies the driver to refuel. For example, by linking the current location information of the vehicle, When you return to your home or company vehicle storage area, the remaining fuel level is at a reference level, e.g. For example, if the tank is less than 1 / 4 full, the processing means may issue a warning and prompt the driver to refuel. - The navigation device and the present system are linked, and the processing means calculates the distance to the destination and the fuel The system compares the remaining fuel level with the fuel level in the vehicle and prompts you to refuel when you depart or along the way to your destination based on the comparison results. For example, if you are near a gas station on your route, you can search for a gas station near your current location. It would be a good idea to output a message prompting the driver to refuel.

[0050] The function of extracting the speedometer and fuel gauge from the image is implemented by the processing means of the present system. This can be achieved by using artificial intelligence (AI). When it is difficult to obtain the image, the positions of the speedometer and fuel gauge in the image and the image information are stored in advance in the processing means. It's a good idea to register.

[0051] The processing means reads a fuel gauge of a vehicle provided to the car sharing service, When it detects that the remaining fuel level is below a certain threshold, the car sharing service will start automatically when the vehicle is refueled. For example, a message like, "If you fill up now, you can receive special benefits." You can receive a discount equivalent to 15 minutes. It would be good to do so.

[0052] The display contents of the display device to be detected include, for example, warning messages in the instrument panel. The warning lights to be checked for their illumination include headlights, etc. Light indicator, turn signal indicator, high beam indicator (high beam warning It is a good idea to include lights such as a parking brake indicator, a half-open door warning light, and a fuel warning light. The processing means of the present system determines whether or not these warning lights are on, and The corresponding vehicle status can be detected.

[0053] The display contents of the display device to be detected include, for example, the display contents of a head-up display. The processing means may display the warning information by a head-up display. It detects the warning content by capturing the change in the displayed image and executes the specified processing according to the warning content. For example, a warning may be issued by sound or voice. A warning may be issued by visual information or by hearing. By changing the information into visual information and issuing it, the effect is that the driver will be more likely to notice the warning. can be.

[0054] For example, the processing procedure of this system can be improved by linking with information on the brightness outside the vehicle and the current location of the vehicle. In this case, the driver did not turn on the headlights even though the situation required the headlights to be turned on. If it detects that the headlight indicator is not on, it will prompt the driver to turn on the headlights. For example, there are situations where the headlights should be turned on when the outside of the vehicle is brighter than a certain level. This includes situations where it is dark outside, or the vehicle is currently inside a tunnel or culvert. The headlight indicator remains on even though the condition that should have turned on the headlights has been resolved. When the headlights remain on, the processing means of the system controls the headlights to be turned off. It is advisable to inform the driver. Some tunnels are brightly lit due to lighting, so please be aware of the lighting outside the vehicle. If the vehicle is in a situation where it is necessary to turn on the headlights based only on the brightness information, Even though the vehicle was within the vehicle's range, it was mistakenly determined that the headlights should not be turned on. It is not possible to determine whether the vehicle is in a tunnel or not based on the current position information. This makes it possible to avoid such erroneous determination. The parking light indicator is on even though the parking light (low beam) should be on. If the system detects that the sidelights are not on, it notifies the driver to turn on the sidelights. It is good.

[0055] The processing means of the present system determines whether or not there is an oncoming vehicle from an image of the front of the vehicle, and if there is an oncoming vehicle, However, if the high beam indicator is still on, the system will switch to low beam. It is advisable to warn the driver to change the headlights. The high / low beam switching operation differs depending on the vehicle model. Therefore, drivers should refrain from using low beam headlights when driving a vehicle of a type different from the one they normally use. It is easy to encounter situations where you intend to turn the headlights on but the high beams are still on. In this case, the function of issuing this warning is particularly useful.

[0056] The processing means of the present system detects that the turn signal is flashing from an image outside the vehicle when it is dark outside the vehicle. The indicator that flashes inside the vehicle is synchronized with the blinking cycle of the turn signal. In this way, the processing means can recognize the installation as a power indicator. The turn signal indicator can be easily found in the main panel.

[0057] The processing means of the present system detects from the image of the inside of the vehicle that no one is inside the vehicle. When it detects that the headlights or width lights are on, it will notify the driver outside the vehicle or the vehicle management system. The user may be notified that he / she has forgotten to turn off the light. This notification may be made, for example, by a message registered in advance. The processing may be performed by sending an e-mail to the mail address of the user. If you forget to turn off the hazard lights or interior lights, you will be notified by email, etc. The lighting status of the hazard lamps is detected from the blinking status of the turn signal indicator. The lighting status of the room light can be detected from the brightness of the room light reflected in the image. It is good to know.

[0058] The room light settings generally include a constant on mode, a constant off mode, and a door open / close mode. There are three types of interlocking modes (hereinafter simply referred to as interlocking modes). In most vehicles, These three modes are cycled through each time the room light on / off switch is pressed. When the engine is turned off in a dark place (at night), the interior light turns on and after a while Some vehicles have a function that automatically turns off the light after a few seconds. If you want to keep the interior lights on for a longer period of time to prepare for disembarking after you have For example, after the engine is turned off, the owner may organize and take out purchased items, or use a mobile phone. When taking out the food, etc., it is recommended to keep the room light on. Drivers can set the room light to not automatically turn off after a certain period of time has elapsed since the engine was turned off. To do this, switch the room light setting to linked mode or constant light mode. When the door is open, the current room light setting is either constant light mode or linked mode. Even though the current room light setting is always on, it is difficult to tell whether the light is on or off. However, if you get off the train thinking that it is in linked mode, the room light will remain on. By using this system, drivers and managers can By realizing that you forgot to turn off the room light and turning it off, you can prevent the room light from being left on for a long time. This prevents the battery from dying in vehicles that have a large battery. A structure where you don't know what mode the light is currently in, such as pressing a button Vehicles with an operating switch that is designed to cycle through the interior light settings each time it is pressed This feature of the system is particularly useful in

[0059] When the room light is set to linked mode, any door is open or ajar. If so, the room light will remain on. Notify that the room light is on Drivers and managers who have been given access to the vehicle may find that the door is ajar or that the door on the less visible side is open. You can realize that there is a possibility that

[0060] When the room light is set to linked mode, the light will turn off after a certain period of time (delay time) has elapsed since the door was closed. Some vehicles have a function that keeps the interior lights on until the vehicle is turned on. In such a vehicle, the processing means of the present system detects that the door is closed and then It is advisable not to judge whether or not the light has been forgotten to be turned off until the light-off delay time has elapsed. The delay time may be set by the system based on input from the user.

[0061] In addition, the processing means may be configured to turn on the room light even if a certain period of time has elapsed since no one was in the vehicle. When it detects that the interior light is still on, it sends a signal to the vehicle controller to turn off the interior light. Send a signal to light it up.

[0062] The processing means of this system is equipped with headlights, width lights, hazard lights, room lights, etc. If the light is on or flashing and the system detects that a vehicle door has been opened, one of the lights It is advisable to have a function to issue an alarm to inform the driver that the light is still on. The driver will be able to realize that he or she has forgotten to turn off the lights before getting out of the vehicle, which will prevent the driver from forgetting to turn off the lights in the first place. .

[0063] The processing means of the present system determines whether the parking brake is applied based on the illumination of the parking brake indicator. When the system detects that the driver is in a braking state, it will detect that the driver is not present from the driver's image. It is a good idea to set the alarm not to sound even if it detects that you are sleeping. When the driver is driving, such as looking at a mobile device while the driving brake is applied, For example, even if a motion that is deemed dangerous is detected, an alarm will not be issued. It would be good to do so.

[0064] The system's processing means provides early warnings to indicate the need for inspection or repair. If the light is detected to remain on for a certain period of time, the dealer or registered It would be good to have a function to notify the vehicle's condition to the repair shop that has registered the vehicle. Users who do not have the device may not notice the warning light on if the vehicle is still capable of driving even with the warning light on. For example, when the engine warning light comes on, it usually means that the vehicle is not drivable. However, even if the engine oil warning light is on, you cannot continue driving. Also, if there is an abnormality in the O2 sensor, fuel economy will decrease, but the vehicle will still be able to be driven. If the vehicle is still drivable and you continue to drive it without inspecting and repairing it, a minor malfunction may become serious. For dealers and repair shops, the warning light is turned on when the This information is useful for sales activities of inspection and repair to vehicle owners. By performing inspection and maintenance early, it is possible to prevent the breakdown from becoming serious.

[0065] For example, the display content of the display device to be detected may be a temperature sensor that measures the temperature outside the vehicle. Displays the temperature readings measured by the temperature display area in the instrument panel. The processing means may include, as a function of image analysis, It would be good to have a function to read the temperature by character recognition from the image displayed in the temperature display area. The processing means performs processing corresponding to the analysis result via the communication device and the communication network. It is advisable to execute a process for transmitting temperature information to an external server.

[0066] For example, sensors could be installed in the homes of ordinary users to obtain data on pollen dispersion. By collecting measurement data from sensors, it is possible to predict detailed pollen dispersion conditions and Weathernews Inc. and other companies provide services that provide information on dust dispersion. The outdoor photos posted by general users and outdoor live cameras installed in general homes are Images collected from these satellites are incorporated into weather forecasts. These weather information services (e.g. Weathernews SOLiVE24) are provided. The system that provides this information service collects outside air temperature information measured in many vehicles. This allows many vehicles to be used as temperature measurement probes. When many vehicles are distributed over a wide area, the temperature information collected from many vehicles is not uniform. For example, it can be used to provide spot information that cannot be provided by general weather information. It will be possible to issue heatstroke warnings, etc.

[0067] For companies providing such weather information services, installing temperature sensors in-house is This provides the effect of being able to collect temperature information over a wide range without having to The company that receives the temperature information will give a gift to the vehicle owner who provided the information. It is advisable to provide a fee-based weather information service free of charge or at a discount.

[0068] Not only temperature information is read from the temperature display area in the instrument panel, but also from the camera. The system performs character recognition of the numbers displayed on roadside temperature signs captured in images captured by the The temperature display sign may be detected by, for example, detecting the temperature in the sign. This can be done by recognizing the words "current temperature" or the like. The acquired outside temperature information is The temperature information obtained from the temperature display area in the instrument panel is also The processing means may transmit the information to an external server.

[0069] The function that detects the operation status of the operating device by image analysis includes the accelerator pedal and brake It is advisable to include a function for detecting the position of the driver's foot relative to the pedal. Depending on the situation, the driver's foot may be on the accelerator pedal or on the brake pedal. In order to allow the driver to easily recognize where the vehicle is located, it is recommended to execute a process to notify the driver. For example, when your foot is on the accelerator pedal and when it is on the brake pedal, The driver may be notified by changing the color of the light emitted by the light-emitting element. Detects whether the driver's foot is on the steering wheel or the brake pedal, and displays the detection result. This system is installed in a vehicle that has a function to display the above information as visual information on a display. The processing means performs image analysis of the contents displayed on the display device and detects the accelerator pedal and the brake pedal. It is advantageous to detect the position of the driver's feet relative to the dall.

[0070] Furthermore, even though image analysis detects an obstacle ahead, the driver's foot is not on the accelerator pedal. When the vehicle is in the "Dal" position, it is advisable to issue a warning by sound or the like. This can prevent the driver from accidentally stepping on the accelerator pedal instead of the brake pedal.

[0071] The steering wheel (hand wheel) is now equipped with a function that uses image analysis to detect the operation status of the control device. It is preferable to include a function for detecting the rotation angle of the operation device. Based on the steering wheel rotation angle and the vehicle speed, the vehicle's traveling direction and travel distance are calculated. The result of the process can be used, for example, in a stand-alone car navigation system. Standalone car navigation devices should be provided with the same information as the system. Based on the given heading and distance information, the vehicle will Even in such a situation, the travel route of the vehicle can be calculated.

[0072] The image analysis function can determine whether the vehicle is in autonomous driving mode (including autonomous driving levels 1 to 5) or in manual driving mode. It is advisable to include a function to detect whether the vehicle is in operation (automation level 0). Indicators that show whether various automatic driving functions displayed on the control panel are active or not The display state of the data can be recognized from the image. For example, when it detects that the autonomous driving function has been deactivated, In addition, even if the automatic driving function is deactivated, the driver may be notified. Regardless of whether the driver has his / her hands on the steering wheel or not, when the processing means detects that the driver has his / her hands on the steering wheel, In this case, a warning may be issued to alert the driver to operate the steering wheel. In addition, the autonomous driving function was deactivated and the driver had their hands on the steering wheel. When the camera does not detect the presence of the camera, the processing means may record the image captured by the camera. Recorded images can be used as evidence that the driver was operating improperly. can.

[0073] In a vehicle equipped with a radar detector, the processing means detects that an automatic driving function is enabled. If you are using a radar detector, it is a good idea to disable the function. This will prevent radar detectors from When the automatic operation state is disabled, the processing procedure is The first step is to enable the radar detector function.

[0074] Whether the vehicle is autonomous or manually driven, and what level (mode) of autonomous driving is in use ) indicators, steering wheel movement, driver's hand position, driver It is advisable to judge this from the direction in which the head is facing, etc.

[0075] (6) The processing means, as a function of the image analysis, detects an image of the interior of the vehicle and a vehicle-attached image. It has the function of recognizing the characters displayed on the stickers, and processes the results of the analysis. The system then executes a process to alert the user according to the information indicated by the recognized characters. It would be good to do so.

[0076] By being alerted, the driver, manager or owner of the vehicle is required to The robot will then pay attention to the text information on the sticker and perform appropriate processing according to the text information. can.

[0077] For example, if the stickers to be analyzed include inspection marks (vehicle inspection stickers), The inspection mark on the inside of the vehicle shows the date on which the vehicle inspection expires. The processing means uses character recognition to read the information on the expiration date of the vehicle inspection validity period, and calculates the expiration date from the current date. If the period from the date of purchase to the expiration date of the vehicle inspection (the vehicle inspection deadline) becomes shorter than the specified period, For example, the processing means may display a message such as "The expiration date of the validity period of the vehicle inspection is It is a good idea to alert the driver by issuing a voice message saying, "You are approaching." If the period until the expiration date becomes shorter, for example, the processing means may display a message such as "Please have your vehicle inspected. It is advisable to issue a voice warning saying, "Please stop."

[0078] For example, stickers that have been inspected may be included in the stickers that are the subject of image analysis. The inspection sticker indicates the date by which the next regular inspection is due. The step uses character recognition to read the date information indicating the deadline for regular inspections, and calculates the period from the current time to the regular inspection date. If the time until the inspection deadline becomes shorter than the specified period, a reminder will be issued to undergo regular inspection. For example, the processing means may output a voice saying, "The deadline for regular inspection is approaching." If the period until the regular inspection deadline becomes even shorter, For example, the processing means may issue a warning in the form of a voice saying, "Please undergo regular inspection." good.

[0079] The image analysis function determines whether or not the inspection mark is present in the image captured by the camera. It is advisable to include a function to process the analysis results if the inspection mark is not captured in the image. In this case, the processing means adjusts the mounting position and attitude of the camera so that the inspection mark is captured. For example, the processing means may execute a process of calling attention to the fact that the inspection mark is not visible. Please adjust the camera position so that the inspection mark is captured. This will allow the camera to be easily adjusted in position and attitude when it is installed. The processing means can display information on a display device in a vehicle, such as a speedometer, a fuel gauge, etc. It has the function of reading the contents of the vehicle and the function of drawing attention so that the display device inside the vehicle is reflected in the image. It's even better to have it.

[0080] The image analysis function is installed on the windshield, driver's seat and passenger seat side windows as required by law. If we include a function to determine whether any stickers other than the approved stickers are attached, As a response to the analysis results, stickers other than those permitted by law may be used. If it is determined that the sticker is attached, the system notifies the driver to remove the sticker. It is advisable to do so.

[0081] (7) The processing means analyzes an image of an area including a passenger seat as a function of the image analysis. It includes a function to analyze the passenger's seat status and executes processing according to the analysis results. It would be a good idea to have a system like this.

[0082] For example, depending on whether there is a person in the passenger seat or whether there is an object on the passenger seat, , appropriate processing can be performed.

[0083] The processing means issues a warning when a passenger in the passenger seat is not wearing a seat belt. The processing means may be configured to detect whether the passenger seat is in the passenger seat as a function of image analysis. It would be good to have a function to determine whether or not a person is sitting in the passenger seat from the image of the captured area. As a process corresponding to the analysis result, when the processing means detects that no person is sitting in the passenger seat, In the above, the function of issuing a warning when the passenger's seat belt is not fastened may be disabled. As an image analysis function, it would be good to have a function to determine whether an object is placed on the passenger seat. As a process corresponding to the analysis result, the processing means performs the following process: It is advisable to execute a process to disable the function that issues a warning when the seat belt is not fastened.

[0084] The vehicle incorrectly determines that a person is in the passenger seat when an object equivalent to a human's weight is placed in the passenger seat. If this system is installed in a vehicle, a warning may be issued if the seat belt is not fastened. When this is selected, the vehicle will issue a seat belt warning if there is an object on the passenger seat. This will prevent the driver from being confused by the warning. This function is available for vehicles that have a warning buzzer that continues to sound when a heavy object is placed on the vehicle. To prevent the vehicle from issuing a seat belt warning, For example, the processing means may transmit a warning to a vehicle controller via an in-vehicle network (CAN). It is advisable to transmit a signal to disable the emission of the signal.

[0085] Regarding the rear seats, in vehicles with similar functions, if there are objects placed on the rear seats, When this is detected, the processing means may cancel the seat belt unfastened warning function.

[0086] (8) The processing means, as a function of the image analysis, It has the function of comparing the image in the vehicle with the image in the vehicle interior at the current time, and processes the analysis results. If the degree of discrepancy between the image and past images of the interior of the vehicle exceeds a certain level, a notification is issued. It would be a good idea to have a system like this.

[0087] The vehicle owner or manager will be notified when the vehicle interior conditions exceed a certain level. The vehicle interior can be seen as a past image of the interior. It is a good idea to use an image of the vehicle with nothing in it except the vehicle. By comparing the image of the inside of the vehicle with the image of the inside of the vehicle, it is determined whether anything has been brought into the vehicle. It is possible.

[0088] By comparing images taken at multiple past points in time with the current image, the system can detect what has been brought into the vehicle. The processing means may calculate the time that the object is left in the vehicle for a long time. It would be good if the device had a function that notifies you when something is detected as being left behind.

[0089] For example, the vehicle should be empty and, ideally, with only the driver in the vehicle. Then, the initial setting button of the system is pressed and the vehicle is driven. The processing means calculates the vehicle speed. Any part of the image that has optical flow is recognized as a window. The driver is recognized as the driver. The windows and other areas other than the driver are not suitable for placing objects. The vehicle is recognized as a certain area and an image of the vehicle without luggage (no luggage image) is created. This completes the initial setting process. After this, when the engine starts, When the vehicle stops, the difference between the image taken when there is no luggage and the current image is taken, and any objects brought into the vehicle are detected. Identify the presence of an object in the vehicle. If the object identified as having been brought into the vehicle does not change position for a certain period of time, This will alert the driver to any objects that have been left in the vehicle for a long period of time. This feature is especially useful for people whose cars tend to get full of dust. It is effective.

[0090] (9) The processing means, as a function of the image analysis, detects a preregistered object in an image of the interior of the vehicle. It has a function to determine whether the subject is in the image or not, and processes the results of the analysis using pre-registered The system can be designed to issue a warning when a vehicle interior image shows a vehicle interior object that has been stolen. stomach.

[0091] The driver, etc., can be made aware of the presence of a pre-registered object in the vehicle by the alarm. For example, you can process the image of an item (a specific bag, wallet, etc.) that you want to check for lost items. The processing means is configured to process the registered information when no one is in the vehicle. If it detects that an object is in the image, it will sound an alarm or notify you that the object is lost. An example of an alarm that can be issued is to sound a buzzer installed inside the vehicle. For example, information may be sent wirelessly to the driver's smartphone or mobile phone, or by e-mail, etc. It is advisable to notify them.

[0092] For example, when the processing means detects an item left behind in the vehicle and detects the movement of opening the door, the processing means issues an alarm. The warning can help drivers and others to notice that they have left something behind before they leave the vehicle. The processing means detects, for example, items left behind in the vehicle and detects the movement of closing the door. The driver should be notified by e-mail or other means that an item has been left behind. The driver should then The system can detect the movement of the door opening and closing, For example, this can be done by detecting optical flow.

[0093] The processing means detects other movements within the vehicle by detecting the optical flow. For example, the processing means may have a function of detecting whether a door is open or closed as a function of image analysis. movement, window opening and closing movement, turn signal operation, steering wheel operation, sun visor top The system detects the operation of lowering the vehicle seat. This information can be obtained without connecting to a USB port (e.g. OBD connector).

[0094] When the sun visors are down, it is very dangerous to use a 360-degree camera mounted on the windshield. When capturing an image of the driver's side, the sun visor gets in the way and it is impossible to capture an image in that direction. Therefore, when the brightness of the front of the vehicle is such that sun visors are not necessary, Nevertheless, if the processing means detects that the sun visor is down, it issues a warning. Whether the sun visor is down or not is determined based on the proportion of the sun visor area that occupies the image. The processing means may then determine:

[0095] (10) The processing means, as a function of the image analysis, The system has the function of capturing an image of an object and recognizing the image. The system allows the user to select the camera as the processing corresponding to the analysis result. It is preferable that the system executes processing according to information written on an object brought close to the camera.

[0096] By bringing an object close to the camera, the user can receive information written on the object. The corresponding processing can be executed.

[0097] For example, the object that is close to the camera may be the driver's license. The validity date on the driver's license is recognized by character recognition, and the period from the present to the validity date is calculated. If the time between renewal and renewal is shorter than the specified period, the user is notified that the renewal date of the driver's license is approaching. It would be good to do so.

[0098] For example, a receipt from a gas station may be placed close to the camera. The processing means performs character recognition on the date, amount of fuel, and amount shown on the receipt, and converts this information into It is a good idea to record this and create a database.

[0099] For example, items that can be placed close to the camera include identification documents (e.g., driver’s licenses, employee ID cards, address books, etc.). The processing means may perform character recognition of the name, etc., written on the identification card, It is advisable to determine whether the person is registered in advance. If the person on the ID card is not registered In this case, it is advisable to perform a process to prohibit the engine from starting. The person who can do this can be limited to those who are registered in advance. It is advisable to record the name of the person on the ID card. The recorded information will be used by the vehicle management system. This can be used to identify the driver in the event of a traffic accident.

[0100] The processing means has an image recognition function, which recognizes an object that the user brings close to the camera outside the vehicle. It is preferable that the device has a function of recognizing objects, and the function of recognizing objects that the user brings close to the camera inside the vehicle. For example, the processing means may have a feature that is not present in the normal image compared to the normal image. When an object occupies a larger area than the camera's viewing angle, it is moved closer to the camera. In particular, the processing means may recognize that the image is captured in a wider range than a predetermined range within the angle of view of the camera. When the system recognizes that the user is holding an object that occupies a large area, the system automatically selects the object. It is best to judge that the object is close to the camera.

[0101] (11) The processing means, as a function of the image analysis, It has a function to perform image analysis, and as a process corresponding to the analysis results, it is possible to provide warnings regarding vehicle driving. It is preferable that the system executes a process of providing the driver with necessary information.

[0102] By alerting the driver to the situation ahead, the system can improve visibility as the vehicle moves forward. This can help prevent accidents that may occur in the future.

[0103] The image analysis function calculates at least one of the following: the distance between the vehicle and the vehicle ahead and the relative speed. It would be a good idea to include a feature that detects high collision risk situations based on vehicle distance or relative speed. It is advisable to detect the presence of a vehicle in the vicinity of the intersection and notify the driver of a high risk of collision by issuing an alarm or other means. The distance is determined by the size of the vehicle in front in the image, e.g. the size of the license plate, the tread (tire The relative speed can be calculated from the following information: the distance between the vehicles, the width of the tail lamps, etc. It is advisable to calculate the change from the rate of change.

[0104] The image analysis function may include a function for detecting the color of a traffic light ahead. When the unit's light is red, it is advisable to notify the driver to stop.

[0105] The image analysis function may include a function for recognizing road signs ahead. It is advisable to alert drivers to the information provided by the sign. For example, if a road sign indicates a speed limit, If so, the processing means obtains the speed limit at the current location by character recognition, and If the road sign indicates a no entry condition, the processing means may: If a vehicle attempts to enter a prohibited road, an alarm should be issued to notify the driver. For example, , the vehicle is attempting to travel or is about to cross a road that violates the traffic rules indicated by road signs. If it detects that a driver is violating traffic rules, it will notify the driver of the violation. This will help prevent or reduce the occurrence of traffic violations. For example, the processing means may recognize the character strings of road signs to determine the section in which the vehicle is currently traveling. When a sign indicating that an automatic speed camera is installed in a section is detected, the driver is warned about excessive speed. It is advisable to inform the users to be careful.

[0106] If the image analysis function includes a function to detect congestion in lanes adjacent to the vehicle's lane, For example, whether congestion is occurring can be determined based on the average speed of vehicles in adjacent lanes and For example, the vehicle speed of the vehicle in the adjacent lane may be determined based on a predetermined distance. When the vehicle distance is below a certain standard value and a certain number of vehicles are lined up, It is advisable to determine that a traffic jam is occurring. When it detects congestion in the adjacent lane, it detects vehicles jumping out from between the traffic jams. It is advisable to execute a process to alert the driver to be careful of the following: The lanes to be checked should include oncoming lanes and adjacent lanes traveling in the same direction as the vehicle. In particular, Shopping malls, theme parks, train stations and other crowded places, such as gas stations In areas where vehicles enter and exit, there is a high possibility that the lanes adjacent to the lane in which the vehicle is traveling in the same direction will also be congested. In such cases, it is effective to detect congestion in the adjacent lane traveling in the same direction as the vehicle. This can prevent accidents caused by sudden jumps out of the road. When the driving speed is below a certain speed, for example, 30 km / h, a warning sound is emitted to warn of pedestrians jumping out into the road. This will help to avoid unnecessary warning sounds. In addition, it is possible to detect situations where there is a high risk of the vehicle jumping out from the front image and warn the driver. For example, if there are a series of parked vehicles on the side of the road, parking meters are installed. If there are parked vehicles in the parking lot, it is a good idea to warn people not to rush out into the street.

[0107] For example, car navigation devices can provide traffic congestion probe information, general road and highway driving information, and surrounding facility information. The system has a function to alert drivers to driving caution based on information about residential areas, schools, etc. By using this system, it is possible to Even if the vehicle is in a traffic jam, a warning can be issued.

[0108] The image analysis function detects animals (dogs, cats, raccoons, wild boars, etc.) around the vehicle before starting off. When the processing means detects an animal or a person around the vehicle, It is a good idea to inform the driver that there are animals around the vehicle. This will prevent accidents immediately after starting. The occurrence of the above can be suppressed.

[0109] The image analysis function detects that the vehicle is moving forward slowly based on the flow of the surrounding scenery. The processing means may include a function for detecting when the vehicle is moving forward slowly. For example, if the force applied to the brake pedal is relaxed after the vehicle has stopped, Therefore, when the vehicle starts to move little by little due to creep, it is necessary to confirm that the vehicle is moving and that the vehicle is being followed. For example, if the vehicle speed is creeping, the driver can be informed of the risk of a collision. When the vehicle is moving forward at a forward speed, it may be judged to be moving forward slowly. The speed of the flow of the surrounding scenery can be detected by detecting the optical flow.

[0110] The image analysis function detects if the vehicle ahead is a large vehicle, such as a bus or a box truck with an aluminum loading platform. The processing means may include a function for determining whether the vehicle ahead is a large vehicle, When it detects that the vehicle is approaching a traffic light, it is advisable to notify the driver of the presence of the traffic light. For example, It is recommended to issue a voice message such as "Watch out for traffic lights." The presence or absence of traffic lights is stored as POI information in advance. For example, when a large vehicle is traveling ahead, It becomes difficult to see the traffic lights of the opposite lane. Anticipate and keep a sufficient distance so that traffic lights ahead are easily visible However, if you neglect to pay attention, you may miss the traffic light in your direction. In addition, in rural areas, there are no traffic lights for the opposite lane or for pedestrians. When the vehicle ahead is a large vehicle, the presence of a traffic light can be notified. This reduces the risk of drivers overlooking traffic signals. Instead of automatically informing the driver, the processing means may select a driving style that reduces the risk of missing a traffic signal. For example, it is a good idea to warn drivers to keep a sufficient distance from the vehicle ahead. For example, the distance to an intersection with traffic lights may be displayed as a warning to the driver. It is advisable to warn the driver when the stopping distance is longer than the vehicle speed. It depends on the weather (rain or not). The safe stopping distance depends on the vehicle speed, the weather (rain or not). For example, if the safe stopping distance in clear weather is 50 m, In this case, the processing means is: when the current weather is fine, the distance to the large vehicle ahead is less than 50m, - Outside what is considered an urban area by navigation devices, there is no traffic light within 100m in the direction of travel When there is an intersection where the machine is installed, it is a good idea to issue a warning.

[0111] The image analysis function determines whether the road ahead is a slope, and if so, The processing means may include a function for detecting the degree of inclination of the slope. It warns you not to go too fast, slow down, or set the vehicle to hill climb mode. It is good.

[0112] The vehicle is a load handling vehicle, such as a forklift, truck, etc., and the handling means is From the captured image, the system recognizes the text, barcodes, and other marks on the luggage and automatically detects the luggage. It is preferable that the vehicle has a function to determine whether the luggage is to be handled by the vehicle itself. The information represented by the detected sign may be registered in the processing means in advance. If it detects that an attempt is being made to load luggage that should not be handled, an alarm will be issued. It is a good idea to put this theory into practice.

[0113] The processing means preferably has a function of detecting a window region in the image from the optical flow. Further, the processing means determines from the optical flow whether the vehicle has entered a tunnel or It is advisable to have a function to determine whether the vehicle has exited the tunnel or is currently traveling in the tunnel. In addition, the processing means determines from the optical flow that the vehicle is traveling under an overpass (under a parallel road). The processing means may further include a function for determining whether the object is in the middle of the optical flow. It would be good to have a function to determine whether or not guardrails are present and whether or not caution is required regarding gutters.

[0114] (12) The processing means, as a function of the image analysis, detects other vehicles captured in the image outside the vehicle. It has the ability to recognize the characters written on the license plates of both vehicles, and processes the analysis results accordingly. The process and network for recording the license plate information identified by character recognition It is preferable that the system executes at least one of the processes of transmitting the information via the network.

[0115] Whether the vehicle is traveling near a vehicle equipped with an on-board device that has the processing means function of this system, or It is possible to identify the license plate information of other vehicles parked nearby. The processing means is a device that captures the current location of the vehicle and the license plate of the vehicle. The date and time information may be transmitted in association with the identified license plate information. The license plate information obtained along with the report and date and time information is used by the License plate information obtained by the automatic vehicle license plate reading device (N system) Both of these are useful information for knowing the routes taken by many vehicles. By linking the bar with GPS information, a system that can be called a wide-area mobile N system can be constructed. It can be built.

[0116] This system is installed in multiple vehicles and transmits location information, date and time via a network. It would be better to build a central system that collects and associates time information and license plate information. This center system records the location information and date and time of a vehicle with specific license plate information. It would be good to have a function to extract information and identify the route the vehicle took.

[0117] (13) The processing means detects, as a function of the image analysis, a window glass edge on a window portion of the vehicle. It has a function to determine whether a window exists in the window part as a process corresponding to the analysis result. It is advisable to configure the system so that, when the edge of the glass is detected, a notification process is executed.

[0118] The presence of window glass edges in the window area means that the window glass has been left open. By receiving notifications from this system, you can realize that you have forgotten to close the windows. For example, when detecting the edge of a window glass, the difference in refractive index between the window glass and the air causes discontinuities in the image. The notification process may be carried out by, for example, sending a wireless signal to the driver's mobile device, etc. It is advisable to use a system that notifies the driver by wire or email. The notification may be processed by sounding a buzzer provided in the vehicle. The window is covered with a tint film, and the treatment means is a means for treating the area where the window is open and the area where the window is closed. It detects that a window is open based on the difference in brightness between the image and the smoked areas of the glass. good.

[0119] (14) The processing means detects, as a function of the image analysis, whether or not the wiper is operating, Detects whether or not it is raining by detecting whether or not there is an increase or decrease in the number of raindrops on the windshield The function is to process the results of the analysis by determining that it is raining. The system may be configured to execute a process of transmitting information notifying the user that the communication device is connected to the network. stomach.

[0120] By receiving information transmitted from this system, the vehicle can detect whether there is rain in the area where the vehicle is traveling. The processing means can know whether it is raining or not by transmitting the information that it is raining to an external device such as a server. The external device may notify the vehicle's current location information to the external device. By collecting information from vehicles, it is possible to create a map of areas where it is raining. Furthermore, the processing means notifies the user that it is raining and transmits the current time information. This allows the external device to receive information about the time-dependent changes in areas where it is raining. Information can be obtained.

[0121] The external device may be, for example, a server of an organization that provides weather information. The method is to take a photo when the wiper is not in the picture, for example, when the wiper is down. It is recommended to upload the image to the server. The image uploaded to the server can be used to calculate the rainfall. The server can obtain information about the amount of rainfall collected from the vehicle. Such data can be used as big data in various application fields. For example, The system can be used to predict localized torrential rains. If there are droplets attached but the amount (number) of droplets does not increase or decrease, it is safe to assume that the rain has stopped. When the processing means determines that the rain has stopped, it is preferable that the processing means notify the server that the rain has stopped. The precipitation forecast can then be used to obtain more detailed information about rainfall.

[0122] (15) The processing means, as a function of the image analysis, determines whether the sky in the image is reflected or not. It has the function of judging the weather, and transmits weather information as a process corresponding to the analysis results. It is preferable that the system executes the process of transmitting the information from the device.

[0123] Weather information collected from multiple vehicles is used as big data on meteorological information. For example, it is possible to distinguish between clear and sunny days based on the color of the sky and the presence or absence of clouds in the image. It is also advisable to determine the weather, such as morning sun, evening sun, cloudy, etc., based on the height of the sun and the brightness of the sky. It would be good to detect the strength of the sun and send information about the strength of the morning sun or afternoon sun. In addition, the vehicle's current location information can be transmitted. The external device that collects this information can be used to The information that the afternoon sun is strong and the vehicle's current location information are sent to other vehicles heading to that location. Drivers who receive this information should be cautious of morning or afternoon sun. You may realize that it is necessary.

[0124] (16) The processing means has a function of detecting sound information from a microphone mounted on a vehicle. The detection result of the sound information is associated with the analysis result of the image analysis function, and It is preferable that the system executes processing according to the received signal.

[0125] Compared to processing based only on the results of sound detection or image analysis, By associating the two, more appropriate processing can be performed.

[0126] The processing means, as a function of image analysis, detects the state of a display unit of a device having a display unit mounted in the vehicle. It has the function of detecting changes in the state of the sound, and processes the detected sound information according to the analysis results. When the change in the display state is synchronized, the detected sound information and the change in the display state are It is advisable to execute a process of storing the information in association with the device.

[0127] When the processing means detects sound information from the microphone, the processing means associates the sound information with a change in the state of the display unit. Based on the attached and stored information, the device that emitted the sound can be identified. For example, the processing means may have a voice recognition function, and may recognize sound information from a microphone as being generated from a specific device. If it was issued from a specific device, it is recognized by voice recognition. This will allow you to avoid unnecessary voice recognition processing. The processing means can also detect whether the sound information from the microphone is generated from a specific device. If it is determined that the sound is generated, the area of ​​the image of the inside of the vehicle where the sound is generated is distinguishable from other areas. For example, the area where the device that generated the sound is captured can be recorded. It is a good idea to record an image of a surrounding rectangle or a closed curve of any shape on top of the original image. It is a good idea to record a semi-transparent geometric pattern over the area where the device generating the noise is located.

[0128] The processing means detects emergency vehicles, such as ambulances, fire engines, and police cars, in the image as a function of image analysis. The processing means may have a function to determine whether an emergency call is received from a microphone. When the system detects the sound of a vehicle siren and an emergency vehicle in the image, it automatically associates the two. It is preferable to determine that an emergency vehicle is in the vicinity of the vehicle. The processing means determines that an emergency vehicle is in the vicinity of the vehicle. A message to alert the driver may be sent.

[0129] (17) There are multiple cameras installed inside the vehicle. The processing means detects the positions of other cameras captured in the image as a function of the image analysis. It has the function of detecting the positional relationship between multiple cameras as a process corresponding to the analysis results. The system may be a system for executing the process.

[0130] It is possible to capture images of a wide area inside the vehicle that would be difficult to capture with a single camera. Once the relative positions of the cameras are identified, the relative positions of the images captured by each camera can be determined. Cut.

[0131] The processing means identifies the models of other cameras captured in the image and identifies the angle of view from the models. The processing means has a function of synthesizing images captured by a plurality of cameras. By combining images, the viewer can see more intuitively than if they were to display individual images from the cameras. This allows the driver to effectively grasp the situation inside the vehicle.

[0132] For example, at least a part of the functions of the processing means may be realized by a viewer such as a personal computer. If you display a camera icon or other sign in the image to indicate the camera's location, The position of the camera in the vehicle can be identified from the image captured by the camera (self). It is difficult to determine the interior of the vehicle from images captured by other cameras that include the camera in question. For example, images captured by three cameras can be synthesized and displayed. When using a composite image, it is recommended to display the positions of the three cameras using icons or other means. It is a good idea to use different icons depending on the model or angle of view.

[0133] The processing means displays the image captured by each camera and indicates the position of the camera within the image. It is recommended to have a function to display the camera's position and the field of view (area captured by the camera). For example, depending on the function, such as a full-sphere 360-degree camera, a half-sphere 360-degree camera, or a wide-angle camera, The user who sees the image can see which camera is which. It is possible to intuitively grasp the area being imaged.

[0134] (18) The camera has an image circle smaller than the diagonal length of the imaging surface. It is a 60 degree camera, The processing means may include a basic pattern for calculating a feature amount of an image as a function of the image analysis. The system has a function of deforming the image according to the position within the image circle. It is good.

[0135] The basic pattern is used to correct the image distortion within the image circle. This can reduce the load on the processing means. The angle is defined for images captured using a standard lens (a typical wide-angle lens). The projection methods to obtain the image circle include, for example, orthogonal projection, equidistant projection, and stereographic projection. There are projection methods such as equi-solid angle projection, and the basic pattern is transformed according to these projection methods. To obtain an image circle, it is recommended to use a circular fisheye lens. The imaging surface is rectangular, and the diameter of the image circle should be shorter than the long side. It is better if it is equal to or shorter than the short side.

[0136] For example, prepare a template pattern of the image you want to detect as a basic pattern. In this case, the template pattern is changed according to the position in the image circle. In addition, for example, when recognizing an image using Har-like features, Each of the multiple rectangular patterns for obtaining the like feature is placed according to its position in the image circle. It is a good idea to deform it as needed.

[0137] When performing image recognition, the area within the image circle that contains the specific object to be detected is selected. It is a good idea to limit the area where the image is congested and perform image analysis within that area. For example, the functions of the in-vehicle device of this system can be realized by a drive recorder. The recommended installation position of the recorder is predetermined, so the driver can easily see the location of the recorder within the image circle. The areas where the vehicle is visible, the areas where traffic lights are visible, and the areas where people in the back seats are visible are mostly decided. Therefore, depending on the person or object to be detected, the image resolution is adjusted to detect the person or object. It is advisable to limit the area where the analysis is performed to the area in which the person or object is captured. For example, When analyzing the image, it is advisable to analyze the area in which the windshield is reflected.

[0138] In addition, for example, a specific area of ​​the image circle can be converted into a flat image (parallel projection image). Then, matching can be performed on the transformed planar image using normal basic patterns.

[0139] In addition, the image in the image circle is mapped onto a sphere using a predetermined transformation function, Zoom into a portion of an image mapped onto a surface and analyze the image as a flat image while ignoring distortion If you want to detect a specific object, such as a road sign, select the area in which the specific object appears. If you want to detect road signs, you can zoom in on the image ahead and perform image analysis. It is advisable to zoom in on the area containing the road sign and perform image analysis.

[0140] (19) The processing means, as a function of the image analysis, from the viewpoint of privacy protection A function for identifying an area of ​​an image that should not be made public as a privacy protection area; As a process corresponding to the analysis result, the person appearing in the privacy protection area is A process for generating an image that cannot be identified by the above-mentioned method from an image captured by the camera. It is preferable that the system executes the above.

[0141] By making it impossible to identify a person, the privacy of that person is protected. For example, the area that should not be made public may be an area in which the interior of a vehicle is shown. In addition, when using a spherical camera to capture images, the interior of the vehicle is also captured. There may be cases where you do not want to keep the image. In such cases, you can black out the area inside the car, for example. Alternatively, the interior area may be pixelated and the exterior image may be recorded as is. The means may be, for example, image analysis to distinguish between an area showing the inside of the vehicle and an area showing the outside of the vehicle. For example, the optical flow occurs at a speed equivalent to the vehicle speed. The area may be outside the vehicle. The processing means may be configured to allow the user to set the privacy protection area in advance. It would be good to have a function that allows this.

[0142] a normal mode for recording images capable of identifying people within the privacy protection area; Images in which people captured in privacy protection areas cannot be identified by appearance It is preferable that the user can select a privacy mode in which the vehicle is recorded. Switch between normal and privacy modes depending on the purpose and situation of use. can be done.

[0143] The privacy area may be an area in which a person is photographed. By analyzing the image, a person is detected, and the area enclosed by the outline of the detected person is treated as a privacy mask. Alternatively, only the face of the detected person can be used as the privacy area. In this case, it is recommended to display the person as a silhouette. It is a good idea to display only the outline of the image, and display an image with no people inside the outline. Objects appear transparent. It is useful to display the edges of moving objects, for example as the outline of a person. .

[0144] The processing means may display an avatar in place of the person in the area in which the person is photographed. The avatar should resemble the original person, for example in terms of gender, body shape, facial appearance, etc. In response to the user's request, an avatar that reflects the user's characteristics can be displayed. The processing means identifies a person appearing in the image and associates the identified person with the It is recommended to display the avatars that are assigned to each person. It is recommended to create an avatar for each person in advance. It is a good idea to move the avatar in accordance with the person's movements. For example, when the person raises their hand, , have the avatar raise its hand, and when the person turns to the side, have the avatar turn to the side as well. The facial expression of the avatar may be changed according to the facial expression of the person. When it detects a smiling, sad, or angry expression, it adjusts the avatar's facial expression. It is advisable to change the facial expression to a smiling, sad, or angry one. It would be good to read the emotions of the person and change the color of the avatar according to the emotion of the person. For example, when a person is detected as angry, the color of the avatar is changed to a color that represents anger, e.g. If necessary, change it to red.

[0145] The processing means extracts the privacy protection area from the image captured by the camera before recording the image. The present invention generates an image in which a person captured in the image cannot be identified by appearance, and It is also advisable to record images that have been taken with a camera without the need for privacy measures when playing back images taken with the camera. - Converts people captured in the protected area into images that cannot be identified by appearance It is good to display it.

[0146] (20) The processing means converts the result of the image analysis into a text as a process corresponding to the analysis result. It is preferable that the system executes a process of digitizing or coding the information and recording it.

[0147] By converting the images into text data or encoding them, it is possible to easily extract the desired images from the recorded images. For example, when a person is detected in an image, the text data can be When a red light is detected in an image, the text data is "Red light The generated text data can be used to generate the text data. Instead of text data, it is recommended to record the image analysis results and the code. It is a good idea to match the above and code (convert to binary data) the results of image analysis.

[0148] (21) The processing means detects areas of black spots in the image as a function of the image analysis. It has the function to detect black spots in the image and process them according to the analysis results. It is preferable that the system executes a process of brightening and restoring the image to be displayed on the display screen.

[0149] Users who view the image can easily recognize objects and people in the black-blurred areas. When capturing images using a spherical camera, the image contains both the bright scenery outside the vehicle and the dark interior of the vehicle. Since the images are taken at the same time, the area where the interior of the car is captured is prone to blackout. The process of detecting the area and restoring it to a brighter state is done by capturing images using a spherical camera. This is particularly effective when the image is brightened. For example, gamma correction, color correction, black and white It is advisable to convert the image and enhance the contrast.

[0150] (22) The processing means allows a user to select a plurality of partitions from an imaging range captured by the camera. and a function of weighting each of the sections, The imaging conditions are determined based on the area and the weight assigned to the area, and the imaging conditions are determined. The system may preferably have a function for causing the camera to capture an image.

[0151] The user weights the areas of the image that he or she wants to see more heavily and weights the areas that he or she does not want to see more lightly. By doing so, it is possible to obtain an image under imaging conditions suitable for the area that the user wants to see. The imaging range can be divided into any shape according to the shape of the area you want to see and the area you do not need to see. For this reason, for example, the image can be divided into square lattice sections uniformly regardless of the image content. This makes it possible to provide more appropriate weighting for the user than if the weighting method were used .

[0152] In particular, when using a 360-degree camera, the vehicle model can be captured more clearly than when using a general camera. The difference in the position of the vehicle within the imaging range increases with each shot. A general-purpose weighting table set in the firmware of the method and the actual expected imaging range The possibility of mismatching the relationship between the two plots increases. The function that allows users to determine weighting using PC browser software, etc., is 360 This effect is particularly noticeable when using a 3-degree camera. In this state, the image captured by the camera is analyzed using AI software on a PC to determine which part of the image range is captured. For example, the weight "5" is assigned to the road area and "1" is assigned to the face area. Weight "3" is assigned to the instrument panel, weight "1" is assigned to the area outside the circle image. It is recommended to assign a weight of “0” to

[0153] The processing means may determine the imaging conditions, such as the gain and the exposure time. The backs of the spherical cameras are placed facing each other to form a spherical camera, and the imaging conditions of the two half-spherical cameras are This makes it possible to synthesize images captured by two hemispherical cameras. This has the effect of making the boundary between the two images less clear.

[0154] The processing means, when making the user divide the area, makes the user surround a part of the area within the image pickup range with a closed curve, The area inside the curved line should be considered as one section. If the area is divided into two by a curve, each of the divided areas can be treated as one section. The step may allow the user to input a weight for each section. It is advisable to allow the user to select from a list such as a pull-down or pop-up.

[0155] The processing means may be provided in place of or in addition to a function for inputting the weights as numerical values. In addition, it is advisable to have a function for inputting the type of partition. The type of partition is assigned a weight in advance. When the type of the section is input, the processing means The weight may be associated with the section. It is a good idea to provide categories such as the screen, car navigation screen, windshield, rear seat, passenger seat, etc. The processing means may have a function that allows a user to change the correspondence between the type of the partition and the weight. It is advisable to allow the user to register a list of image types in advance.

[0156] The processing means allows the user to weight each section, for example, within the imaging range. It is recommended to have a function to set a section by allowing the user to specify the area to be viewed. It is recommended to give the area you want to see a higher weight than the other areas. This function is In the function that allows the user to input the type of partition, the partition is divided into the partition that the user wants to see and other partitions. This corresponds to an example of dividing the products into two types:

[0157] The processing means may allow the user to divide the imaging range in an interactive manner (wizard format). For example, the processing means may say, "Please surround the unwanted area." or "Please paint the unwanted area." Please fill in the area.' message is displayed, and the area the user has enclosed or painted is The crushed area may be recognized as an "unnecessary area." "Please fill in the area of ​​the pedestrians," "Please fill in the area of ​​the road," The message is output and the filled area is designated as a section with a driver type or a road type. It is better to recognize it as a section with a different shape. The operation of enclosing it with a closed curve or filling it is as follows: For example, it is preferable to use a touch panel or a pointing device such as a mouse. .

[0158] The processing means performs a function of dividing the imaging range into a plurality of sections based on the captured image, This can be achieved by AI. For example, it can identify the interior and exterior of the car, the seats, the dashboard, etc. from images. The camera automatically detects the vehicle's interior, such as the dashboard or dashboard, and adjusts the imaging range for each automatically detected component. The processing means displays the classification result on the display screen and determines whether the classification is acceptable. It is better to let the user make the decision. If the user determines that the classification made by the AI ​​is incorrect, The processing means may have a function of allowing the user to modify the division. If the user determines that the input or type is incorrect, the processing means displays a pull-down or pop-up It is a good idea to have the user select the correct type from a populated list.

[0159] (23) The processing means allows a user to select a plurality of partitions from an imaging range captured by the camera. and a function to weight each of the sections, and correspond to the analysis results. The process is to compress blocks of areas with relatively small weights into blocks of areas with relatively large weights. It is preferable that the system executes a process of compressing the image in blocks larger than those used for compressing the image.

[0160] The user can increase the weight of the partitions with high importance and decrease the weight of the partitions with low importance. This allows for more compression of less important areas, resulting in a uniform compression of the entire image. Compared to compression, the image data is reduced while maintaining high image quality for important areas. For example, the processing means may reduce the size of the weighted area by 8×8 pixels. In the low-weighted area, each pixel is treated as a block, and in the low-weighted area, each pixel is treated as a block. It is recommended to compress the data in this way.

[0161] The area outside the circular image formed on the rectangular imaging surface of the camera contains the image of the lens barrel. The image outside the circle is unnecessary, so fill this area in black. This can reduce the amount of image data that includes pixels outside the circle. In addition, the pixels outside the circle are excluded, and only the pixels inside the circle are raster scanned. This can reduce the size of the image data.

[0162] (24) The processing means includes, as a function of the image analysis, a function of detecting a following vehicle and a function of detecting a following vehicle. It has a function to detect the distance to the following vehicle, and processes the analysis results to detect the distance to the following vehicle. The system may be configured to execute a process to detect aggressive driving by a following vehicle based on the distance between the following vehicle and the vehicle. .

[0163] The processing means executes a process according to the determination result to provide an appropriate countermeasure against aggressive driving. For example, even if the vehicle speed is equal to or greater than a certain threshold, When the same following vehicle is present within a specified distance for a specified period of time or longer. It is preferable to determine that the following vehicle is engaging in aggressive driving.

[0164] It is better to use a camera that has the function of measuring the distance to the object for each pixel. For example, a back-illuminated time-of-flight distance image sensor is used as the image sensor for the camera. It is a good idea to use a sensor.

[0165] When the processing means determines that the following vehicle is tailgating, the processing means performs processing according to the determination result. For example, the system can notify drivers, record rear images, advise drivers, and enforce traffic laws. When reporting to a traffic enforcement agency, In the case of a case where a driver is driving aggressively, the driver may attach an image of the vehicle and the driver. Image analysis shows that the vehicle engaged in aggressive driving has overtaken and circled in front of the vehicle. When it detects that a vehicle is suddenly stopped, it notifies the driver that they may be harassed by someone stopping suddenly. It is recommended to do so.

[0166] The processing means detects people and vehicles around the vehicle and controls the vehicle according to the distance to the people and the vehicle. For example, the processing means may determine the speed of the vehicle, the distance to the person ahead, and whether the person is moving forward from the vehicle. Based on the result of the judgment of whether the vehicle is on the planned driving route, it is judged that there is a high possibility of contact with a person. Once this is determined, control should be performed to stop the vehicle.

[0167] (25) The function of the processing means according to any one of (1) to (24) above is implemented by a computer. It would be a good idea to create a program to help the data be realized.

[0168] The inventions shown in (1) to (24) above can be combined in any manner. For example, the invention shown in (1) includes all or part of the invention, and at least one of the inventions listed in (2) and onward. In particular, the invention shown in (1) may be configured to include at least a part of the above. It is preferable that the invention be an invention that includes at least a part of the configuration of at least one of the following inventions. , Extract any configuration from the inventions shown in (1) to (24) and combine the extracted configurations The applicant of this application intends to obtain rights to the invention including these configurations. do Effect of the Invention

[0169] Not only does it record images captured by cameras installed in vehicles, but it also This can be utilized more effectively.

[0170] The effects of the present invention are not limited to the above, and the configurations disclosed in the present specification and drawings, etc. The effects achieved by the above are also disclosed, and the configuration achieving said effects is also disclosed in the divisional application. For example, in this specification, The above-mentioned sections clearly indicate the effect that can be achieved, and even if there is no description that "can...", the effect can be achieved. In addition, even without such a description, the effect of the configuration can be understood. exist. [Brief description of the drawings]

[0171] [Figure 1] FIG. 1 is a diagram showing the windshield, dashboard, etc. inside a vehicle in a state in which a drive recorder as a system according to a first embodiment is mounted in the vehicle. [Diagram 2] FIG. 2 is a block diagram of a drive recorder as a system according to the first embodiment. [Diagram 3] FIG. 3 is a block diagram of a system according to a modified example in which the functions of the processing unit are realized by two devices, an in-vehicle device and a server. [Figure 4] FIG. 4 is a block diagram of a system according to a modified example in which the functions of the processing unit are realized by an in-car device and an external device such as a personal computer. [Diagram 5]FIG. 5 is a diagram of an imaging surface for explaining a method of dividing an imaging range into a plurality of sections in a method according to a comparative example. [Figure 6] FIG. 6 is a diagram of an imaging surface for explaining a method of dividing an imaging range into a plurality of sections in a system according to the twenty-second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0172] [First Example] A system according to a first embodiment of the present invention will be described with reference to FIGS. The system function is realized by a drive recorder installed in the vehicle.

[0173] FIG. 1 shows a front windshield, dash cam, and other images of a vehicle with a drive recorder 1 mounted thereon. The vehicle in which the drive recorder 1 is installed is a passenger car. The live recorder 1 is located above the windshield 2 of the vehicle and is positioned in the room mirror near the center in the left-right direction. The drive recorder 1 is attached to the passenger seat adjacent to the rearview camera 3. The drive is attached to the inner surface of the windshield 2 by a mounting member such as the above. The recorder 1 is connected to a cigarette lighter socket 5 of the vehicle via a power cable 4. When the accessory power is turned on, power is supplied from cigarette lighter socket 5 to dash cam 1. The drive recorder 1 includes a main body 10 and a 360-degree spherical camera 11. The camera 11 is fixed in position and attitude relative to the vehicle during normal use. .

[0174] FIG. 2 is a block diagram of a drive recorder 1 as a system according to the first embodiment. The drive recorder 1 includes a processing unit 20, a camera 11, a GPS receiver 12, and an SD card reader. 13, an acceleration sensor 14, a speaker 15, a display 16, and an operation button 17. Strictly speaking, the SD card reader 13 should be called an "SD card reader / writer." However, in this specification, it will simply be referred to as an "SD card reader."

[0175] The speaker 15 and the display 16 are output means for notifying the user of various information. The speaker 15 outputs a warning sound or voice based on the control of the processor 20. The display 16 outputs various information based on the control of the processing unit 20. The operation buttons 17 allow the user to operate the drive recorder 1 in various ways. It functions as an input means for giving commands.

[0176] The camera 11 is a 360-degree spherical camera that captures images of the vehicle in the forward / backward, up / down, and left / right directions. For this reason, the camera 11 captures the entire area as a video. The camera captures images of the outside of the vehicle through the driver's seat, passenger seat, rear seat, interior, and exterior windows. The camera 11 also captures the interior of the vehicle, including the instrument panel, center console, etc. The captured video is input as image data to the processing unit 20. Based on the image data input from 1, the SD card connected to the SD card reader 13 The image is recorded.

[0177] The GPS receiver 12 receives the position information of the vehicle at the current time based on the instruction of the processing unit 20. The location information is based on the signal from the GPS satellites and includes the time, the vehicle's speed, The information includes longitude, latitude, altitude, etc. The processing unit 20 records the history of this position information. Carry out the analysis.

[0178] The SD card reader 13 is controlled by the processor 20 to insert the SD card into the SD card slot. Read data from or record data to a connected SD card.

[0179] The acceleration sensor 14 detects the acceleration and inclination in each of the three axes (x-axis, y-axis, z-axis). A three-axis type sensor is used to detect the acceleration. The acceleration sensor 14 constantly processes the detected value. The processing unit 20 acquires the three-axis acceleration information, for example, every 10 ms.

[0180] The processing unit 20 includes a CPU 20a, a ROM 20b, a RAM 20c, and other memories, The processing unit 20 includes a ROM 20b and other peripheral circuits. The processing unit 20 executes these programs to perform various functions. Various programs include the operating system (OS), GPS information processing program, These include programs such as image analysis programs.

[0181] The processing unit 20 executes a GPS information processing program to process the GPS information received by the GPS receiver 12. The received GPS information is stored in the SD card connected to the SD card reader 13. The unit 20 executes an image processing program to convert the video captured by the camera 11 into a time-series video. Image data associated with the information and location information is constantly recorded in a designated recording area of ​​the SD card. .

[0182] The processing unit 20 executes an image analysis program to analyze the image captured by the camera 11. The image is analyzed. Furthermore, the processing unit 20 executes various processes corresponding to the analysis results. .

[0183] By mounting the system according to the first embodiment on a vehicle, a camera mounted on the vehicle Not only can the captured images be recorded, but the images can also be used more effectively. When the image captured by the camera 11 is converted into a video, the subject can be analyzed by analyzing the video. It will be possible to easily determine the movement of the subject and the presence or absence of the subject before and after a certain point in time. .

[0184] Next, a modification of the first embodiment will be described. In the first embodiment, the drive recorder 1 is installed in a passenger vehicle. In the modified example, however, the drive recorder 1 may be installed in another type of The camera is attached to a vehicle such as a freight vehicle or a special vehicle. In a modified example, the camera 11 captures images of both the inside and outside of the vehicle. For example, the terms "inside the vehicle" and "outside the vehicle" can be interpreted as follows: good.

[0185] When the vehicle body clearly separates the interior and exterior spaces, the body The inside of the vehicle is called the "interior" and the outside of the body is called the "outside". For example, the passenger space In the case of a van, where the passenger space and cargo space are connected, It is better to call it the inside of the car. In the case of a sedan, where the passenger space and the cargo space are separated, Both the passenger space and the cargo space may be referred to as the interior of the vehicle. In an automobile, the passenger space and the interior of the enclosed cargo area are called the interior of the vehicle. In the case of an open-top freight vehicle (flatbed vehicle) with a raised loading platform, the empty space that accommodates the cargo on the loading platform is It is difficult to clearly distinguish between the space inside and the space around it, but it is possible to accommodate open cargo. The passenger space of an open freight car can also be called the interior of the car. In some special vehicles, such as forklifts, both the passenger space and the cargo space are open. The passenger space and cargo space are clearly separated from the surrounding space. Even in such a case, it is possible to maintain both passenger space and cargo space. It is better to call it the interior of the vehicle. For example, the space above the seats where the passengers sit, and the space above the seats that hold the cargo and allow it to go up and down. The space above the forks that are moving can be called the interior of the vehicle. The space that can be thought of as moving with the vehicle is called the inside of the vehicle, and the space around it is called the outside of the vehicle. It is good.

[0186] In the first embodiment, a 360-degree spherical camera is used as the camera 11. Alternatively, a wide-angle camera may be used. When using a wide-angle camera, multiple It is advisable to mount cameras on a vehicle to enable imaging of a wide area.

[0187] In the first embodiment, the function of recording the image captured by the camera 11 is realized by the in-vehicle device. In the first embodiment, the image captured by the camera may be realized by a server that communicates with the in-vehicle device. The function of analyzing the captured images is realized by the in-vehicle device, but it is realized by the server that communicates with the in-vehicle device. Good too.

[0188] As described above, the processing unit 20 of the present system may be configured as one vehicle-mounted device, or may be configured as multiple As in the first embodiment, the function of the processing unit 20 may be implemented by the drive recorder 1. A microcomputer and an application program stored in the memory In addition, the functions of the processing unit 20 may be realized by, for example, a dual function of the in-vehicle device and the server. This will be achieved with one device, with the in-vehicle device recording the images and the server analyzing the images. In addition, the functions of the processing unit 20 may be realized by an in-vehicle device and an external device such as a personal computer, and the image may be recorded. It is preferable that the on-board device records the images, and the external device analyzes and displays the images.

[0189] FIG. 3 shows a modified system in which the functions of the processing unit are realized by two devices, an in-vehicle device and a server. 1 is a block diagram of a system in which a vehicle-mounted drive recorder 1 includes a communication device 21. 30 includes a processing unit 31, a communication device 32, an input device 33, and a display 34. The communication device 21 of the live recorder 1 and the communication device 32 of the server 30 are both connected to a communication network. The device is connected to a communication network 40 and transmits and receives data to and from other devices via the communication network 40. The processing unit 20 of the drive recorder 1 and the processing unit 31 of the server 30 are connected to a communication network 40. Data communication is carried out between them.

[0190] The input device 33 is operated by a user to input various commands and data to the processing unit 31. The processing unit 31 displays the processing results and various messages to the user on the display 34. The message is displayed.

[0191] FIG. 4 shows a modified example in which the functions of the processing unit are realized by an in-vehicle device and an external device 50 such as a personal computer. The external device 50 includes a processing unit 51, an SD card reader 52, and a , an input device 53, and a display 54. 13, the SD card 60 on which the image data is recorded is inserted into the SD card reader 52 of the external device 50. When the external device 50 is connected to the SD card 60, the processing unit 51 of the external device 50 can read and write the images recorded on the SD card 60. The image data can then be accessed.

[0192] The processing unit 51 writes data to the SD card 60 based on a command input from the input device 53. The processing unit 51 reads the recorded image data and performs image analysis or image processing. The analysis or processing results are displayed on the display 54.

[0193] Next, the second to twenty-fourth embodiments will be described. In addition, the following various embodiments will be described in detail. The system may be, for example, any of the block diagrams shown in FIG. 2, FIG. 3, and FIG. 4. This is achieved in a variety of ways.

[0194] [Second Example] Next, a second embodiment will be described. In the second embodiment, the processing unit of the system performs image analysis. The function of this is to determine whether the movement of a person in an image corresponds to a specific movement registered in advance. When the movement of a person corresponds to a specific movement, the processing unit performs the analysis. As a process corresponding to the above, recording of an image capturing a specific movement of a person, Sending images to the server 30 (Fig. 3), notifying that a person has made a specific movement, and issuing an alarm. At least one of the following operations is performed:

[0195] The images recorded or sent to the server show that a person was performing a specific movement. By reporting that a person has made a certain movement, , it can notify people outside that there is a person making a specific movement. This allows a person who is making a specific movement to become aware that a warning has been issued to them. This motivates them to stop making certain moves.

[0196] Next, an example of a "person making a specific movement" will be described. A person who performs a specific movement may be, for example, a driver of a vehicle. This includes actions other than driving, such as looking at a mobile device such as a smartphone, eating, etc. Actions include eating, looking away, dozing, etc. For example, whether or not the person is looking away The position of the driver's eyes is detected and the recorded image is used to determine whether the driver is aware of the situation. The processing unit can be used to improve driving safety and analyze the causes of accidents. If it detects that you are using a mobile device, it will warn you not to hold the mobile device while driving. If the driver notices the warning and stops the specified movement, the driver can avoid the danger in advance. The server 30 to which the data is sent may be, for example, a server of a vehicle manager, or The server is the police server. The vehicle manager can then use the data sent to determine the driver's driving behavior. Based on the images transmitted, the police can assess the driver's driving behavior. , and analysis of the cause of accidents can be carried out.

[0197] A person who makes a specific movement is, for example, a suspicious person outside the vehicle. Actions of trying to break into a building, such as trying to break a glass window with a hammer, By issuing an alarm, it is possible to detect suspicious people trying to break into the vehicle. This can discourage the intruder from entering the vehicle. The images sent are used as useful information to catch thieves after a car theft has occurred. It can be used to notify vehicle owners and drivers that a person has made a specific movement. This allows the owner or driver to quickly return to the vehicle and prevent damage from vehicle theft. In addition, the driver or owner who receives the notice can also report the matter to the police. This function can be useful for crime prevention.

[0198] A person who behaves in a specific way may be, for example, a suspicious person in a car. Methods of car theft, such as illegal entry into a parked car, suspicious behavior in a parked car, The act of removing the device from the inside of the vehicle will be used to make the thief flee in a panic. This will have the effect of reducing the damage caused by car theft. The images sent to 30 provide useful information for arresting thieves after a car break-in occurs. This can be used to inform the vehicle owner or driver that a person has made a certain movement. This notification allows the owner or driver to immediately return to the vehicle or call the police. This will help reduce the damage caused by car theft. Examples of such attacks include sending emails with a photo of the examiner attached. For example, vehicle managers, drivers, owners, police, etc.

[0199] An example of a person who moves in a specific manner is a passenger in a commercial vehicle carrying unspecified people. Specific movements include various robbery methods, such as brandishing a knife or raising a gun. The processing unit executes a process to issue an alarm, If the thief thinks twice about committing the crime, the damage can be prevented. The images transmitted to the server can be used as useful information for arresting the robber. By notifying the vehicle manager that a person has made a certain movement, the manager can detect the occurrence of a robbery. The processing unit can also use the communication function to report robbery cases to the police. It has the function of reporting the occurrence of such incidents to the police.

[0200] For example, a person who moves in a specific way may be a baby or infant riding in a car. For example, the movement of an infant waking up from sleep can be detected by image analysis. The processing unit notifies the guardian by e-mail or the like. There are times when you want to go shopping or run some errands with the engine running. At this time, if you forcefully wake up the baby, he or she will become upset, so it is best to let the baby sleep. However, parents should be careful to keep their infants and young children safe when they are away from the vehicle. I'm worried that my baby might be awake. By using this system, I can protect my baby when he wakes up. An email is sent to the mobile device of the parent, so the parent can immediately rush to the vehicle and take care of the baby. You can deal with awakening.

[0201] For example, a person who needs care or rescue outside the vehicle may be considered as a person making a specific movement. For example, actions specific to a person requiring care, or actions corresponding to situations where rescue is deemed necessary, An example of a specific movement of a person requiring assistance is walking with a cane, a bent waist, and a shaky gait. Examples of actions that may indicate rescue is necessary include falling down and foaming at the mouth. Examples of actions that correspond to a situation where someone is collapsing while spewing blood are included.

[0202] The processing unit detects people making specific movements by capturing images of the front and rear of the vehicle. a function for selecting at least one of the images of the vehicle interior and the image of the vehicle interior; and The processing unit detects the person from the at least one image. This allows the processing unit to filter out unnecessary or low-importance images. Since the analysis of the above is not performed, there is no need to provide the processing unit with excessive processing power.

[0203] The processing unit has a function to detect a person making a specific movement, and a function to detect drowsiness of the driver. For example, the processing unit measures the pulse rate from the variation in the redness of the driver's skin, Based on the variation, the processor determines whether the driver is drowsy. This is a system that will improve the accuracy of drowsiness detection by combining body temperature measurement and pulse measurement using microwaves. When the processing unit of this system detects drowsiness, it lowers the temperature setting of the air conditioner, Or execute a process to issue a warning by voice or sound.

[0204] [Third Example] Next, a third embodiment will be described. The processing unit of the system according to the third embodiment performs image analysis. The function of determining whether a person in an image has specific characteristics that have been registered in advance The processing corresponding to the analysis results is based on the specific characteristics of the person in the image. Execute.

[0205] The system according to the third embodiment performs appropriate processing according to the characteristics of the person appearing in the image. It is possible to do so.

[0206] The processing unit is a facial recognition device that determines whether a person in an image has a specific characteristic. The processing unit performs face authentication of the person sitting in the driver's seat, and If a person does not match the pre-registered person, they will not be able to access the vehicle and start the engine. For example, the processing unit may communicate with the engine control unit via an in-vehicle network, such as a CAN. A signal is sent to the engine control unit (ECU) to prohibit the engine from starting. This allows the system to register in advance who can drive a vehicle and prohibit unregistered people from driving the vehicle. As a result, the crime prevention function of the vehicle can be improved. It is also effective as a measure against pranks.

[0207] According to a modification of the third embodiment, the processing unit controls the side brake to prevent the accelerator from being depressed. Prevents the parking brake from being released when the shift lever is in the drive position. Prevent entry, sound an alarm, send an email with a picture to the registered email address The device has the function of performing at least one of the following processes:

[0208] According to another modification of the third embodiment, the processing unit performs face authentication of a person outside the vehicle, If the person corresponds to a pre-registered person, the system has the function of controlling the vehicle and unlocking the doors. For example, if the vehicle is provided to a car-sharing service, The registered members of the parking service are registered in the processing unit in advance. By moving your face within the field of view of both cameras, you can easily see your face even if you do not have a card key. This will allow people to use the vehicles of car sharing services. It will be easier to use.

[0209] According to yet another modification of the third embodiment, the vehicle is a business vehicle in which unspecified persons ride. The processing unit has a function of determining whether a passenger is a wanted criminal by face recognition. The facial photos of wanted criminals are sent from the police database via LTE or wireless LAN. The facial photo is then transmitted to the processing unit of the system via the communication. The processing unit registers the received facial photo and If the processing unit determines that the passenger is a wanted criminal, it will suspend the vehicle's management. The communication device 21 (FIG. 3) and the communication network (FIG. 3) are used to notify the person in question, the police, etc. Furthermore, the processing unit has a function of notifying the driver in a manner that is not noticeable to the passengers. The vehicle manager, police, driver, etc. must be aware that a wanted criminal has entered the vehicle and must act promptly. You can take appropriate action quickly.

[0210] For example, in a taxi, the driver and passengers are the only people in the vehicle, so the driver's safety is a concern. There is a high demand for this system in terms of securing the driver. When the system detects a criminal in a taxi, it notifies the taxi company and the police. This means that the criminal is arrested when he or she enters the vehicle before actually committing a crime inside the vehicle. It is possible to detect criminals getting on board. This makes it possible to prevent crimes from occurring. .

[0211] According to yet another variation of the third embodiment, the processing unit detects whether a person in the image has a particular characteristic. The function to determine whether a person is a human is to compare the appearance of the person in the image with a specific appearance registered in advance. It has a function to compare the appearance and determine whether they match (hereinafter referred to as the appearance authentication function). As the fixed appearance, the appearance specific to a wanderer, the appearance specific to a criminal, etc. are registered in the processing unit in advance. Examples of criminal appearances include those carrying knives on the street and those carrying dangerous objects. The processing unit also uses images from infrared cameras, ultraviolet cameras, etc. By combining this with images from regular cameras, it is possible to detect people carrying illegal drugs. When the processing unit detects a person with a specific appearance in the image, the processing unit The police and the manager of the vehicle are notified via the communication network 40 (FIG. 3).

[0212] According to yet another modification of the third embodiment, the processing unit detects the behavior of a specific missing person, wanderer, etc. It has the function of registering the appearance of a person at the time of disappearance individually as search information. For example, The following items were included: height, type of clothing, color of clothing, whether or not the person is wearing a hat, whether or not they have a cane, whether or not they have a bent back, The processing unit has a function of registering individual search information, such as whether or not the person is wandering. The processing unit has the function of registering areas where people are believed to be present (search target areas). When the vehicle is located in the vicinity of the vehicle, the system executes a process to detect a person who matches the individual search information. Such individual search information is collected from databases (servers) of police and local support centers, etc., and In the present system, a processing unit of the system receives the person in the image having a specific characteristic. The processing unit of the in-vehicle device performs the function of determining whether or not the vehicle has entered the search area. 30 (FIG. 3) to the processing unit of the vehicle.

[0213] According to yet another variation of the third embodiment, the processing unit detects whether a person in the image has a particular characteristic. The function to determine whether a person sitting in the passenger seat or the back seat is fastened is to check whether the person is wearing a seat belt. This system has a function to determine whether or not a person is using a smartphone (hereinafter referred to as the appearance authentication function). When the processing unit detects a person who is not wearing a seat belt, the processing unit prompts the person to wear a seat belt. Issue a message.

[0214] In addition, the processing unit detects when a person in the back seat approaches an entrance to a highway without fastening a seat belt. When the driver is seated, the processing unit issues a message to encourage the driver to fasten his or her seat belt. When the distance from the current position of both vehicles to the entrance of the expressway falls below the reference distance, As another example, the processing unit may perform a check to see if the ETC card has been inserted. When the vehicle receives a signal for the expressway, it is determined that the vehicle is approaching the entrance to the expressway. The processing unit analyzes the image in front of the vehicle, and when it detects the entrance to the expressway in the image, It is determined that the vehicle is approaching the entrance to a road.

[0215] [Fourth Example] Next, a fourth embodiment will be described. The processing unit of the system according to the fourth embodiment performs image analysis. It has a function to count the number of people in an image, and processes the results of the analysis by The device performs a process of recording or transmitting headcount data.

[0216] The system according to the fourth embodiment can count people without using a special device. It becomes possible to count the number of people.

[0217] According to a modification of the fourth embodiment, a camera is attached to a bus, and the processing unit counts the number of people. The image of the inside of the vehicle is used as the original image. The processing unit displays the counting The results are displayed and notified to the driver. The driver then informs the passengers on board the vehicle. This function can be used in place of roll calls during tours. If the passenger count does not reach 0 when all passengers should disembark, the crew can detect if someone is left behind in the car, such as a baby. People who cannot express their will, people with disabilities who cannot get off by themselves, etc. It is possible to detect overstaying.

[0218] According to another variant of the fourth embodiment, the vehicle in which the camera is mounted has a predetermined number of passengers. The processing unit of this system is a vehicle of a type that is used for a vehicle, such as a tourist bus or a long-distance express bus. The processing unit has a function of registering the number of passengers in advance. The number of people on board is compared with the number of people on the flight, and if there is a shortage or surplus, a warning is given to the crew. This warning is issued by the speaker 15 (Fig. 2) or by a warning sound on the display 16 (Fig. 2). This is done by displaying a warning message. This prevents trains from departing with insufficient passengers. This prevents accidents caused by passengers being caught in the train. It also prevents passengers from being caught by passengers who are not authorized to ride. It is easy to identify and encourage passengers to disembark.

[0219] According to yet another modification of the fourth embodiment, a camera (e.g., a semi-transparent camera mounted on the ceiling) is installed inside a route bus. The processing unit performs image analysis to determine the number of passengers on board and the number of standing passengers. It also has the function of counting the number of passengers sitting on the bus. In addition, it also has the function of providing information on the bus's current location, the bus door, and It also processes the opening and closing signals, the bus alighting buzzer, the IC card touch when getting on and off the bus, and the fare box signal. The control unit acquires these events and the images at the time of their occurrence, using each event as a trigger. This information is stored in the data lake by the server 30 (Figure 3). This data lake will be used to determine where passengers boarded and disembarked, and the passengers’ addresses and telephone numbers during that time. It will be possible to analyze the seating situation using big data. This information can be valuable to consulting companies that handle urban planning, etc. The system combines information transmitted over in-vehicle networks such as CAN and vital information of the driver. This information is used to notify the driver when any abnormality occurs in the vehicle. This is useful for identifying causal relationships, such as whether the cause was the driver, the passengers, the driver himself, or the vehicle. It can be used.

[0220] According to yet another modification of the fourth embodiment, the processing unit is configured to: The processing unit of this system records the counting results as person-counting data. It has the function of easily extracting places with a large number of people based on the recorded number of people data. The processing unit records the image that is the basis of the counting, using the counting result as a trigger. For example, a function to record an image when the number of people exceeds a predetermined threshold. Have.

[0221] [Fifth Example] Next, a fifth embodiment will be described. The processing unit of the system according to the fifth embodiment performs image analysis. As a function of the system, the display contents of the vehicle's display device or the operating status of the vehicle's operating devices can be detected from images taken inside the vehicle. It has the function of detecting the situation and changes the display content or operation of the display device as a process corresponding to the analysis result. The device performs processing according to the operating status of the device.

[0222] The display device is connected to the device that acquires the data that is the basis for the display content, or to the operating status of the operating device. This system can obtain the data according to the needs of the user without the need to install any additional equipment. For example, these can be done without connecting any additional equipment to the OBD connector. In particular, you can connect a third-party product to the OBD connector to obtain data. Even if the vehicle manufacturer prohibits the collection of such data, this system can As vehicle display devices, various gauges, warning lights in the vehicle (warning lamps), etc. and head-up displays that project images onto the windshield, etc.

[0223] In a modification of the fifth embodiment, the processing unit of the system displays the speedometer as the display content of the display device. It has the function of detecting the displayed vehicle speed. The processing unit converts the position of the needle of the analog speedometer into a pattern. The processing unit recognizes the vehicle speed and obtains the vehicle speed data by character recognition of the numbers on the dial. has the ability to obtain vehicle speed data by character recognition of the numbers displayed on the digital speedometer. The acquired vehicle speed data can be used for various processes. An example of the configuration will be described.

[0224] The processing unit has a function to automatically detect the speedometer by image recognition. The processing unit is capable of detecting the speedometer in various vehicles, regardless of the vehicle type. In addition, the processing unit detects the position and shape of the speedometer of the vehicle in which the processing unit is installed in the image. This allows the processing unit to easily detect the speedometer. can be done.

[0225] A typical standalone car navigation device uses G to obtain the current location. It uses location information from PS. When GPS signals cannot be received, such as in tunnels, The vehicle's current position must be determined in another way. For example, immediately after the GPS signal is lost, The vehicle's current position on the map is moved at a constant speed of approximately 80% of the previous speed for five minutes. The mode in which such processing is performed is called tunnel mode. This is effective not only when entering a tunnel, but also when traveling under an elevated road. If you encounter a traffic jam while processing in tunnel mode, the car will drive at a constant speed. In the method assuming that the current position is correct, the error in the current position becomes excessively large and is practically unusable. It is not a good idea to install an acceleration sensor on a vehicle and use the acceleration data from the acceleration sensor. By using this, the accuracy of the current location information in tunnel mode can be improved. However, it is difficult and expensive for users to install an acceleration sensor, so it is not practical. There is none.

[0226] This system has the function of a standalone car navigation system. The processing unit is a speedometer The vehicle speed is read from the display, and map matching is performed based on this read speed information. This allows the vehicle's current location to be updated using the vehicle tracking function. This can improve the accuracy of location information.

[0227] The processing section is located in the area where the road in the tunnel and the ground road run parallel to each other, and in the area where the elevated road and the ground road under the elevated road run parallel to each other. In places where the two roads run side by side, the vehicle will decide which road to drive on based on whether tunnel mode is activated or not. For example, when tunnel mode is activated, The vehicle's position is mapped to the road inside the tunnel or the ground road under the elevated road, and the vehicle is in tunnel mode. When the system is not activated, the vehicle's position is calculated based on the road conditions on the ground or on the highway that runs parallel to the road in the tunnel. It has the function of mapping overpasses.

[0228] In another modification of the fifth embodiment, the processing unit detects the display content of the display device to be detected as follows: It has the function of detecting the remaining fuel level displayed on the fuel gauge. The processing unit performs image recognition of the fuel gauge. When the remaining amount of fuel falls below a predetermined reference value, the processing unit detects the remaining amount of fuel. When the system detects that the vehicle is about to refuel, it notifies the driver of the refueling request. By linking this with the vehicle information, when you return to your home or company vehicle storage area after going out, If the remaining fuel level falls below a certain threshold, for example 1 / 4 of a full tank, the processing unit issues a warning and advises the driver to refuel. In addition, the processing unit will guide the user to the destination by linking the car navigation device with this system. Compare the distance and fuel remaining and, based on the comparison, For example, when you are near a gas station along the route, the system will prompt you to refuel. The system outputs a message encouraging the driver to refuel at a gas station.

[0229] The function to extract the speedometer and fuel gauge from the image is a human-machined system installed in the processing unit of this system. This is realized by artificial intelligence (AI). The processing unit also pre-registers the positions of the speedometer and fuel gauge in the image. It has the function of registering the location and image information. It uses artificial intelligence to extract the speedometer and fuel gauge from the image. When it is difficult to extract the image, the processing unit uses the registered image information to extract the image from the Extract the speedometer and fuel gauge from

[0230] The processing unit reads the fuel gauge of the vehicle provided for the car sharing service and calculates the remaining fuel If it detects that the amount is below a certain threshold, the car-sharing service will refuel the vehicle. For example, the driver can receive a special benefit by filling up the tank with gas. You can receive a discount equivalent to 15 minutes. do.

[0231] In yet another modification of the fifth embodiment, the processing unit is The function to determine whether the warning light is on or off is included in the headlights. Light indicator, turn signal indicator, high beam indicator (high beam Warning lights, parking brake indicator, door ajar warning light, fuel warning light, etc. The processing means of the system determines whether or not these warning lights are on, and takes appropriate action in response to the warning lights. The corresponding vehicle state can be detected.

[0232] In yet another modification of the fifth embodiment, the processing unit is configured to display the information by a head-up display. The processing unit detects the warning information displayed by the head-up display. Detects warning content by capturing changes in the displayed image and executes a specified process according to the content of the warning For example, a warning can be issued by sound or voice. A warning by visual information can be converted to audio information. By issuing the warning before the vehicle approach, the effect is that the driver will be more likely to notice the warning.

[0233] In yet another modification of the fifth embodiment, the processing unit is configured to receive information on brightness outside the vehicle and a current position of the vehicle. In conjunction with the information, the headlights are not turned on even though it is a situation where they should be turned on. If the headlight indicator is not lit, the headlight will be turned on. The headlights should be turned on when, for example, the vehicle is When the outside is darker than a certain brightness, or the vehicle is currently inside a tunnel or culvert This includes cases where the headlights are turned on even though the situation in which they should be turned on has been resolved. When the headlight illumination indicator remains illuminated, the processing unit The function to notify the driver is as follows. Since there are tunnels with bright interiors due to lighting, If the vehicle determines whether or not it is necessary to turn on the headlights based only on the brightness information outside the vehicle, ,Even though the vehicle is in a tunnel, it is mistakenly determined that the headlights should not be turned on. The processing unit determines whether the vehicle is in a tunnel or not based on the current position information of the vehicle. By determining the above, it is possible to avoid such an erroneous determination. Although it was a situation in which not only the headlights but also the parking lights (small lights) should be turned on, Regardless of the type, if the vehicle width light indicator is not lit, the vehicle width light will be turned on. It has the function of notifying the driver to

[0234] In yet another modification of the fifth embodiment, the processing unit determines whether or not there is an oncoming vehicle from an image in front of the vehicle. and detects that the high beam indicator is on even though there is an oncoming vehicle. When the headlights are turned off, the system warns the driver to switch to low beam mode. Since the high / low switching operation is different, the driver may be required to operate a vehicle of a different model from the one he or she normally drives. When driving, there may be an incident where the high beam remains on even though the driver intended to switch to low beam. In such cases, the function that issues this warning is particularly useful.

[0235] In yet another modification of the fifth embodiment, the processing unit extracts a window from an image of the outside of the vehicle when it is dark outside the vehicle. It detects when the turn signal is flashing and displays the blinking indicators inside the car in sync with the blinking cycle of the turn signal. This function allows the driver to recognize the indicator as a turn signal indicator. The processing unit is configured to easily locate a turn signal indicator in an instrument panel. can be done.

[0236] In yet another modified example of the fifth embodiment, the processing unit determines from the image of the interior of the vehicle that no one is present in the vehicle. If the system detects that the headlights or parking lights are on despite the detection of a vehicle accident, the system will The system has a function to notify the driver or vehicle manager that the light has been forgotten to be turned off. For example, this is done by sending an email to a pre-registered email address. In addition, the processing unit notifies the driver by e-mail, etc., of the hazard lights and room lights that have been forgotten to be turned off. The hazard lamp lighting status is determined by the blinking status of the turn signal indicator. The lighting status of the room light is detected based on the brightness of the room light captured in the image. Detects.

[0237] The room light settings generally include a constant on mode, a constant off mode, and a door open / close mode. There are three types of interlocking modes (hereinafter simply referred to as interlocking modes). In most vehicles, These three modes are cycled through each time the room light on / off switch is pressed. When the engine is turned off in a dark place (at night), the interior light turns on and after a while Some vehicles have a function that automatically turns off the light after a few seconds. If you want to keep the interior lights on for a longer period of time to prepare for disembarking after you have For example, after the engine is turned off, the owner may organize and take out purchased items, or use a mobile phone. When taking out the food, etc., it is recommended to keep the room light on. Drivers can set the room light to not automatically turn off after a certain period of time has elapsed since the engine was turned off. To do this, switch the room light setting to linked mode or constant light mode. When the door is open, the current room light setting is either constant light mode or linked mode. Even though the current room light setting is always on, it is difficult to tell whether the light is on or off. However, if you get off the train thinking that it is in linked mode, the room light will remain on. By using this system, drivers and managers can By realizing that you forgot to turn off the room light and turning it off, you can prevent the room light from being left on for a long time. This prevents the battery from dying in vehicles that have a large battery. A structure where you don't know what mode the light is currently in, such as pressing a button Vehicles with an operating switch that is designed to cycle through the interior light settings each time it is pressed This feature of the system is particularly useful in

[0238] When the room light is set to linked mode, any door is open or ajar. If so, the room light will remain on. Notify that the room light is on Drivers and managers who have been given access to the vehicle may find that the door is ajar or that the door on the less visible side is open. You can realize that there is a possibility that

[0239] When the room light is set to linked mode, the light will turn off after a certain period of time (delay time) has elapsed since the door was closed. Some vehicles have a function that keeps the interior lights on until the vehicle is turned on. In such a vehicle, the processing unit of the present system detects that the door is closed and then starts the extinguishing process. It has a function that prevents the system from judging whether the light has been forgotten to be turned off or not until the light delay time has elapsed. The control unit has a function of setting the light-off delay time based on an input from the user.

[0240] In addition, the processing unit detects whether the room light is on even if a certain amount of time has passed since no one was in the vehicle. If it detects that the light is still on, it will send a signal to the vehicle controller to turn off the interior light. It has the function of transmitting a signal to

[0241] In yet another modification of the fifth embodiment, the processing unit is configured to detect headlights, sidelights, hazard lights, and Detects when the vehicle door is opened while the headlights, interior lights, etc. are on or flashing. and has the ability to sound an alarm to notify you if any of the lights remain on. This allows the driver to realize that they have forgotten to turn off the lights before they get out of the car, preventing them from forgetting to turn them off. It is possible.

[0242] In yet another modification of the fifth embodiment, the processing unit controls the lighting of a parking brake indicator. When the system detects that the parking brake is applied based on the light, it notifies the driver This function prevents the system from issuing an alarm even if it detects that the driver is dozing off from images. In addition, when the parking brake is applied, the driver may Even if the system detects that you are engaged in actions that would be considered dangerous while driving, such as looking at a smartphone, It has a function that does not issue an alarm.

[0243] In yet another variation of the fifth embodiment, the processing section is adapted to detect when early inspection or repair is required. If the warning light that notifies the driver that the vehicle is in a dangerous condition remains on for a certain period of time, It has the function of notifying the vehicle's dealer and registered repair shops of the vehicle's condition. For example, Users who are not interested in cars may not like to drive even if the warning light is on. In many cases, people do not notice the light and leave it unattended. For example, if the engine warning light comes on, Normally, the vehicle will become unable to be driven, but even if the engine oil warning light is on, In addition, if the O2 sensor is abnormal, the fuel economy will worsen, but the vehicle will continue to run. If the vehicle is still capable of being driven, continuing to drive it without inspection and repair may result in minor malfunctions. For dealers and repair shops, the warning light is a warning. This information will be useful for sales activities regarding inspection and repair to owners of vehicles with the light on. Vehicle owners can prevent breakdowns from becoming serious by having their vehicles inspected and serviced early. do.

[0244] In yet another modification of the fifth embodiment, the processing unit detects the temperature outside the vehicle by a temperature sensor. The temperature readings are displayed in the temperature display area in the instrument panel. The processing unit has the function of image analysis to detect the temperature of the instrument panel. The temperature display area of ​​the device can be read by character recognition from the image displayed on the device. The processing unit transmits the analysis result to an external device via a communication device and a communication network. A process for sending temperature information to the server is executed.

[0245] For example, sensors could be installed in the homes of ordinary users to obtain data on pollen dispersion. By collecting measurement data from sensors, it is possible to predict detailed pollen dispersion conditions and Weathernews Inc. and other companies provide services that provide information on dust dispersion. The outdoor photos posted by general users and outdoor live cameras installed in general homes are Images collected from these satellites are incorporated into weather forecasts. These weather information services (e.g. Weathernews SOLiVE24) are provided. The system that provides this information service collects outside air temperature information measured in many vehicles. This allows many vehicles to be used as temperature measurement probes. When many vehicles are distributed over a wide area, the temperature information collected from many vehicles is not uniform. For example, it can be used to provide spot information that cannot be provided by general weather information. It will be possible to issue heatstroke warnings, etc.

[0246] For companies providing such weather information services, installing temperature sensors in-house is This provides the effect of being able to collect temperature information over a wide range without having to The company that receives the temperature information will give a gift to the vehicle owner who provided the information. It is advisable to provide a fee-based weather information service free of charge or at a discount.

[0247] The processing unit further reads temperature information from a temperature display area in the instrument panel. Not only that, but the temperature was also displayed on roadside temperature signs captured in the images captured by the camera. The system has the function of acquiring outside temperature information by character recognition of the numbers displayed on the temperature display sign. The information is obtained by recognizing the words "Current Temperature" on the sign. The information is processed in the same way as temperature information obtained from the temperature display area in the instrument panel. The server then sends the data to an external server.

[0248] In yet another modified example of the fifth embodiment, the processing unit detects the operation status of the operation device by image analysis. The function to detect the position of the driver's feet on the accelerator pedal and brake pedal is The processing unit detects whether the driver's foot is The driver can easily recognize whether the accelerator pedal or the brake pedal is in position. For example, if the driver's foot is not on the accelerator pedal, By changing the color of the light emitted by the light-emitting element when the driver is in the brake pedal position, Therefore, the driver is notified.

[0249] In yet another modification of the fifth embodiment, the accelerator pedal or the brake pedal is pressed. This function detects whether the foot is on the pedal and displays the detection result as visual information on the display. This system is installed in a vehicle that has the function of displaying the information on the display. The system analyzes the images of the vehicle and detects the position of the driver's feet relative to the accelerator pedal and brake pedal. It has the ability to know.

[0250] In yet another modification of the fifth embodiment, the processing unit detects an obstacle ahead by image analysis. If the driver's foot is on the accelerator pedal even though the vehicle is in a stopped state, a warning will be issued by sound, etc. The driver will be able to recognize the warning and depress the accelerator pedal and brake pedal. The occurrence of errors can be reduced.

[0251] In yet another modified example of the fifth embodiment, the processing unit detects the operation status of the operation device by image analysis. It has a function to detect the rotation angle of the steering wheel. The processing unit processes the steering wheel rotation angle in response to the operation status of the operation device. and the vehicle speed, the processing unit executes a process of calculating a traveling direction and a traveling distance of the vehicle. The processing result is provided to the stand-alone car navigation device. The navigation device uses the information on the direction of travel and distance traveled provided by the system. Based on this, it is possible to calculate the vehicle's driving route even in situations where GPS signals cannot be received. can.

[0252] In still another modification of the fifth embodiment, the processing unit is configured to perform the image analysis by detecting when the vehicle is moving Detects whether the vehicle is in automatic driving mode (including automatic driving levels 1 to 5) or manual driving mode (automatic driving level 0) This detection is performed by the processor which automatically detects the vehicle status displayed on the instrument panel. Recognizing from an image the display state of an indicator showing whether various driving functions are in operation The processing unit performs the process corresponding to the analysis result by determining that the automatic driving function has been deactivated. When the system detects this, it executes a process to notify the driver that the autonomous driving function has been deactivated. Even though the automatic driving function is deactivated, the driver has their hands on the steering wheel. When the processing unit detects that the driver is not in a driving position, the processing unit instructs the driver to operate the steering wheel. In addition, the system issues a warning to alert the driver that the autonomous driving function has been deactivated and that the driver is in standby mode. The processing unit detects the steering wheel not being touched and detects the The recorded images are recorded as event records separately from the regular records. , can be used as evidence that the driver's operation was inappropriate.

[0253] The processing unit detects when the autonomous driving function is enabled in a vehicle equipped with a radar detector. The radar detector function can be disabled. This prevents radar detectors from This can eliminate unnecessary alarms. When the automatic operation state is disabled, the processing The part enables the radar detector function.

[0254] Whether the vehicle is autonomous or manually driven, and what level (mode) of autonomous driving is in use ) is determined by the processing unit detecting the movement of indicators, steering wheel, and driver's hand position. The position of the vehicle and the direction in which the driver's head is facing are used to determine this.

[0255] [Sixth Example] Next, a sixth embodiment will be described. The processing unit of the system according to the sixth embodiment performs image analysis. The function of this system is to detect the characters displayed on the stickers attached to the vehicle from the images of the inside of the vehicle. It has the function of recognizing characters, and as a process corresponding to the analysis results, it responds to the information indicated by the recognized characters. If necessary, a warning process will be performed.

[0256] By being alerted, the driver, manager or owner of the vehicle is required to The robot will then pay attention to the text information on the sticker and perform appropriate processing according to the text information. can.

[0257] In a modified example of the sixth embodiment, the processing unit is a vehicle inspection sticker. The inspection mark on the inside of the vehicle has the function of recognizing the characters that are in the vehicle inspection validity period. The date is displayed. The processing unit performs character recognition on the information on the expiration date of the vehicle inspection validity period. The period from the present time to the expiration date of the validity period of the vehicle inspection (the deadline for the vehicle inspection) is If the period is shorter than the period, the driver is notified to have the vehicle inspected. The system will warn users by playing a voice message saying, "The expiration date of the validity period is approaching." If the period until the expiration date of the validity period of the vehicle inspection becomes shorter, for example, the processing unit may The device will issue a voice warning saying, "Please be careful."

[0258] In another variation of the sixth embodiment, the processing unit is a maintenance inspection sticker. The maintenance sticker will show the deadline for the next regular inspection. The processing unit recognizes the date information indicating the deadline for periodic inspection by character recognition. If the period from the present time to the due date for regular inspection becomes shorter than the specified period, For example, the processing unit may send a message saying, "The deadline for your regular inspection is approaching. The system will issue a voice message saying, "The deadline for the regular inspection is approaching." If the time becomes shorter, the processing unit will issue a warning, for example, saying "Please have your periodic inspection done." will be issued.

[0259] In yet another modification of the sixth embodiment, the processing unit performs the image analysis function by It has the function of judging whether the inspection mark is present in the image captured by the camera. In the case where the inspection mark is not shown in the image, the processing unit performs the process of detecting the inspection mark. The system executes a process to call attention to the necessity of adjusting the mounting position and attitude of the camera so that the For example, the processing section may say, "The inspection mark is not visible. We will change the camera position so that the inspection mark is visible. Please adjust your posture. The processing unit can easily adjust the position and attitude of the display device in the vehicle. For example, it has the function of reading the contents displayed on the speedometer, fuel gauge, etc., and the display devices in the vehicle are It has the function of drawing attention by being reflected in the image.

[0260] In yet another modification of the sixth embodiment, the processing unit performs the following as a function of image analysis: There are no stickers other than those permitted by law on the windshield, driver's side window, or passenger's side window. It has a function to determine whether a sticker is attached or not. If the processing unit determines that a sticker other than that permitted by law is attached, In this case, a process is executed to notify the driver or the like to remove the sticker.

[0261] [Seventh Example] Next, a seventh embodiment will be described. The processing unit of the system according to the seventh embodiment performs image analysis. The function of this function is to analyze the image of the area in which the passenger seat is reflected, and respond to the analysis results. As a process, a process according to the situation of the passenger seat is executed.

[0262] For example, depending on whether there is a person in the passenger seat or whether there is an object on the passenger seat, , appropriate processing can be performed.

[0263] In a modified example of the seventh embodiment, the drive recorder 1 (FIG. 1) is installed in a vehicle seated in the passenger seat. The device will be installed in vehicles that have a function to issue a warning when a person is not wearing a seat belt. The image analysis function of the Department of Science and Technology is to determine whether a person is sitting in the passenger seat from the image of the area that includes the passenger seat. As a process corresponding to the analysis result, the processing unit determines whether or not a passenger is seated in the passenger seat. When it detects that no one is sitting in the passenger seat, it issues a warning that the passenger seat belt is not fastened. The processing unit further disables the detection of an object placed on the passenger seat as a function of image analysis. As a process corresponding to the analysis result, the processing unit judges whether or not there is an object on the passenger seat. When this is done, the function that issues the seat belt warning for the passenger seat is disabled. Execute the process.

[0264] The vehicle incorrectly determines that a person is in the passenger seat when an object equivalent to a human's weight is placed in the passenger seat. If this system is installed in a vehicle, a warning may be issued if the seat belt is not fastened. When this is selected, the vehicle will issue a seat belt warning if there is an object on the passenger seat. This will prevent the driver from being confused by the warning. This function is available for vehicles that have a warning buzzer that continues to sound when a heavy object is placed on the vehicle. The function is effective. To prevent the vehicle from issuing a seat belt warning, The processing unit then issues a warning to the vehicle controller through the in-vehicle network (CAN). Send a signal to disable the

[0265] In a vehicle having a similar function for the rear seats, the processing unit detects whether an object is placed on the rear seats. When it is detected that the seat belt is not fastened, the processing means cancels the seat belt unfastened warning function.

[0266] [Eighth Example] Next, an eighth embodiment will be described. The processing unit of the system according to the eighth embodiment performs image analysis. The function of the system is to compare the current image of the interior of the car with the past image of the interior of the car and perform analysis. As a process corresponding to the result, the degree of discrepancy between the current image of the interior of the car and the past image of the interior of the car is calculated. If the value exceeds a predetermined level, a notification process is executed.

[0267] The vehicle owner or manager will be notified when the vehicle interior conditions exceed a certain level. You can see what has changed.

[0268] In a modified example of the eighth embodiment, the processing unit stores a vehicle interior image in advance, which is attached to the vehicle, as a past image of the interior of the vehicle. An image of the vehicle with nothing in it other than the items being checked was used, and this image was compared with the current By comparing the image of the inside of the vehicle with the image of the inside of the vehicle, it is determined whether anything has been brought into the vehicle. Has function.

[0269] The processing unit further compares the image at the current time with images at multiple past times. The processing unit has a function of calculating the time that an object brought into the vehicle remains in the vehicle. The system will notify you if it detects that something has been left in the car for a long time. To have Noh.

[0270] Next, a specific process of the processing unit will be described. First, in a state where no luggage is loaded on the vehicle, If possible, press the initial setting button of this system with only the driver in the car. The processing unit detects the part of the image where the optical flow corresponds to the vehicle speed. The part that moves is recognized as a window. The other part that moves is recognized as a driver. Any area other than the areas you have identified is considered an area where objects may be placed, and if there is no luggage in the vehicle, The process up to this point is the initial setting process. is completed.

[0271] After that, when the engine is started or stopped, the processing unit compares the image of the no-cargo state with the current image. The system detects the presence of an object brought into the vehicle by taking the difference between the image and the actual image. If the position of the object does not change for a certain period of time, the processing unit issues a warning. This function allows the driver to recognize objects that have been left in the vehicle for a long period of time. This is especially useful for people whose cars quickly become filled with trash.

[0272] [Ninth Example] Next, a ninth embodiment will be described. The processing unit of the system according to the ninth embodiment performs image analysis. The function of the system is to determine whether a pre-registered object is captured in the image of the inside of the vehicle. As a process corresponding to the analysis results, an alarm is issued when a pre-registered object is captured in an image of the inside of the vehicle. The system executes a process to issue a notification.

[0273] The driver, etc., can be made aware of the presence of a pre-registered object in the vehicle by the alarm. Cut.

[0274] In a modification of the ninth embodiment, the processing unit is configured to check for items that are left behind (a specific bag The processing unit has the function of registering images of personal items (bags, wallets, etc.) in the vehicle. When a registered object is detected in the image, an alarm will be issued or the object will be forgotten. For example, an alarm may be issued by sounding a buzzer installed in the vehicle. As an example of notification of the defect, for example, electronic mail may be sent wirelessly to the driver's smartphone or mobile phone. Notification will be given via email or other means.

[0275] In another modification of the ninth embodiment, the processing unit detects an item left behind in the vehicle and opens the door. When it detects any movement toward the vehicle, it will sound an alarm, and the driver or other person in charge will be warned to leave the vehicle. The processing unit further detects an item left behind in the vehicle, and When the system detects the movement of closing the door, it notifies the driver by email or other means that an item has been left behind. The function allows drivers to realize that they have left something behind after they have left the vehicle. The movement of opening and closing the door is detected, for example, by detecting optical flow.

[0276] In yet another modification of the ninth embodiment, the processing unit detects optical flow. For example, the processing unit may process the image data to detect other movements in the vehicle. Analysis functions include door opening and closing movements, window opening and closing movements, turn signal operation, steering The system has the function of detecting wheel operation, raising and lowering of sun visors, etc. This can be done without connecting to the vehicle's in-car network connector (e.g. OBD connector). You can obtain information about them.

[0277] When the sun visors are down, it is very dangerous to use a 360-degree camera mounted on the windshield. When capturing an image of the driver's side, the sun visor gets in the way and it is impossible to capture an image in that direction. Therefore, when the brightness of the front of the vehicle is such that sun visors are not necessary, However, if the processing unit detects that the sun visor is down, it issues a warning. Whether the sun visor is down or not is determined by the processing unit based on the ratio of the sun visor area to the image. is determined.

[0278] [Tenth Example] Next, a tenth embodiment will be described. The processing unit of the system according to the tenth embodiment processes an image The analysis function is to capture an object that the user brings close to the camera and recognize the image. As a processing method for the analysis results, processing according to the information written on the object that the user brings close to the camera is performed. Execute.

[0279] By bringing an object close to the camera, the user can receive information written on the object. The corresponding processing can be executed.

[0280] In a modification of the tenth embodiment, the processing unit is provided with a function for performing image recognition of the driver's license of the driver. The processing unit recognizes the character of the validity date written on the driver's license and calculates the validity date from the current date. When the period until the validity date is shorter than the specified period, the renewal date of the driver's license is approaching Inform the user.

[0281] In another variation of the tenth embodiment, the processing unit receives an image of a gas station receipt. The processing unit recognizes the date, amount of fuel, and amount shown on the receipt. This information is then recorded and put into a database.

[0282] In yet another variation of the tenth embodiment, the processing unit is configured to receive an identification card (e.g., a driver's license) The processing unit has the function of performing image recognition of the ID card (such as ID card, employee ID card, resident registration card, etc.). The processing unit recognizes the name and other information written on the person's name and determines whether the person is a person who has been registered in advance. If the person on the certificate is not registered, the engine will be prohibited from starting. This makes it possible to limit the people who can drive a vehicle to those who are registered in advance. In addition, the processing unit records the name of the person on the identification card. It can be used to identify drivers when managing vehicle operations.

[0283] In yet another modification of the tenth embodiment, the processing unit has a function of image recognition, The function to recognize objects that the user brings close to the camera outdoors, and the function to recognize objects that the user brings close to the camera inside the car For example, the processing unit compares the image with a normal image and When something not present in a normal image occupies a larger area than the camera's field of view, In particular, the processing unit detects an object approaching the camera within a predetermined viewing angle of the camera. If the user recognizes that they are holding something that occupies a larger area than the range, the system will It is determined that the user has approached the camera.

[0284] [Eleventh Example] Next, an eleventh embodiment will be described. The processing unit of the system according to the eleventh embodiment processes an image The analysis function is to perform image analysis of objects captured in the forward image, and the analysis results are As a corresponding process, a process is executed to provide the driver with information that should be noted regarding vehicle driving. do.

[0285] By alerting the driver to the situation ahead, the system can improve visibility as the vehicle moves forward. This can help prevent accidents that may occur in the future.

[0286] In a modification of the eleventh embodiment, the processing unit determines a distance between the vehicle and the vehicle ahead as a function of image analysis. The processing unit has a function of calculating at least one of the following distance and the relative speed. Based on relative speed, the system detects high collision risk situations and warns of high collision risk. The distance between vehicles is calculated based on the size of the vehicle ahead in the image, e.g. Information such as the size of the license plate, the size of the tread (the distance between the tires), and the width of the taillights The relative speed is calculated from the rate of change in the distance between the vehicles.

[0287] In another modification of the eleventh embodiment, the processing unit may include a function of image analysis, such as a front signal The processing unit detects the color of the traffic light and warns the driver to stop when the traffic light is red. Notify to prompt.

[0288] In still another modification of the eleventh embodiment, the processing unit performs the following as a function of image analysis: The processing unit has the function of recognizing road signs. The processing unit gives the driver attention according to the information indicated by the road signs. For example, if a road sign indicates a speed limit, the processing unit recognizes the The system obtains the speed limit at the current location and notifies the driver of the speed limit. If the vehicle is about to enter a prohibited road, the processing unit issues an alarm. The processing unit notifies the driver when the vehicle is traveling in violation of the traffic rules indicated by the road signs. If it detects that you are trying to drive in a way that violates traffic rules, Notify the driver of the violation. This prevents the violation of traffic rules from occurring or The processing unit recognizes the character strings of the road signs to determine the current driving If a sign indicating that the section is equipped with an automatic speed camera is detected, the driver is notified of the speed limit. This will notify you to be careful not to use too much.

[0289] In still another modification of the eleventh embodiment, the processing unit performs the image analysis function It has the function of detecting congestion in lanes adjacent to the driving lane. The processing unit determines based on the average traveling speed of the vehicles in the adjacent lanes and the distance between the vehicles. When the speed of the vehicle in the adjacent lane is below a specified standard value and the distance between the vehicles is below a specified standard value, When a predetermined number of vehicles are lined up, the processing unit determines that a traffic jam is occurring. The processing section responds to the analysis results by detecting congestion in lanes adjacent to the vehicle's lane. When it detects that a vehicle is about to suddenly jump out from between vehicles in a traffic jam, it warns the driver to be careful. In the lanes adjacent to the lane in which the vehicle is traveling, there are oncoming lanes and lanes in the same lane as the vehicle. This includes adjacent lanes traveling in the same direction. In places where vehicles gather, such as gas stations, and where vehicles enter and exit, There is a high possibility that the adjacent lanes in the same direction will also be congested. In such a case, It is effective to detect congestion in the adjacent lane. The processing unit detects whether the traveling speed of the vehicle is less than a predetermined speed, for example, 30 kph. When the vehicle speed is less than m, the system will not emit the warning sound for warning of sudden sudden movements of vehicles. In addition, the processing unit can detect the obstacles in the forward image. It has the function of detecting situations where there is a high risk of slipping out and issuing a warning. If there are adjacent parked vehicles at the same time, you may park in a parking lot equipped with a parking meter. When there are cars around, etc., warn people not to rush out in front of you.

[0290] For example, car navigation devices can provide traffic congestion probe information, general road and highway driving information, and surrounding facility information. The system has a function to alert drivers to driving caution based on information about residential areas, schools, etc. By using this system, it is possible to Even if the vehicle is in a traffic jam, a warning can be issued.

[0291] In still another modification of the eleventh embodiment, the processing unit performs the following as a function of image analysis: It has the function of detecting animals (dogs, cats, raccoons, wild boars, etc.) and people around the vehicle. When the processing unit detects an animal or person around the vehicle, it notifies the driver that there is an animal or person around the vehicle. This will help prevent accidents from occurring immediately after starting off.

[0292] In yet another modification of the eleventh embodiment, the processing unit performs the following as a function of image analysis: The processing unit detects that the vehicle is moving forward slowly based on the flow of the scenery. If it detects that the vehicle is moving forward slowly, it will issue an alarm. For example, if the vehicle stops After that, the vehicle starts to move little by little due to the creep phenomenon caused by the slackening of the pressure on the brake pedal. When the vehicle is moving, it will inform the driver that there is a risk of a rear-end collision. The processing unit detects when the vehicle is moving at a speed that causes the vehicle to move forward due to a creep phenomenon. The processing unit determines the speed of the flow of the surrounding scenery based on the optical This is done by detecting the physical flow.

[0293] In yet another modification of the eleventh embodiment, the processing unit performs a front run as a function of image analysis. A machine that determines whether a vehicle is a large vehicle, such as a bus or an aluminum box truck. When the processing unit detects that the vehicle ahead is a large vehicle approaching a traffic light, Notify the driver of the presence of a traffic light. For example, issue a voice message saying "Watch out for the traffic light." The presence or absence of a vehicle is registered in advance as POI information in the processing unit. For example, When driving, it becomes difficult to see traffic lights in the direction you are heading. Look at the traffic lights, predict the presence of traffic lights, and keep a sufficient distance to avoid the traffic lights in the direction of travel. However, if such care is neglected, progress will be hindered. There is a risk of missing the traffic lights in the direction you are heading. Also, in rural areas, it is difficult to see the traffic lights in the opposite direction. In some cases, pedestrian signals are not installed. If the vehicle ahead is a large vehicle, By notifying drivers of the presence of a traffic signal, the risk of drivers overlooking the traffic signal can be reduced. In addition to the function of directly informing the driver of the presence of a traffic signal, the processing unit also For example, the processing unit may be configured to: Please keep a sufficient distance between the vehicles. The system issues a warning when the distance to the intersection is longer than the vehicle's safe stopping distance. The safe stopping distance varies depending on the vehicle speed and weather (rain or not). The distance is set appropriately based on the vehicle speed and the weather (rain or not). For example, If the safe stopping distance in fine weather is 50m, the processing unit determines that the current weather is fine and the large horse in front is The distance to the vehicle is less than 50m and the vehicle is outside what is called an urban area in car navigation devices. If there is an intersection with traffic lights within 100 m in the direction of travel, a warning will be issued.

[0294] In still another modification of the eleventh embodiment, the processing unit performs the following as a function of image analysis: Determine whether the road is sloped, and if so, detect the degree of slope. The processing unit detects whether the vehicle is going too fast or not depending on the slope of the hill. It warns of a drop in power output and setting the vehicle to hill climb mode.

[0295] In yet another variation of the eleventh embodiment, the vehicle is a vehicle for handling luggage, e.g. The processing unit detects the text displayed on the package from the captured image. The function of recognizing signs such as character strings and bar codes and determining whether the cargo is one that the vehicle should handle. The information indicated by the tag attached to the baggage to be handled by the vehicle is registered in advance in the processing unit. The processing unit detects that the vehicle is attempting to load a load that is not intended for the vehicle to handle. When it detects this, it executes a process to issue an alarm.

[0296] In yet another modified example of the eleventh embodiment, the processing unit calculates an image from the optical flow. The processing unit detects the area of ​​the window in the image. Whether the vehicle has entered the tunnel, whether the vehicle has exited the tunnel, whether the vehicle is traveling in the tunnel In addition, the processing unit determines whether the vehicle is under the overpass ( The processing unit has a function to determine whether the vehicle is traveling on a parallel road. It has the function of determining whether or not there are guardrails and whether or not attention needs to be paid to gutters based on the flow of traffic.

[0297] [Twelfth Example] Next, a twelfth embodiment will be described. The processing unit of the system according to the twelfth embodiment processes an image As a function of the analysis, the characters written on the license plates of other vehicles captured in the image outside the vehicle can be read. The character recognition function recognizes the characters that are identified by the analysis results. At least one of the processes of recording the member plate information and transmitting it via a network Also execute the other one.

[0298] Whether the vehicle is traveling near a vehicle equipped with an on-board device that has the processing function of this system, or can identify license plate information of other vehicles parked nearby. The department will collect the current location information of the vehicle equipped with this onboard device and the date and time when the license plate was captured. The location information and date and time are transmitted in association with the identified license plate information. The license plate information obtained along with the information is the vehicle number plate installed on the road. - Along with the license plate information obtained by the automatic reading device (N system), This is useful information for knowing the route of many vehicles. By linking this with S information, we will build a system that can be called a wide-area mobile N system. can be done.

[0299] In a modified example of the twelfth embodiment, the network is transmitted from the present system installed in each of a plurality of vehicles. Through the network, location information, date and time information, and license plate information are collected and associated. This center system will collect specific license plate information. It has the function of extracting the location information and date and time information of the vehicle and identifying the route that the vehicle took.

[0300] [Thirteenth Example] Next, a thirteenth embodiment will be described. The processing unit of the system according to the thirteenth embodiment processes an image The analysis function has the function of determining whether or not there is a window glass edge on the vehicle window. In response to the analysis results, if the edge of the window glass is detected in the window area, a notification process is executed. To carry out.

[0301] The presence of window glass edges in the window area means that the window glass has been left open. By receiving notifications from this system, you can realize that you have forgotten to close the windows. The processing unit detects discontinuities in the image caused by the difference in refractive index between the window glass and the atmosphere, The processing unit detects the edge of the window glass. As a notification process, the processing unit wirelessly notifies the driver of the notification by a mobile terminal or the like. The processing unit also processes notifications by phone, email, etc. The notification is processed by sounding a buzzer installed in the vehicle. When the window film is applied, the processing unit detects the area where the window is open and the window glass strip. It detects whether a window is open based on the difference in image brightness from the smoke area.

[0302] [14th Example] Next, a fourteenth embodiment will be described. The processing unit of the system according to the fourteenth embodiment processes an image The analysis could include whether the wipers are working or not, or whether there are more or less raindrops on the windshield. It has the function of detecting the presence or absence of rain and determining whether it is raining or not, and processes the analysis results accordingly. When it is determined that it is raining, the device transmits information to notify the user that it is raining. The process of sending the data from the device is executed.

[0303] By receiving information transmitted from this system, the vehicle can detect whether there is rain in the area where the vehicle is traveling. The processing unit can detect whether it is raining or not by transmitting the information to an external device such as a server. The external device has the function of notifying the vehicle's current location information as well as the external device. By collecting information from vehicles, it is possible to create a map of where it is raining. Furthermore, the processing unit notifies the user that it is raining and transmits the current time information. This allows external devices to receive information about changes in the area where it is raining over time. You can obtain information related to the

[0304] In a modification of the fourteenth embodiment, the external device is a server of an organization that provides weather information. The processing unit detects when the wiper is not in the image, e.g. when the wiper is down. The server has the function of uploading the image at the time of upload. The server has the function of acquiring information about rainfall, such as the amount of rainfall, from the captured images. The rainfall and other data collected from both will be used as big data in various application fields. For example, the server is used to predict localized torrential rains. If there are raindrops on the glass but the number of them does not increase or decrease, it is determined that the rain has stopped. When the processing unit determines that the rain has stopped, it notifies the server that the rain has stopped. can obtain more detailed information about rainfall.

[0305] [Fifteenth Example] Next, a fifteenth embodiment will be described. The processing unit of the system according to the fifteenth embodiment processes an image As an analysis function, it has the function of judging the weather from the sky conditions captured in the image, and the analysis results As a corresponding process, a process of transmitting weather information from the communication device is executed.

[0306] Weather information collected from multiple vehicles is used as big data on meteorological information. The server that collects this information can then use it to calculate the color of the sky, clouds, etc. The processing unit determines whether the weather is clear, sunny, cloudy, etc. based on the presence or absence of the sun's height. Also, the strength of the morning sun or afternoon sun is detected from the brightness of the sky, and information that the morning sun or afternoon sun is strong is transmitted. The processing unit transmits the current location of the vehicle along with information about the weather. The server that collected this information then sends the information about the strong morning and afternoon sun, as well as the current location of the vehicle. The function of providing this information to drivers of other vehicles heading to that location is to Drivers who receive this training will be aware that they need to be careful of morning or evening sun.

[0307] [16th Example] Next, a sixteenth embodiment will be described. A processing unit of the system according to the sixteenth embodiment It has a function to detect sound information from a microphone installed in the camera, and the detection result of the sound information and the image analysis are The analysis function correlates the result of the analysis with the result of the analysis, and executes processing according to the result of the correlation.

[0308] Compared to processing based only on the results of sound detection or image analysis, By associating the two, more appropriate processing can be performed.

[0309] In a modification of the sixteenth embodiment, the processing unit is provided with a function of image analysis. It has the function of detecting changes in the state of the display of the device with a display, and takes action according to the analysis results. The principle is that when the detected sound information and the change in the display state are synchronized, the detected sound The information is stored in association with the device whose display state has changed.

[0310] In another modification of the sixteenth embodiment, when the processing unit detects sound information from the microphone, Sound is generated based on information stored in association with changes in the state of the display unit. For example, the processing unit has a voice recognition function and can recognize Determine whether the sound information is generated from a specific device, and If the voice recognition function is canceled, unnecessary voice recognition processing is eliminated. In addition, the processing unit can avoid the execution of the process by determining whether the sound information from the microphone is coming from a specific device. If it is determined that the sound was emitted from another device, the area of ​​the device that emitted the sound in the image inside the vehicle is For example, the device that generated the sound can record the information that can be distinguished from the other areas. An image of a rectangle or a closed curve of any shape enclosing the captured area is recorded by overlaying it on the original image. Or, a semi-transparent geometric pattern is superimposed on the area where the device that produced the sound is recorded. .

[0311] In yet another modification of the sixteenth embodiment, the processing unit performs the image analysis function by A function to determine whether an emergency vehicle, such as an ambulance, fire engine, or police car, is included in the image. The processing unit detects the sound of an emergency vehicle siren from a microphone and detects an emergency vehicle in an image. When the vehicle is detected, the processing unit associates the two and determines that an emergency vehicle is in the vicinity of the vehicle. If it determines that an emergency vehicle is present, it will send a message to the driver to warn them.

[0312] [17th Example] Next, a seventeenth embodiment will be described. In the seventeenth embodiment, a camera installed in a vehicle The processing unit detects the positions of other cameras captured in the image as a function of image analysis. It has the function of detecting the positional relationship between multiple cameras as a process corresponding to the analysis results. Perform the process.

[0313] It is possible to capture images of a wide area inside the vehicle that would be difficult to capture with a single camera. Once the relative positions of the cameras are identified, the relative positions of the images captured by each camera can be determined. Cut.

[0314] According to a modification of the seventeenth embodiment, the processing unit detects the models of other cameras that appear in the image. The processing unit further has a function to identify the image captured by the multiple cameras and to identify the angle of view from the model. By combining images, the individual images from the cameras can be displayed. Compared to using a handheld device, this allows users to intuitively grasp the situation inside the vehicle.

[0315] According to another modification of the seventeenth embodiment, at least a part of the functions of the processing unit is implemented in a visual display such as a personal computer. The processing unit displays the camera position in the image to be displayed. The camera's location in the vehicle is displayed according to the location of the camera (own) that is capturing the image. It is difficult to identify the target from the image captured by the camera. The processing unit detects the position of the camera in the vehicle from the image of the camera. When the images captured by the cameras are composited and displayed, the positions of the three cameras are displayed in the composite image. In addition, icons can be displayed according to the camera model or angle of view. do.

[0316] According to yet another modification of the seventeenth embodiment, the processing unit displays the images captured by each camera. The position of the camera and the angle of view of the camera (the angle of view of the camera) are also displayed in the image. For example, a 360-degree camera with a full 360-degree view and a half 360-degree view have the function of displaying the area in which the view is captured. Depending on the camera's function (wide-angle camera, etc.), the angle of view of the camera is displayed in the image. The user can intuitively grasp which area each camera is capturing.

[0317] [18th Example] Next, an 18th embodiment will be described. In the 18th embodiment, a camera mounted on a vehicle The 360-degree camera has an image circle smaller than the diagonal length of the imaging surface. The image processing unit generates a basic pattern for calculating the image feature amount as an image analysis function. It has the function of deforming depending on the position within the circle.

[0318] The processing unit uses the basic pattern without correcting the image distortion in the image circle. This allows the load on the processing unit to be reduced. The basic pattern is an image captured using a standard lens (a general wide-angle lens). The projection method to obtain the image circle is, for example, orthogonal projection, equidistant projection, etc. There are various projection methods such as shadow, stereographic projection, and equi-solid angle projection. The processing unit , we transform the basic pattern. As a camera lens, we use a circle to get the image circle. The camera uses a fisheye lens. The imaging surface of the camera is rectangular, and the diameter of the image circle is long. shorter than the side and equal to or shorter than the short side.

[0319] As a basic pattern, a template pattern of the image to be detected is registered in advance in the processing unit. The processing unit deforms the template pattern according to the position in the image circle. In addition, the processing unit may perform a function of recognizing an image using, for example, a Harlike feature. Each of the multiple rectangular patterns for obtaining the Har-like feature is placed within the image circle. It has the function of transforming depending on the position.

[0320] In a modified example of the eighteenth embodiment, the processing unit performs image recognition using an image search function. It has the function of limiting the area within the camera that contains a specific object to be detected. This function is realized by the drive recorder. Generally, the recommended installation position of the drive recorder is predetermined, so the image circle The area where the driver is reflected, the area where the traffic lights are reflected, the area where the person in the back seat is reflected, etc. Therefore, depending on the person or object to be detected, the method of detecting the person or object can be changed. In order to detect the presence of a person or object, the area where image analysis is performed is limited to the area where the person or object is captured. When analyzing the image of the front of both vehicles, the processing unit performs the following processing for the area in which the windshield is reflected: Perform analysis.

[0321] In another variation of the eighteenth embodiment, the processing unit averages a specific area of ​​the image circle. The plane image is converted into a parallel projection image, and the normal basic pattern is used for the converted plane image. Matching is performed using this method.

[0322] In yet another modified example of the eighteenth embodiment, the processing unit processes the image in the image circle as A part of the image mapped onto the sphere is mapped onto the sphere using a predetermined transformation function. Zoom in and ignore distortion to perform image analysis as a flat image. When a specific object is to be detected, the system zooms in on the area in which the object is captured and performs image analysis. When detecting a road sign, the system zooms in on the area of ​​the image ahead that contains the road sign and performs image analysis. conduct. [19th Example] Next, a 19th embodiment will be described. In the 19th embodiment, the processing unit has a function of image analysis. From the viewpoint of privacy protection, areas of an image that should not be made public are marked as privacy. It has a function to identify the protected area and performs privacy protection as a process corresponding to the analysis result. The camera captures an image in which people in the area cannot be identified by appearance. Then, a process is performed to generate the image from the image.

[0323] By making it impossible to identify a person, the privacy of that person is protected. The processing unit has a function to distinguish between areas in which the interior of the vehicle is captured and areas that should not be made public. In particular, when capturing images using a spherical camera, the interior of the vehicle is captured. In some cases, you may not want to keep the image of the vehicle. In such a case, the processing unit may black out the area inside the vehicle. or the ability to blur the interior area and record the exterior image as is. The processing unit performs image analysis to distinguish between areas showing the inside of the vehicle and areas showing the outside of the vehicle. For example, the processing unit may detect the optoelectronics at a speed corresponding to the vehicle speed. The processing unit has a function to determine that the area where the traffic flow occurs is outside the vehicle. It has a function that allows you to set the sea protection area in advance.

[0324] In a modification of the 19th embodiment, the processing unit identifies a person within the privacy protection area. The normal mode records images that can be used to capture people in the privacy protection area, and A privacy mode that records images that do not allow objects to be identified by their appearance, The user can select the mode of communication depending on the purpose and situation of the vehicle. You can switch between normal and privacy modes.

[0325] According to another modification of the nineteenth embodiment, the processing unit processes the area in which a person is photographed as a privacy The processing unit detects people by performing image analysis and The processing unit has a function of determining an area surrounded by a contour of a person as a privacy area. The function of the camera is to set only the face of the detected person as a privacy area. The processing unit has a function of displaying a person as a silhouette. Alternatively, the processing unit displays only the outline of the person. The function is to display the image when there is no person inside the outline. The processing unit has the function of displaying the edges of moving objects as the outline of a person. Have.

[0326] In yet another modified example of the 19th embodiment, the processing unit adds a It has the function of displaying an avatar instead of the original person. The avatar displayed will be similar to the original person. For example, an avatar that reflects characteristics such as gender, body shape, and facial shape is displayed. The person and the avatar are associated in advance, and the processing unit identifies the person appearing in the image, The system has the function of displaying the avatar associated with the identified person. The avatar is The image is created for each person in advance and registered in the processing unit. The processing unit For example, when a person raises their hand, the avatar's hand is raised. When the person turns to the side, the avatar also turns to the side. For example, the processing unit detects whether a person is smiling, When it detects a sad or angry facial expression, it changes the avatar's facial expression to a smiling or The processing section has the function of changing the facial expression of a person into a happy, sad, or angry expression. It has the ability to read emotions and change the color of the avatar according to the person's emotions. When it detects that something is angry, it will change the color of the avatar to a color that represents anger, such as red. Change to.

[0327] In yet another modification of the 19th embodiment, the processing unit may include a step of detecting the image captured by the camera before recording the image. From the captured image, the person appearing in the privacy protection area is identified by appearance. The processing unit has a function of generating an image that cannot be captured by a camera and recording the generated image. When playing back images captured by the camera, people who are captured in the privacy protection area may It has the ability to convert objects into images that cannot be identified by their appearance and display them.

[0328] [20th Example] Next, a twentieth embodiment will be described. In the twentieth embodiment, the processing unit The process of converting the results of image analysis into text data or coding and recording them is included. Execute.

[0329] By converting the images into text data or encoding them, it is possible to easily extract the desired images from the recorded images. For example, the processing unit detects a person in the image and "Person detection" as text data, and "Red light detection" as text data when a red light is detected in the image. The processing unit generates a "red light" by In still another modification, the processing unit Instead of text data, image analysis results are associated with codes, and image analysis results are converted into codes. It has the function of converting data into binary data.

[0330] [21st Example] Next, a 21st embodiment will be described. In the 21st embodiment, the processing unit has a function of image analysis. It has the function of detecting areas of black crush in the image and performs processing according to the analysis results. Then, the process is performed to brighten the areas of the image that are blacked out and display them on the display screen. do.

[0331] Users who view the image can easily recognize objects and people in the black-blurred areas. When capturing images using a spherical camera, the image contains both the bright scenery outside the vehicle and the dark interior of the vehicle. Since the images are taken at the same time, the area where the interior of the car is captured is prone to blackout. The process of detecting the area and restoring it to a brighter state is done by capturing images using a spherical camera. This is particularly effective when the image is brightened. Correction, conversion to black and white, contrast enhancement, etc. are performed.

[0332] [Twenty-second Example] Next, a 22nd embodiment will be described. In the 22nd embodiment, the processing unit notifies the user of the camera. A function to divide the image capture range into multiple sections and to weight each section. The imaging conditions are determined based on the brightness of each section and the weighting of the section. The imaging device has a function of determining the imaging conditions and causing the camera to capture an image under the determined imaging conditions.

[0333] The user weights the areas of the image that he or she wants to see more heavily and weights the areas that he or she does not want to see more lightly. By doing so, it is possible to obtain an image under imaging conditions suitable for the area that the user wants to see. The imaging range can be divided into any shape according to the shape of the area you want to see and the area you do not need to see. can be done.

[0334] FIG. 5 shows a photograph of a comparative example for explaining a method of dividing an imaging range into a plurality of sections. A diagram of the image plane is shown. An image is captured on a rectangular or square imaging plane 71 of a forward-facing 360-degree camera. A circle 73 is formed, and the rectangular or square imaging surface 72 of the rear 360-degree camera An image circle 74 is formed. The view ahead is reflected through the front window, while the dashboard and instrument cluster inside the car are also reflected. The display screen of the car navigation system, etc. are reflected in the image. The driver, passenger seat, rear seat, and roof of the car are captured in the image circle 74. In addition, the scenery to the side and rear of the vehicle is reflected through the side and rear windows. I am.

[0335] The imaging surfaces 71 and 72 are divided into a number of sections by a rectangular grid. A weighting can be applied to each.

[0336] FIG. 6 shows a method for dividing the imaging range into a plurality of sections in the system according to the 22nd embodiment. As in the example shown in FIG. 5, the imaging plane of the forward-facing 360-degree camera is shown. An image circle 73 is formed on the imaging surface 71, and an image circle 73 is formed on the imaging surface 72 of the rear 360-degree camera. An image circle 74 is formed.

[0337] When the user divides the image, the processing unit 51 (FIG. 4) divides a part of the image area into a closed curve 7. 5, and the inner area surrounded by the closed curve 75 is set as one section. By drawing multiple closed curves 75, multiple sections are set. If the inner area is divided into two by a curve, each of the divided areas will be considered as one section.

[0338] As shown in the comparative example in Figure 5, the image is divided into a rectangular grid pattern regardless of the image content. This makes it possible to weight the items more appropriately for the user than if the weights were assigned to the items themselves. do.

[0339] In particular, when using a 360-degree camera, the vehicle model can be captured more clearly than when using a general camera. The difference in the position of the vehicle within the imaging range increases with each shot. The general-purpose weighting table set in the firmware of the camera and the actual expected imaging range are compared. The possibility of mismatching of the parcels increases. Using a general-purpose weighting table The function that allows users to determine the weighting using PC browser software, etc., is 360-degree The effect is particularly noticeable when using a camera.

[0340] The processing unit determines the imaging conditions, such as the gain and exposure time. In a configuration in which the backs of the cameras are facing each other to form a spherical camera, the processing unit The imaging conditions for both cameras are made the same. This allows the images captured by the two hemispherical cameras to be synthesized. Sometimes the effect is to blur the boundary between the two images.

[0341] In a modification of the 22nd embodiment, a processing unit (e.g., AI software of a PC) performs weighting. Without the camera, the system analyzes images captured by the camera and determines what is captured in which part of the image range. For example, the processing unit assigns a weight of "5" to the road area and a weight of "4" to the face area. The weight "3" is added to the instrument panel, and the weight "1" is added to the outside of the circle image. The weight is assigned as “0”.

[0342] The processing means, when making the user divide the area, makes the user surround a part of the area within the image pickup range with a closed curve, The area inside the curved line may be regarded as one section. The processing means may be It is recommended to have the user input the weights as numbers. The numbers to be input are displayed in a pull-down menu or a pop-up menu. It is advisable to allow the user to select from a list such as a pop-up.

[0343] In another modification of the twenty-second embodiment, the processing unit has the following function instead of allowing the user to input the weight as a numerical value: In addition to the function to input numerical values, it has a function to input the type of the partition. A weight is assigned to each of the partitions in advance, and when the partition type is input, the processing unit The weight associated with the section is associated with the section. The types of seats, meter panels, car navigation screens, windshields, rear seats, passenger seats, etc. The processing unit is provided with a function that allows a user to change the correspondence between the type of the partition and the weight. The list of partition types can be registered in advance in the processing unit by the user.

[0344] In yet another modification of the twenty-second embodiment, the processing unit allows the user to assign weights to each section. It has a function to set a zone by specifying the area you want to see within the imaging range. The weight of the area that the user wants to see is increased compared to the weight of other areas. The function allows the user to input the type of partition, and the partition is selected according to the partition the user wants to see, This corresponds to an example of dividing the area into two types: with and without other areas.

[0345] In yet another modification of the twenty-second embodiment, the processing unit prompts the user in an interactive manner (wizard format). The processing section has a function to allow the user to divide the image area. " or "Please fill in the unnecessary areas." The area surrounded or filled by the user is recognized as "unwanted area." Then, the processing unit executes the following commands: "Please fill in the driver's area.", "Please fill in the road area." Please select the area with the driver's type. The processing unit recognizes the area as a section having the type of road or a closed curve. The user can then use a pointing device such as a touch panel or a mouse to perform the action of filling or filling the area. It has the function of using it.

[0346] In yet another modification of the twenty-second embodiment, the processing unit determines the imaging range based on the captured image. The function of dividing the image into multiple sections is realized by artificial intelligence (AI). Automatically distinguishes between inside and outside the vehicle, seats, dashboard, instrument panel, etc. The processing unit has a function to divide the imaging range into separate components. The processing unit displays the classification on the display and allows the user to judge whether the classification is acceptable. If the user determines that the classification made by the AI ​​is incorrect, the user can The processing section can select the correct item from the pull-down or pop-up list. The weight or type corresponding to the partition is incorrect. If the user determines that the weight or type is incorrect, the user can easily correct the weight or type.

[0347] [Twenty-third embodiment] Next, a 23rd embodiment will be described. In the 23rd embodiment, the processing unit notifies the user of the camera. A function to divide the image capture range into multiple sections and to weight each section. The function to process the analysis results is to reduce the block size when compressing blocks with relatively small weights. A process of compressing partitions with relatively large weights by setting the lock larger than the block size when compressing the partitions Execute.

[0348] The user can increase the weight of the partitions with high importance and decrease the weight of the partitions with low importance. This allows for more compression of less important areas, resulting in a uniform compression of the entire image. Compared to compression, the image data is reduced while maintaining high image quality for important areas. For example, the processing unit can reduce the capacity of the weighting block by 8×8 pixels. In the low-weighted area, 16 × 16 pixels are treated as one block. Compression is performed using the

[0349] In a modification of the 23rd embodiment, the processing unit paints the area outside the image circle black. The area outside the image circle formed on the rectangular imaging surface of the camera The lens barrel may be captured in the image of the area outside the image circle. By filling in the unwanted image with black, the image data including the pixels outside the image circle is In another variant, the processing unit is The pixels on the outside of the image circle are excluded, and only the pixels inside the image circle are raster scanned to create an image. This allows the volume of image data to be reduced.

[0350] [24th Example] Next, a 24th embodiment will be described. In the 24th embodiment, the processing unit has a function of image analysis. It has the function of detecting the following vehicle and the function of detecting the distance to the following vehicle, and the analysis result is In response to this result, the system detects aggressive driving by the following vehicle based on the distance to the following vehicle. Perform the process.

[0351] The processing unit executes a process according to the result of the determination to take an appropriate measure against aggressive driving. For example, the processing unit may perform a process when the speed of the vehicle is equal to or greater than a predetermined reference value. However, the same following vehicle continues to exist within a predetermined judgment distance for a predetermined judgment time or more. When this happens, it is determined that the following vehicle is engaging in aggressive driving.

[0352] In a modification of the 24th embodiment, the camera measures the distance to the object for each pixel. For example, a back-illuminated time-series camera sensor is used. An off-flight type range image sensor is used.

[0353] In another modification of the twenty-fourth embodiment, the processing unit detects that the following vehicle is tailgating. When it is determined that the vehicle is in a collision, the process may be performed according to the result of the determination, for example, by notifying the driver or recording the rear image. It records the traffic, gives advice to the driver, and reports the traffic to traffic enforcement agencies (e.g. the police). When reporting to the enforcement agency, the processing unit will identify the vehicle and its driver. The processing unit detects by image analysis that the vehicle engaging in aggressive driving is the vehicle itself. If the system detects that a vehicle has passed you and gone around in front of your vehicle, it will harass you by making a sudden stop. Notify the driver of the possibility.

[0354] In yet another modification of the twenty-fourth embodiment, the processing unit detects people and vehicles around the vehicle. The processing unit has a function to control the vehicle according to the distance to the person and the vehicle. The speed of the vehicle, the distance to the person ahead, and the result of judging whether the person is on the vehicle's planned driving route If it is determined that there is a high possibility of contact with a person based on this information, control is performed to stop the vehicle.

[0355] The scope of the present invention is not limited to the configurations or limitations explicitly described in the specification, but is limited to the scope of the present invention. Combinations of the various aspects of the invention disclosed herein are also intended to be within the scope of the present invention. The claimed invention is a device that is not patentable, but is not a device that is patentable. Even if the configuration is not specified in the claims, the configuration disclosed in this specification may be used in the future. It is our intention to include these claims in the future.

[0356] The present invention is not limited to the configurations described in the above-mentioned embodiments. The components of the modified examples may be arbitrarily selected and combined. Any component of the embodiment and any component or invention described in the means for solving the invention Any of the components described in the means for solving the above problems may be arbitrarily combined with any of the components that embody the above problems. It is also possible to obtain rights to these by amending the present application or filing a divisional application, etc. Have the intention to do so.

[0357] In addition, by converting an application to a design application, it is possible to obtain rights to the entire design or a partial design. The drawings show the entire device in solid lines, but the entire design as well as the device itself are For example, a drawing that includes a partial design claimed for a part of the device. In addition to designing a component as a partial design, a part of the device may be designated as a partial design regardless of the component. The drawings are included as separate designs. It may be included in the overall design, or may be included as a part of the component. The applicant intends to obtain a patent for a partial design in which any part of the design is designated as the dashed line part. [Explanation of symbols]

[0358] 1. Drive recorder 2. Windshield 3. Rearview mirror 4 Power cable 5 Cigarette lighter socket 10. Drive recorder body 11 Camera 12 GPS receiver 13 SD card reader 14 Acceleration Sensor 15 Speaker 16 Display 17 Operation buttons 20 Processing section 21 Communication equipment 30 Servers 31 Processing section 32 Communication equipment 33 Input Devices 34 Display 40 Communication Network 50 External equipment 52 SD card reader 53 Input Devices 54 Display 60 SD card 71, 72 Imaging surface 73, 74 Image Circle 75 Closed curve

Claims

1. A system having a processing means having a function of performing image analysis of an image captured by a camera attached to a vehicle and executing a process corresponding to the analysis result based on the analysis result, The processing means has, as a function of the image analysis, a function of detecting a change in the state of a display unit of an apparatus having a display unit mounted in the vehicle, and, as a process corresponding to the analysis result, executes a process of associating the detected sound information with the apparatus whose display unit state has changed, and storing the detected sound information and the apparatus whose display unit state has changed, when the detected sound information and the change in the state of the display unit are synchronized; The processing means has a function of, when detecting sound information from a microphone, identifying the device that has emitted the sound based on the association between the stored sound information and a change in the state of the display unit. A system characterized by.

2. Equipped with a function to record images captured by the camera The system of claim 1 .

3. The processing means has a voice recognition function, judges whether the sound information from the microphone is generated from a specific device, and cancels the voice recognition function if the sound information is generated from a specific device.

3. The system according to claim 1 or 2, characterized in that

4. The processing means has a function of recording an image surrounding the area in the image of the inside of the vehicle in which the device that generated the sound is shown, superimposing the image on the original image, when the processing means determines that the sound information from the microphone is generated from a specific device.

4. A system according to claim 1, wherein:

5. The processing means has a function of recording a semi-transparent pattern superimposed on the area of ​​the image inside the vehicle in which the device that generated the sound is captured when the processing means determines that the sound information from the microphone is generated from a specific device.

4. A system according to claim 1, wherein:

6. A program for causing a computer to realize the functions of the processing means of the system according to any one of claims 1 to 5.

Images