Image processing device and image processing program

Abstract

To provide an image processing device and an image processing program which assist efficient disinfection of a space with the possibility that a virus adheres to somewhere.SOLUTION: A video acquisition section 21 acquires a video of a space with a person, which has been taken by an imaging section. A video analysis section 22 analyzes the acquired video so as to detect places which the person has touched with his / her hands in the space. A picture generation section 24 generates displaying pictures allowing all the touched places which the person has touched with his / her hands to be recognized from the start of the imaging by the imaging section. The imaging section may be installed in a vehicle so as to image a person in the interior of the vehicle.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device that processes video captured by an imaging unit, and an image processing program. [Background technology]

[0002] As of 2020, COVID-19 (hereafter referred to as the novel coronavirus) infections are spreading worldwide. Measures to prevent infection with the novel coronavirus include wearing masks, washing hands, and maintaining social distance. In addition, to prevent infection through objects, people are urged to frequently disinfect objects that they have touched.

[0003] Because the inside of a car is a small, enclosed space, if the car is used in a poorly ventilated environment, viruses attached to the interior fixtures can easily remain. For example, in rental cars or car sharing, where you cannot know what the previous passenger was like when they got in, it is necessary to thoroughly clean and disinfect the interior of the car. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2011-257849 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, when cleaning and disinfection is required in a short period of time, it is difficult to thoroughly clean and disinfect the entire interior of the car. When cleaning and disinfecting the interior of a car, if it is possible to identify areas where the virus may be present, it is possible to disinfect those areas first, making the disinfection process more efficient.

[0006] The present embodiment has been made in consideration of these circumstances, and its purpose is to provide a technology that assists in the efficient disinfection of spaces where viruses may be present somewhere. [Means for solving the problem]

[0007] In order to solve the above problem, an image processing device according to one aspect of the present embodiment includes: an image acquisition unit that acquires an image of a vehicle interior space in a vehicle in which a person is present, the image being captured by an imaging unit installed in the vehicle; an image analysis unit that analyzes the acquired image and detects a location in the vehicle interior space that the person has touched with his / her hands; and an image generation unit that generates a display image that can recognize all of the accessories that the person has touched with his / her hands from the start of imaging by the imaging unit. an object recognition unit that determines whether a face of a person included in the video is a surveillance target; Equipped with The image analysis unit detects a place touched by the person who is the monitoring target. do.

[0008] Any combination of the above components and any conversion of the expression of this embodiment between an apparatus, a method, a system, a recording medium, a computer program, etc. are also valid as aspects of this embodiment. Effect of the Invention

[0009] This embodiment can assist in the efficient disinfection of spaces where viruses may be present. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of installation of an imaging device in a vehicle. [Diagram 2] 1 is a diagram illustrating an example of a configuration of a person monitoring system according to a first embodiment. [Diagram 3] FIG. 2 is a diagram showing an example of a frame image captured while a vehicle is traveling in the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a contact place image according to the first embodiment. [Diagram 5] FIG. 11 is a diagram showing another example of a contact place image according to the first embodiment. [Figure 6] FIG. 11 is a diagram showing a first example of a stay place image according to the second embodiment. [Figure 7] FIG. 11 is a diagram showing a second example of a stay place image according to the second embodiment. [Figure 8] 8(a)-(c) are diagrams showing a third example of a stay place image according to the second embodiment. [Figure 9] FIG. 11 is a diagram showing a fourth example of a stay place image according to the second embodiment. [Figure 10] FIG. 11 is a diagram showing a fifth example of a stay place image according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] A first embodiment of the present invention relates to a person monitoring system that photographs occupants with a camera installed inside a vehicle cabin and monitors and records places that the occupants have touched with their hands.

[0012] Fig. 1 is a diagram showing an example of installation of an imaging device 10 inside a vehicle C1. In Fig. 1, the imaging device 10 is attached to the rearview mirror. The imaging device 10 may also be installed on a center visor or center console. The imaging device 10 may also be installed on the ceiling inside the vehicle cabin so as to overlook the interior of the vehicle cabin from above. For example, if a circular fisheye lens is used, the entire hemispherical circumference of the vehicle cabin can be captured in the angle of view.

[0013] A dedicated camera may be installed as the imaging device 10, or a camera of a drive recorder or an in-vehicle monitoring system may be used. The in-vehicle monitoring system is a system that monitors the entire vehicle interior including the passenger seat and the rear seats, and can detect not only whether the driver is dozing or looking away, but also the number of passengers sitting in the passenger seat and the rear seats, and whether all passengers including the driver are wearing seat belts.

[0014] The imaging device 10 includes a lens, a solid-state imaging element, and a signal processing circuit. For example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Devices) image sensor can be used as the solid-state imaging element. The solid-state imaging element converts light incident through the lens into an electrical video signal and outputs it to the signal processing circuit. The signal processing circuit performs signal processing such as A / D conversion and noise removal on the video signal input from the solid-state imaging element, and outputs the signal to the image processing device 20.

[0015] The imaging device 10 may include a depth sensor for generating a distance image. For example, a 3D-LiDAR (Light Detection and Ranging) can be used as the depth sensor. The LiDAR irradiates laser light in the shooting direction, measures the time it takes to receive the reflected light, and detects the distance to an object in the shooting direction. The imaging device 10 may also be configured with a two-lens stereo camera. In this case, a distance image can be generated based on the parallax of two images, even without providing a depth sensor.

[0016] 2 is a diagram showing an example of the configuration of the person monitoring system 1 according to the first embodiment. The person monitoring system 1 includes an image processing device 20, a display device 30, a recording medium 40, and an audio output device 50. The image processing device 20 includes a video acquisition unit 21, a video analysis unit 22, an image storage unit 23, an image generation unit 24, and a warning control unit 25. These components can be realized by a combination of hardware resources and software resources, or by hardware resources alone. As the hardware resources, a CPU, a ROM, a RAM, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an ISP (Image Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), and other LSIs can be used. As the software resources, a program such as firmware can be used.

[0017] The image processing device 20 may be mounted on a dedicated IC board, or may be integrated with a car navigation system or a drive recorder system. The image processing device 20 may also be configured as one function of a general-purpose information terminal device such as a smartphone or tablet. The imaging device 10 and the image processing device 20 are connected by wire or wirelessly. When the image processing device 20 is mounted on a dedicated IC board, the image processing device 20 and the imaging device 10 may be installed in the same housing or in separate housings.

[0018] The video acquisition unit 21 acquires the video captured by the imaging device 10. The video analysis unit 22 analyzes the video acquired by the video acquisition unit 21. In the first embodiment, the video analysis unit 22 detects a place in the vehicle cabin that an occupant has touched with his / her hand.

[0019] The video analysis unit 22 includes an object recognition unit 22a, an object tracking unit 22b, and a three-dimensional space recognition unit 22c. The object recognition unit 22a searches for an object within a frame of a video acquired by the video acquisition unit 21. The object recognition unit 22a has a classifier for a specific object generated by learning a large number of images in which the specific object is captured as dictionary data. In the first embodiment, a classifier for a person's hand is prepared. The classifier for the person's hand includes at least a classifier for the back of the hand in an open state. Note that it is preferable to prepare a plurality of classifiers for the hand, which view hands in various states from various angles.

[0020] In addition to the hand identifier, identifiers for various accessories present in the vehicle cabin may be prepared, such as a steering wheel, a seat belt, a shift lever, etc.

[0021] The object recognition unit 22a searches within the video frame using a classifier for each object. For example, HOG (Histograms of Oriented Gradients) features can be used to recognize the object. Haar-like features and LBP (Local Binary Patterns) features may also be used. When an object exists within the frame, the object recognition unit 22a captures the object within a rectangular detection frame.

[0022] The object tracking unit 22b tracks the object recognized by the object recognition unit 22a in the subsequent frames. For example, a particle filter or a mean shift method can be used to track the object. In the first embodiment, the object to be tracked is the hand of a person recognized by the object recognition unit 22a.

[0023] The three-dimensional space recognition unit 22c maps the object detected by the object recognition unit 22a in the three-dimensional space specified by the distance image acquired from the imaging device 10. In the first embodiment, the three-dimensional space recognition unit 22c detects the place touched by the occupant's hand in the three-dimensional space. When the equipment present in the vehicle cabin is also recognized as an object, the equipment touched by the occupant's hand can also be identified.

[0024] When the three-dimensional space recognition unit 22c detects a frame image in which the occupant's hand is touching an equipment item, the three-dimensional space recognition unit 22c temporarily stores the detected frame image in the image storage unit 23. The image storage unit 23 is a volatile memory for temporarily storing the frame image.

[0025] The image generating unit 24 generates an image for displaying contact places (hereinafter, referred to as a contact place image) that allows the user to recognize all contact places touched by the occupant in the vehicle cabin from the start of imaging by the imaging device 10, based on the frame images stored in the image holding unit 23. The image generating unit 24 may generate, as the contact place image, a heat map image in which the more likely the place is to have been touched by the occupant's hand, the more noticeable the color. For example, a heat map image classified into three stages based on the possibility of contact may be generated. For example, a heat map image may be generated in which the area touched by the occupant's hand is displayed in red, the area that may have been touched in yellow, and the area that has not been touched in green. Note that the area that has not been touched may be colorless.

[0026] The image generating unit 24 may generate a contact place image in which the area touched by the occupant's hand is displayed the brightest, the area that may have been touched is displayed the next brightest, and the area that has not been touched is displayed the darkest. Alternatively, a contact place image may be generated in which the area touched by the occupant's hand flashes.

[0027] The area in which the display mode is highlighted may be an accessory unit or an area that is actually touched. When highlighting an accessory unit, for example, even if only a part of a seat belt is touched, the entire seat belt is highlighted.

[0028] The image generating unit 24 may generate a contact location image in a display manner that makes the contact location more noticeable as the number of times the occupant has touched the contact location increases. For example, the saturation of a contact location that has been touched more frequently may be increased.

[0029] The image generating unit 24 may generate a digest video for displaying the contact location by connecting multiple frame images in which the occupant's hand is touching something, which are stored in the image holding unit 23. The image generating unit 24 may also generate a single still image by superimposing multiple frame images in which the occupant's hand is touching something.

[0030] The image generating unit 24 may generate an image simply listing the equipment touched by the occupant's hand in the vehicle interior using text information. In this case, it is not necessary to store the frame image of the occupant's hand touching the equipment in the image storage unit 23, and it is sufficient to store management information for identifying the equipment touched by the occupant's hand.

[0031] When a user who is attempting to clean and disinfect the interior of the vehicle performs an operation on an operation unit (not shown) to instruct the display of a contact place image, the image generation unit 24 generates a contact place image and displays the generated contact place image on the display device 30.

[0032] The display device 30 may be a display (e.g., a liquid crystal display, an organic EL display) of a car navigation system, a display audio system, or a drive recorder system installed in the vehicle C1. The display device 30 may also be a display of a smartphone or a tablet linked to the car navigation system, the display audio system, or the drive recorder system. The display device 30 may also be a display in a meter panel installed in the vehicle. The display device 30 may also be a display dedicated to the person monitoring system 1 according to this embodiment.

[0033] The image generating unit 24 may generate a contact place image in real time while the vehicle C1 is traveling and display the contact place image on the display device 30. In this case, when an occupant touches something in the vehicle interior, the contact place image is updated. The image generating unit 24 may display, as the contact place image, an AR image in which an attention marker indicating the contact place is superimposed on the video captured by the imaging device 10 on the display device 30.

[0034] The recording medium 40 is a non-volatile recording medium for recording the video captured by the imaging device 10. For example, a semiconductor memory card (e.g., an SD card) or an optical disk can be used. The contact location image generated by the image generating unit 24 can also be recorded on the recording medium 40.

[0035] In the above explanation, it is assumed that the video analysis unit 22 can acquire distance images from the imaging device 10. Providing a depth sensor in the imaging device 10 or configuring the imaging device 10 to have a twin-lens configuration increases costs. In the following explanation, a processing example will be described in which the video analysis unit 22 cannot acquire distance images from the imaging device 10. In this processing example, the three-dimensional space recognition unit 22c is not provided, and a place in the vehicle cabin that the occupant has touched with his / her hand is detected only from the two-dimensional image.

[0036] In this processing example, a large number of images of occupants touching each piece of equipment in the vehicle cabin in various positions are learned, and a plurality of classifiers for recognizing the state in which the occupant is touching the equipment are prepared as dictionary data. A set of classifiers for recognizing the state in which the occupant is touching the equipment may be prepared for each vehicle model. In this case, the position of each piece of equipment in the captured video can be identified with high accuracy, and it can be determined with high accuracy whether the occupant is touching the equipment.

[0037] Further, as dictionary data, a classifier for recognizing the face of a person wearing a mask may be prepared, and a classifier for recognizing a state in which an occupant is coughing may be prepared.

[0038] When the object recognition unit 22a detects an occupant without a mask in the frame image, it outputs a detection signal of not wearing a mask to the warning control unit 25. When the object recognition unit 22a detects an occupant coughing in the frame image, it outputs a detection signal of coughing to the warning control unit 25. When the distance between two occupants sitting in the back seat in the frame image is equal to or less than a set value, the object recognition unit 22a outputs a detection signal of approaching distance to the warning control unit 25. When the warning control unit 25 receives a detection signal of not wearing a mask, coughing, or approaching distance from the object recognition unit 22a, it causes the audio output device 50 to output a warning sound or warning message.

[0039] The audio output device 50 may be a speaker of a car navigation system or a display audio system installed in the vehicle C1. The audio output device 50 may also be a speaker of a smartphone or a tablet linked to the car navigation system, the display audio system, or the drive recorder system. The audio output device 50 may also be a speaker dedicated to the person monitoring system 1 according to the present embodiment.

[0040] When a detection signal indicating that a mask is not being worn is input from the video analysis unit 22, the warning control unit 25 outputs an attention-calling message such as "Please wear a mask" to the audio output device 50. When a detection signal indicating that a cough is being detected is input, the warning control unit 25 outputs an attention-calling message such as "Please ventilate the room," "Please open a window," "Please set the air conditioner to the outside air introduction mode," or "Please operate the negative ion generator" to the audio output device 50. The person monitoring system 1 according to this embodiment may be linked to an air conditioning system in the vehicle C1. In that case, when a detection signal indicating that a cough is being detected is input, the warning control unit 25 can automatically switch the air conditioning system to the outside air introduction mode.

[0041] When a detection signal of approaching distance is input from the video analysis unit 22, the warning control unit 25 outputs an attention-calling message such as "Please leave a little more space between you seats" to the audio output device 50. When the video analysis unit 22 detects three occupants in the back seats but no occupant in the passenger seat, it may output a seat change instruction signal to the warning control unit 25. When a seat change instruction signal is input from the video analysis unit 22, the warning control unit 25 outputs an attention-calling message such as "One person, please move to the passenger seat" to the audio output device 50.

[0042] By registering an image of a specific person, the specific person can be excluded from the target of monitoring processing of the place touched by the hand. Specifically, the face of the person to be excluded from the target of monitoring processing is captured by the imaging device 10 and registered as dictionary data in the object recognition unit 22a. Note that a still image showing the face of the person may be input to the image processing device 20 from outside.

[0043] When the object recognition unit 22a recognizes a person's face in the frame image, it determines whether or not the person's face is the face of a person registered as an excluded person. If the face is the face of a person to be excluded, the object recognition unit 22a or the three-dimensional space recognition unit 22c excludes the hand of the person from the monitoring target. If the face of the person recognized in the frame image is not the face of a person to be excluded, the above-mentioned detection and recording process of the place where the hand touches is applied.

[0044] The owner of the vehicle C1 and his / her family members may be registered as excluded persons. Also, persons who are determined to be unlikely to be infected based on the results of a PCR test, an antigen test, or an antibody test may be registered. For example, by registering the owner of the vehicle C1 and his / her family members as excluded persons, the load on the image processing device 20 can be reduced.

[0045] 3 is a diagram showing an example of a frame image F1 captured while a vehicle C1 is traveling according to the first embodiment. In this frame image F1, a driver P1, a passenger P2 in the front passenger seat, and two passengers P3 and P4 in the back seats are detected as people. A steering wheel O1, a bottle holder O2a on a center console O2, and the like are detected as equipment in the vehicle cabin. In addition, the hand Dh1 of the driver P1 is tracked as a monitoring target. Although not shown, the hands of the passengers P2, P3, and P4 are also monitored.

[0046] Fig. 4 is a diagram showing an example of a contact place image Ft1 according to the first embodiment. When a user who is about to clean and disinfect the interior of the vehicle issues an instruction to display the contact place image Ft1, the contact place image Ft1 is displayed on the display device 30. Fig. 4 shows an example in which the driver P1 touches the steering wheel O1 and the bottle holder O2a while driving. In Fig. 4, the steering wheel O1 and the entire area of ​​the center console O2 including the bottle holder O2a are marked in red.

[0047] Fig. 5 is a diagram showing another example of the contact place image Ft2 according to embodiment 1. In Fig. 5, the steering wheel O1 and the bottle holder O2a are marked in red, and the entire area of ​​the center console O2 where the bottle holder O2a is installed (excluding the area of ​​the bottle holder O2a) is marked in yellow.

[0048] The contact location image Ft may be an image of the interior of the vehicle that is actually captured with a warning marker superimposed thereon, which indicates an area where the virus may be present, or may be an image of a simplified schematic diagram of the interior of the vehicle that has a warning marker superimposed thereon. For equipment that the passenger always touches with their hands (e.g., steering wheel, shift lever, turn signal lever, door handle, etc.), a warning marker may be superimposed by default. The contact location image Ft may be generated using a heat map image.

[0049] As described above, according to the first embodiment, a contact place image showing an area that may have been touched by an occupant is generated and displayed, so that a user who is going to clean and disinfect can intuitively understand which area should be disinfected first. This is particularly useful for people who are not in the vehicle C1, such as cleaning staff at a rental car company or gas station attendants, and who need to clean and disinfect the interior of the vehicle efficiently in a short time. Of course, it is also difficult for people who are in the vehicle C1 to understand and remember the places touched by all the occupants, so it is useful for them to clean and disinfect while looking at the contact place image.

[0050] Next, a person monitoring system 1 according to embodiment 2 will be described. In embodiment 1, a person monitoring system 1 was described that captures an image of an occupant with a camera installed in a vehicle cabin, detects the occupant's hands, and monitors and records the places the occupant touches with their hands. In contrast, embodiment 2 relates to a person monitoring system 1 that detects a person in a room in a building with a camera installed in the room, and monitors and records the places where the person has stayed.

[0051] The person monitoring system 1 according to the second embodiment is assumed to be used for monitoring a large indoor space such as a conference room or a convention hall. Indoor monitoring cameras are generally installed on the ceiling of a room. When monitoring a large room with a single monitoring camera, it is necessary to shoot with a wide angle, which reduces the size of people in the frame image. In addition, a large number of people may be included in the frame image. In contrast, in the first embodiment, since a narrow vehicle interior space is shot, the size of people in the frame image does not fall below a certain size. In addition, in the case of a standard vehicle, the maximum number of people included in the frame image is five.

[0052] As described above, it is difficult to detect the hands of each person by image recognition from frame images captured by the imaging device 10 installed in a conference room, a convention hall, etc., and to track the detected hands. Therefore, in the second embodiment, the head or whole body of a person is detected, and the detected head or whole body is tracked.

[0053] In the following explanation, an example is assumed in which the imaging device 10 is installed in the center of the ceiling of a room and captures an image of the entire room from above. In this case, the monitoring target is a person's head as viewed from above. Note that the imaging device 10 may also be installed at the corner between the ceiling and a wall and capture the entire room from an angle looking down. In this case, the monitoring target is a person's head or whole body as viewed from an angle.

[0054] The image processing device 20 is configured as one function of a general-purpose information terminal device such as a PC, a tablet, a smartphone, etc. The image processing device 20 and the imaging device 10 are connected by wire (for example, a LAN cable or a USB cable) or wirelessly (for example, wireless LAN), and the image acquisition unit 21 acquires images from the imaging device 10.

[0055] In the second embodiment, the video analysis unit 22 detects a place in a room where a person has stayed. The object recognition unit 22a has a classifier for a person's head or whole body as dictionary data. In addition to the classifier for a person's head or whole body, classifiers for various furniture present in the room may be prepared. For example, classifiers for a desk, a chair, etc. may be prepared.

[0056] The object recognition unit 22a may set an associated region around the detection region of the person in the frame image. For example, an area in the image equivalent to a circle with a radius of about 1 m to 2 m centered on the person may be set as the associated region. Hereinafter, the area including the detection region of the person and the associated region set around the person is referred to as the behavior region. The conversion ratio between the distance in the real space and the distance in the image, and the setting range of the associated region can be changed by the user.

[0057] The object tracking unit 22b tracks the head or the whole body of the person recognized by the object recognition unit 22a. When multiple people are recognized in the frame image, each person is tracked.

[0058] The three-dimensional space recognition unit 22c detects the location where the person stayed within the three-dimensional space specified by the distance image acquired from the imaging device 10. When furniture present in the room is also recognized as an object, the furniture that is presumed to have been touched by the person can also be identified. For example, when an overlap occurs between the person's activity area and the detection area of ​​the furniture, it is presumed that the person touched the furniture.

[0059] When the three-dimensional space recognition unit 22c detects a frame image in which it is estimated that a person has touched the fixture, the three-dimensional space recognition unit 22c temporarily stores the detected frame image in the image storage unit 23. Note that in the second embodiment, all frame images in which a person is detected may be temporarily stored in the image storage unit 23.

[0060] The image generating unit 24 generates an image for displaying stay locations (hereinafter referred to as a stay location image) that allows the user to recognize all the stay locations where the person has stayed in the room from the start of imaging by the imaging device 10, based on the frame images stored in the image holding unit 23. The image generating unit 24 may generate a stay location image that allows the user to recognize the movement trajectory of the person in the room. The image generating unit 24 may also generate a stay location image that allows the user to recognize the trajectory of the activity area accompanying the movement of the person in the room.

[0061] The image generating unit 24 may generate a heat map image as the stay location image in which the closer the location is to the location where the person stayed, the more noticeable the color is displayed. For example, a heat map image may be generated in which the area where the person stayed is displayed in red, the associated area is displayed in yellow, and the other areas are displayed in green (or colorless). Also, a heat map image may be generated in which the area where the person stayed is displayed in red, and the color gradually changes from red to yellow to green (or colorless) as the area moves away from the area where the person stayed.

[0062] The image generating unit 24 may generate a stay place image in which the brightness of the area where the person stayed is the brightest and the brightness is gradually decreased as the person moves away from the area where the person stayed. Also, the image generating unit 24 may generate a stay place image in which the area where the person stayed blinks.

[0063] When an overlap occurs between the person's activity area and the detection area of ​​the furniture, the image generating unit 24 may display the entire furniture in a prominent manner, or may display only the overlapping area in a prominent manner. When the height of the furniture from the floor can be estimated, the image generating unit 24 may display the furniture in a prominent manner as the height of the furniture from the floor approaches the height of the person's hands when sitting or standing.

[0064] The image generating unit 24 may display an area in a more conspicuous manner, the more overlapping the detection areas of multiple people are. For example, the saturation may be increased, the more overlapping the detection areas are. The image generating unit 24 may display an area in a more conspicuous manner, the more overlapping the activity areas of multiple people are.

[0065] The image generating unit 24 may generate a digest video for displaying the stay location by connecting a plurality of frame images in which overlapping areas occur between the person's activity area and the equipment detection area stored in the image holding unit 23. The image generating unit 24 may also generate a single still image by superimposing a plurality of frame images in which overlapping areas occur between the person's activity area and the equipment detection area.

[0066] The image generating unit 24 may generate an image in which the furniture in the room that is presumed to have been touched by the person is simply listed in text information. In this case, it is not necessary to store the frame image in the image storage unit 23, and it is sufficient to store management information for identifying the furniture that is presumed to have been touched by the person.

[0067] When a user who is about to clean and disinfect the room performs an instruction operation on an operation unit (not shown) to display an image of the place of stay, the image generation unit 24 generates an image of the place of stay and displays the generated image of the place of stay on the display device 30.

[0068] The display device 30 may be a display of a general-purpose information terminal device such as a PC, a tablet, or a smartphone. The display device 30 may also be a projector connected to the information terminal device. In this case, the projector projects the stay place image onto a screen. The display device 30 may also be a head-mounted display connected to the information terminal device. In this case, the head-mounted display can display the stay place image as a VR image. Note that when the display image for the contact place is generated as a stereoscopic image, the head-mounted display can display the stereoscopic VR image.

[0069] The image generating unit 24 may generate a stay place image in real time while the room is in use (e.g., during a meeting) and display it on the display device 30 (e.g., a projector). In this case, the stay place image is updated when a person enters or leaves the room, or when a person moves to another location within the room. The image generating unit 24 may display, on the display device 30, an AR image in which an attention marker indicating the person's stay place is superimposed on the video captured by the imaging device 10, as the stay place image.

[0070] In the second embodiment as well, when the distance between people in a frame image is equal to or less than a set value, the object recognition unit 22a can output a distance approach detection signal to the warning control unit 25. When the warning control unit 25 receives a distance approach detection signal from the object recognition unit 22a, it causes the audio output device 50 to output a warning sound or a warning message. Note that even when a person not wearing a mask or a person coughing is detected, it can cause the audio output device 50 to output a warning sound or a warning message.

[0071] Also in the second embodiment, by registering an image of a specific person, the registered person can be excluded from targets of monitoring processing of places of stay.

[0072] FIG. 6 is a diagram showing a first example of a stay place image Fs1 according to the second embodiment. When a user who is about to clean and disinfect the room instructs the display device 30 to display the stay place image Fs1, the stay place image Fs1 is displayed. FIG. 6 shows a stay place image Fs1 in a case where three people P1, P2, and P3 are using the first desk D1 on the left side while the room is in use. The second desk D2 on the right side is not in use. In FIG. 6, the first desk D1 on the left side is marked in red as a caution area.

[0073] The object recognition unit 22a judges whether or not the detection area of ​​each of the persons P1, P2, P3 overlaps with the detection area of ​​the first desk D1 in each frame image, and whether or not the detection area of ​​each of the persons P1, P2, P3 overlaps with the detection area of ​​the second desk D2. The object recognition unit 22a judges the desk with the overlap as the used desk (the first desk D1 in FIG. 6).

[0074] Fig. 7 is a diagram showing a second example of a stay place image Fs2 according to the second embodiment. Fig. 7 shows a stay place image Fs2 displaying a movement trajectory of a person P1 in a room. In the image shown in Fig. 7, the entrance of the room is on the right side, and the person P1 moves left under the second desk D2 and the first desk D1, moves up the left side of the first desk D1, and moves right above the first desk D1 to reach the current location. Note that a movement trajectory of a behavior area with an associated area added around the person P1 may be displayed.

[0075] 8(a)-(c) are diagrams showing a third example of a stay place image Fs3 according to the second embodiment. The third example is an example of a room in which the positions of desks and chairs are fixed, such as an examination hall or a large classroom. In the stay place image Fs3a shown in FIG. 8(a), a first person P1 is sitting in a first chair S1 on the left side. A second chair S2 on the right side is vacant. In this case, the first chair S1 on the left side and the first desk D1 are marked in red as attention areas.

[0076] In the stay place image Fs3b shown in Fig. 8(b), a first person P1 is sitting in a first chair S1 on the left side, and a second person P2 is sitting in a second chair S2 on the right side. In this case, both the first chair S1 and the first desk D1 on the left side, and the second chair S2 and the second desk D2 on the right side are marked in red as attention areas.

[0077] The stay location image Fs3c shown in Fig. 8(c) shows the state after the second person P2, who was sitting in the second chair S2 on the right side, has left. Since the second person P2 stayed at the position of the second chair S2 on the right side and the second desk D2, the second chair S2 on the right side and the second desk D2 are marked in red as attention areas even after the second person P2 has left.

[0078] FIG. 9 is a diagram showing a fourth example of a stay place image Fs4 according to the second embodiment. In the examples shown in FIG. 6 to FIG. 8(a)-(c), the entire area of ​​the desk that is presumed to have been touched by a person is marked in red. In contrast, in the fourth example shown in FIG. 9, only the area of ​​the entire desk that is likely to have been touched by a person is marked in red as an attention area. In FIG. 9, a first range R1 at a predetermined distance from the position of a first person P1 and a second range R2 at a predetermined distance from the position of a second person P2 are marked in red as attention areas. This marked area expands as the persons P1 and P2 move.

[0079] The predetermined distance is set to, for example, 2 m, taking social distancing into consideration. This predetermined distance may be switched depending on whether or not a person is wearing a mask. In that case, the predetermined distance for a person not wearing a mask is set longer than the predetermined distance for a person wearing a mask.

[0080] Furthermore, if the object recognition unit 22a can detect the movement of a person's hand, the area in the entire desk where the hand moved may be marked in red as an attention area. In this case, the area to be marked in the entire desk may be an area where the actual hand movement was detected plus a predetermined buffer area. Furthermore, the area to be marked in the entire desk may be an area that overlaps with the action areas of persons P1 and P2.

[0081] In addition, the overlapping range R3 of the first range R1 and the second range R2 may be marked in a relatively conspicuous color as a higher level attention area. For example, the range where the first range R1 and the second range R2 do not overlap may be marked in green, and the overlapping range R3 between the two may be marked in red. Also, the range of a predetermined distance of one person may be marked in green, the overlapping range of a predetermined distance of two people may be marked in yellow, and the overlapping range of a predetermined distance of three or more people may be marked in red.

[0082] FIG. 10 is a diagram showing a fifth example of a stay place image Fs5 according to the second embodiment. In the stay place image Fs5 shown in FIG. 10, an area R5 of a passageway in a room where people move around a lot is marked in red as an attention area. The color may be changed according to the number of times a person has passed through the passageway. For example, 1 to 5 times may be marked in green, 6 to 10 times may be marked in yellow, and 11 times or more may be marked in red. Although the entire passageway area R5 is marked as an attention area in FIG. 10, only the movement trajectory that the person actually passed through, i.e., only the part where the person and the passageway overlap may be marked.

[0083] As described above, according to the second embodiment, a stay place image showing the area where a person stayed is generated and displayed, so that a user who is going to clean and disinfect can intuitively understand which area should be disinfected first. This is particularly effective for people who did not use the room (e.g., did not participate in a meeting), such as external cleaning companies, and who need to clean and disinfect the room efficiently in a short time. Of course, it is difficult for people who used the room to grasp and remember the movements of everyone who was in the room, so cleaning and disinfecting while looking at the stay place image is useful.

[0084] The present invention has been described above based on the embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention.

[0085] In the first embodiment, for example, while driving, the driver basically holds the steering wheel with his / her hands. On the other hand, when the driver is making other movements (for example, touching a bottle holder to drink water, etc.), the image is significantly different from the image of the normal state in which the driver holds the steering wheel. The object recognition unit 22a may store all frame images in which the driver is not holding the steering wheel with his / her hands in the frame images in the image storage unit 23. The image generation unit 24 may connect a plurality of frame images stored in the image storage unit 23 to generate a digest video for displaying the contact location. The image generation unit 24 may also superimpose a plurality of frame images stored in the image storage unit 23 to generate a single still image. In this case, although the accuracy is reduced, the contact location image can be generated simply.

[0086] In the second embodiment, the object recognition unit 22a may store all frame images in which a person is detected in the frame image in the image storage unit 23. The image generation unit 24 may connect a plurality of frame images stored in the image storage unit 23 to generate a digest video for displaying the stay place. The image generation unit 24 may also superimpose a plurality of frame images stored in the image storage unit 23 to generate a single still image. In this case, although accuracy decreases, it is possible to simply generate a stay place image.

[0087] In the first embodiment, an example of monitoring the interior of a vehicle has been described, with a person's hands as the monitoring target. In this regard, the monitoring is not limited to the example of monitoring the interior of a vehicle, so long as it is a small space where a person's hands can be easily detected. Even if it is a room in a building, the person monitoring system 1 according to the first embodiment functions effectively in the case of monitoring a small room that only a few people can enter.

[0088] In the second embodiment, an example has been described in which a person is the monitoring target and a room in a building is monitored. In this regard, the person monitoring system 1 according to the second embodiment can also be applied to monitoring indoor spaces in moving objects such as trains, buses, airplanes, and ships. It can also be applied to monitoring outdoor spaces within a certain range.

[0089] In addition, the new coronavirus dies in about seven days at the longest, depending on the material it is attached to and the environmental conditions. Therefore, if the set time has passed since the vehicle or room was left unused, there is no need to display the warning area in the image. [Explanation of symbols]

[0090] C1 vehicle, 1 person monitoring system, 10 imaging device, 20 image processing device, 21 video acquisition unit, 22 video analysis unit, 22a object recognition unit, 22b object tracking unit, 22c three-dimensional space recognition unit, 23 image storage unit, 24 image generation unit, 25 warning control unit, 30 display device, 40 recording medium, 50 audio output device.

Claims

1. an image acquisition unit that acquires an image of a vehicle interior space in which a person is present, the image being captured by an imaging unit installed in the vehicle; an image analysis unit that analyzes the acquired image and detects a place in the vehicle interior space that the person touched with his / her hand; an image generating unit that generates a display image that allows the user to recognize all accessories that the user has touched with his / her hands since the image capturing unit started capturing images; an object recognition unit that determines whether a face of a person included in the video is a surveillance target; Equipped with The video analysis unit is an image processing device that detects the locations touched by the person being monitored.

2. the image generation unit generates, as the display image, a heat map image in which an area more likely to have been touched by the person's hand is displayed in a more prominent color; The image processing device according to claim 1 .

3. the image generating unit generates a display image in such a manner that a contact location that has been touched by the person's hand more frequently is more noticeable; 3. The image processing device according to claim 1 or 2.

4. A computer comprising: A process of acquiring an image of a vehicle interior space in which a person is present, the image being captured by an imaging unit installed in the vehicle; A process of analyzing the acquired video to detect a location in the vehicle interior space that the person touched with their hand; A process of generating a display image that can recognize all accessories that the person has touched with his / her hands since the imaging unit started imaging; and executing a process of recognizing an object to determine whether or not a face of a person included in the video is a surveillance target; The detection process is an image processing program that detects the location touched by the person being monitored.

Images