Information processing device

Abstract

To increase the detection accuracy of an image background.SOLUTION: An information processing apparatus comprises a control unit which executes: acquiring a real image as an image captured by an imaging unit; acquiring a calibration image as an image captured by the imaging unit in the past and as an image serving as a reference for a current background; creating a model to which the estimation of an area where a shadow(s) exists and color tone correction are executed based on the real image; and creating an estimated background image as an image to which the shadow(s) corresponding to the real image has been added and color tone correction has been executed with respect to the calibration image based on the calibration image and the model and as a background image corresponding to the real image.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] This disclosure relates to an information processing apparatus.

Background Art

[0002] There is a known technique that simultaneously represents and learns the texture, color, brightness pattern, and movement of a scene in which a background that varies dynamically is photographed by a Gram matrix, and enables separation between the background and the foreground based on information such as texture even when a luminance value similar to the background is input (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of this disclosure is to improve the detection accuracy of the background of an image.

Means for Solving the Problems

[0005] One aspect of this disclosure is acquiring a real image that is an image captured by an imaging unit, acquiring a calibration image that is an image captured by the imaging unit in the past and serves as a reference for the current background, generating a model for estimating an area where a shadow exists and correcting a color tone based on the real image, generating an estimated background image that is an image obtained by adding a shadow corresponding to the real image to the calibration image and correcting the color tone based on the calibration image and the model, and that is an image of the background corresponding to the real image, an information processing apparatus including a control unit that executes the above.

[0006] Another aspect of this disclosure is a method for providing MaaS (Mobility as a Service) using the information processing device described above.

[0007] Other aspects of this disclosure include an information processing method in which a computer performs the processing in the above-mentioned information processing apparatus, a program for causing the computer to perform the processing, or a storage medium that non-temporarily stores the program. [Effects of the Invention]

[0008] According to this disclosure, the accuracy of background detection in images can be improved. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows the hardware configuration of the vehicle according to the embodiment. [Figure 2] This figure shows an example of a functional component of an ECU according to an embodiment. [Figure 3] This diagram shows an example of an image displayed when the control unit determines whether or not a foreign object is present. [Figure 4] This is a diagram to explain the estimated shadow model. [Figure 5] This is a flowchart of the process for generating a floor replacement image in the ECU of a vehicle according to the embodiment. [Figure 6] This flowchart shows an example of the process for generating an estimated shadow model. [Modes for carrying out the invention]

[0010] Conventional background subtraction methods assume that images are captured by a fixed camera and detect fast-moving objects as the foreground. However, in a moving vehicle, in addition to the background visible through the windows, shadows inside the vehicle may also move relatively quickly. If such shadows are extracted as the foreground, there is a risk of misinterpreting, for example, a person or object entering a restricted area.

[0011] Furthermore, it is conceivable that people and objects could be detected by comparing images taken at the time of factory shipment with images acquired in real time. However, if, for example, the interior is changed or stickers are applied to the floor after the product has left the factory, there is a risk that these will be mistakenly identified as foreign objects.

[0012] Therefore, an information processing device according to one aspect of the present disclosure includes a control unit configured to perform the following: acquire a real image which is an image captured by an imaging unit; acquire a calibration image which is an image previously captured by the imaging unit that serves as a reference image for the background at the present time; generate a model that estimates areas where shadows exist and corrects the color tone based on the real image; and generate an estimated background image which is an image in which shadows corresponding to the real image are added to the calibration image and the model and the color tone is corrected, and the background image corresponds to the real image.

[0013] The real image is, for example, an image captured in real time by the imaging unit. The real image may also be, for example, an image of the current moment. The calibration image is an image corresponding to the background at the current moment, for example, an image captured after stickers have been applied to the floor or the interior has been changed, and is an image captured in a state free of foreign objects. The calibration image is captured by the imaging unit each time the floor surface is changed, or when false detections of foreign objects increase due to the effect of dirt. Furthermore, the calibration image is an image captured in a state free of people. The real image and the calibration image are acquired by the control unit after they have been captured by the imaging unit.

[0014] In addition, the control unit generates a model for estimating an area where a shadow exists and correcting the color tone based on the real image. This model, for example, extracts a shadow candidate area that is a candidate for the shadow area from the luminance distribution of the real image, extracts an estimated shadow area by comparing the shadow candidate area with the previously generated estimated background image among the shadow candidate areas, and determines correction parameters for bringing the color tone of the calibration image closer to the color tone of the real image by comparing the real image with the calibration image. This model excludes the area where a person exists from the area where a shadow exists.

[0015] In this way, the control unit generates a model for estimating the area where a shadow exists based on the real image. Then, by applying this model to the calibration image, an estimated background image that is an image with a shadow added is generated for the calibration image. This estimated background image is an image that pseudo-generates the background at the time when the real image was captured. Also, the estimated background image is an image from which foreign objects have been excluded. In this way, an image corresponding to the background can be obtained.

[0016] Hereinafter, embodiments of the present disclosure will be described based on the drawings. The configurations of the following embodiments are examples, and the present disclosure is not limited to the configurations of the embodiments. Also, the following embodiments can be combined as much as possible.

[0017] <First Embodiment> FIG. 1 is a diagram showing the hardware configuration of a vehicle 10 according to an embodiment. The vehicle 10 is, for example a vehicle used in, for example, MaaS and is a vehicle capable of autonomous driving. The vehicle 10 includes an ECU 100 which is an electronic control unit, a camera 21, and an output unit 22. These components are interconnected by a CAN bus which is a vehicle-mounted network bus.

[0018] ECU 100 has a computer configuration. ECU 100 includes a processor 101, a main memory unit 102, an auxiliary storage unit 103, and a communication unit 104. These are interconnected by a bus. Note that the processor 101 is an example of a control unit.

[0019] The processor 101 is a CPU (Central Processing Unit), a DSP (Digital Signal Processor), etc. The processor 101 controls the vehicle 10 and performs various information processing operations. The main memory unit 102 is a RAM (Random Access Memory), a ROM (Read Only Memory), etc. The auxiliary storage unit 103 is an EPROM (Erasable Programmable ROM), a hard disk drive (HDD), a removable medium, etc. The auxiliary storage unit 103 stores an operating system (OS), various programs, various tables, etc. The processor 101 loads the program stored in the auxiliary storage unit 103 into the working area of the main memory unit 102 and executes it. Through the execution of this program, each component, etc. is controlled. Thereby, the ECU 100 realizes a function that meets a predetermined purpose. The main memory unit 102 and the auxiliary storage unit 103 are computer-readable recording media. The communication unit 104 is a means for communicating with a center server, a user terminal, etc. via a network. The communication unit 104 is, for example, a LAN (Local Area Network) interface board, a wireless communication circuit for wireless communication. For example, remote monitoring of the vehicle 10 can be performed via the communication unit 104.

[0020]

[0021] ​Camera 21 is a means for imaging the interior of the vehicle 10. Camera 21 images at least the floor surface near the entrance and exit of the vehicle 10. Camera 21 takes images using an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The images obtained by the imaging may be still images or moving images.

[0022] The output unit 22 is a means of presenting information to passengers or crew members, and may include, for example, an LCD (Liquid Crystal Display) panel, an EL (Electroluminescence) panel, or a lamp. This could be a warning light, or a speaker, etc. Alternatively, the output unit 22 may be a means of notifying an external center server or the like of an abnormality via the communication unit 104.

[0023] Next, the functional components of the ECU 100 of the vehicle 10 will be described. Figure 2 is a diagram showing an example of the functional components of the ECU 100 according to this embodiment. The ECU 100 includes a control unit 110 as a functional component. The processor 101 of the ECU 100 executes the processing of the control unit 110 by a computer program on the main memory unit 102.

[0024] The control unit 110 analyzes the image acquired from the camera 21 to determine the presence or absence of foreign objects. Figure 3 shows an example of an image when the control unit 110 determines the presence or absence of foreign objects. The control unit 110 generates or acquires an ideal floor image 31, a calibration image 32, a real image 33, an estimated floor image 34, and a floor replacement image 35 and stores them in the auxiliary storage unit 103, and performs a determination based on these images. These images correspond to the floor surface 42 including the entrance / exit 41 of the vehicle 10, and are images corresponding to the same location. A door 41A is provided at the entrance / exit 41. The ideal floor image 31 is an example of an ideal background image, the estimated floor image 34 is an example of an estimated background image, and the floor replacement image 35 is an example of a background replacement image. .

[0025] The ideal floor surface image 31 is an image taken at the time of factory shipment and represents the initial state. Since this image is common to other vehicles, the control unit 110 may obtain an image taken in another vehicle of the same type from, for example, a central server. The control unit 110 performs machine learning to determine whether or not foreign objects are present. Based on this machine learning model, the control unit 110 determines whether or not foreign objects are present. This machine learning uses the ideal floor surface image 31 taken in another vehicle of the same type. Alternatively, this machine learning may be performed in other vehicles of the same type. Alternatively, when a user provides predetermined input for acquiring the ideal floor surface image 31 at the time of factory shipment, the camera 21 may capture an image of the floor surface, and the image acquired at this time may be used as the ideal floor surface image 31. In this case, a user interface for capturing the ideal floor surface image 31 may be provided.

[0026] The calibration image 32 is an image captured when a change occurs on the floor surface 42 of the vehicle 10. In the example shown in Figure 3, it is an image captured after a sticker 43 has been applied to the floor surface 42. The calibration image 32 is an image captured when no foreign objects are present. For example, when a user makes a predetermined input to acquire the calibration image 32, the camera 21 may take an image, and the image acquired at this time may be used as the calibration image 32. In this case, there may be a user interface for capturing the calibration image 32. For example, the calibration image 32 may be captured daily before the vehicle 10 is put into operation.

[0027] The real image 33 is an image captured in real time by the camera 21. The real image 33 may contain foreign objects 44 such as objects or people, and shadows 45. The control unit 110 causes the camera 21 to capture the real image 33 at a predetermined interval, for example.

[0028] The estimated floor image 34 is an image of the floor surface at the time of acquisition of the simulated real image 33 (which may be the current time). The estimated floor image 34 is generated by adding shadows to the calibration image 32 and further correcting the color tone. The control unit 110 generates the estimated floor image 34 using an estimated shadow model. The estimated shadow model is a model that estimates the state of the floor surface 42 and the shadows. Conventional techniques can be used for extracting shadows from the real image 33. For example, the estimated shadow model includes information on candidate shadow regions, estimated shadow regions, lighting correction parameters, and estimated human regions.

[0029] Figure 4 is a diagram illustrating the estimated shadow model. The candidate shadow region 33A is an area that can be considered a shadow and is extracted from the overall brightness distribution of the actual image 33. Known techniques can be used for the model that performs this extraction. The estimated shadow region 33B is an area within the candidate shadow region 33A that has a texture similar to the previous estimated floor image 34A and is extracted from the candidate shadow region 33A. The previous estimated floor image 34A is the estimated floor image generated in the previous routine and is stored in the auxiliary storage unit 103. In this case, an area with a texture similar to the shadow part of the previous estimated floor image 34A is extracted. However, the candidate shadow region 33A is the area excluding the estimated human area described later. Known techniques can be used for the model that performs this extraction. The illumination correction parameter is a correction coefficient to bring the color tone of the calibration image 32 closer to the color tone of the actual image 33. The control unit 110 determines the illumination correction parameter for the estimated shadow region 33B and for the estimated bright area, which is the area other than the estimated shadow region 33B. The lighting correction parameters are determined by comparing the calibration image 32 with the real image 33. Known techniques can be used for determining the lighting correction parameters. The estimated human region is the region where a person is estimated to be present, and is typically obtained through general deep learning methods.

[0030] In this way, an estimated shadow model is generated. Because this estimated shadow model does not change over time with respect to the actual image 33, it is possible to exclude fast-moving shadows as described later.

[0031] The floor surface replacement image 35 is an image obtained by replacing the background of the actual image 33 with the ideal floor surface image 31. The control unit 110 detects the floor surface 42 by comparing the actual image 33 with the estimated floor image 34. Known techniques can be used for this detection. For example, the background subtraction method may be used. Then, the control unit 110 generates the floor surface replacement image 35 by replacing the portion of the floor surface 42 detected in the actual image 33 with an image of the same portion of the ideal floor surface image 31. At this time, the foreign object 44 remains in the floor surface replacement image 35 without being replaced.

[0032] Based on the floor replacement image 35 generated in this manner, the control unit 110 detects foreign objects 44. For example, a model for detecting foreign objects 44 generated by deep learning or the like may be stored in the auxiliary storage unit 103. Known techniques can be used for detecting foreign objects 44 based on the floor replacement image 35. For example, foreign objects 44 may be detected by a heuristic method that determines whether the foreground is larger than a predetermined area. Alternatively, a simple two-class classification technique of normal and abnormal may be used.

[0033] When the control unit 110 detects a foreign object 44, it performs, for example, an alert process. In this alert process, for example, a warning is sent from the output unit 22. For example, an announcement may be made by voice to move the foreign object 44 away from the entrance / exit 41, or a message may be displayed on the screen advising the user to move the foreign object 44 away from the entrance / exit 41. Alternatively, the control unit 110 may choose not to open or close the door 41A when it detects a foreign object 44.

[0034] Next, the process for generating the floor replacement image 35 in the ECU 100 of the vehicle 10 will be described. Figure 5 is a flowchart of the process for generating the floor replacement image 35 in the ECU 100 of the vehicle 10 according to this embodiment. The process shown in Figure 5 is executed in the ECU 100 at predetermined intervals. It will be explained assuming that the ideal floor image 31 is already stored in the auxiliary storage unit 103.

[0035] In step S101, the control unit 110 determines whether or not there is a request to capture a calibration image 32. The calibration image 32 is captured when the state of the floor surface 42 changes. For example, when the state of the floor surface 42 changes, a request to capture a calibration image 32 is sent from the center server via the communication unit 104. Alternatively, for example, when a predetermined input is made to the user interface inside the vehicle 10, the control unit 110 determines that there is a request to capture a calibration image 32. If the determination in step S101 is positive, the process proceeds to step S102; if the determination is negative, the process proceeds to step S103.

[0036] In step S102, the control unit 110 captures a calibration image 32. The control unit 110 instructs the camera 21 to capture an image and stores the acquired image as the calibration image 32 in the auxiliary storage unit 103.

[0037] In step S103, the control unit 110 captures a real image 33. The control unit 110 instructs the camera 21 to capture an image and stores the acquired image as a real image 33 in the auxiliary storage unit 103.

[0038] In step S104, the control unit 110 generates an estimated shadow model. Figure 6 is a flowchart showing an example of the process for generating the estimated shadow model. In step S201, the control unit 110 extracts candidate shadow regions 33A based on the actual image 33. Here, Areas that can be considered shadows are extracted from the brightness distribution of the entire image. In step S202, the control unit 110 extracts areas from the candidate shadow areas 33A that have a texture similar to the estimated floor image (previous estimated floor image 34A) generated by the routine shown in Figure 5, as estimated shadow areas. Whether or not the textures are similar may be determined according to conditions stored in the auxiliary storage unit 103 beforehand. In step S203, the control unit 110 compares the calibration image 32 with the actual image 33 to determine illumination correction parameters for the estimated shadow areas 33B and estimated bright areas, respectively.

[0039] Returning to Figure 5, in step S105, the control unit 110 generates an estimated floor image 34 based on the calibration image 32 captured in step S102 and the estimated shadow model generated in step S104. The generated estimated floor image 34 is stored in the auxiliary storage unit 103. In step S106, the control unit 110 detects the floor surface 42 based on the actual image 33 captured in step S103 and the estimated floor image 34 generated in step S105. At this time, the floor surface 42 is detected with the foreign objects 44 visible in the actual image 33 excluded. In other words, the control unit 110 detects the background portion.

[0040] In step S107, the control unit 110 acquires an ideal floor surface image 31. Since the ideal floor surface image 31 is stored in the auxiliary storage unit 103 beforehand, the control unit 110 reads the ideal floor surface image 31 from the auxiliary storage unit 103. In step S108, the control unit 110 generates a floor surface replacement image 35 by replacing the portion of the floor surface 42 detected in step S106 with the corresponding portion in the ideal floor surface image 31 acquired in step S107.

[0041] The floor replacement image 35 generated in this way is an image in which the foreign object 44 remains, and the shadow and sticker have been removed. Furthermore, since the floor surface (which can also be used as the background) is in the same state as when machine learning was performed to detect the foreign object 44, the foreign object 44 can be detected with high accuracy using the trained machine learning model.

[0042] In this case, if stickers or other materials are applied to the floor surface when machine learning is performed, or if the interior is changed to a different pattern, there is a risk that the machine learning model will incorrectly detect the stickers or interior materials as foreign objects. Furthermore, while vehicle 10 is in motion, the light entering the vehicle through the windows changes moment by moment, causing the shape of the shadows cast on the floor to change moment by moment as well. Therefore, conventional logic may have detected the shadows as foreign objects.

[0043] In contrast, in this embodiment, by replacing the area identified as the floor surface with a pre-prepared ideal floor surface image 31, foreign objects 44 can be easily detected, for example, using a trained machine learning model. Therefore, large-scale calculations such as performing machine learning again after the vehicle 10 starts operating are unnecessary, and maintenance costs can be significantly reduced.

[0044] <Other Embodiments> The embodiments described above are merely examples, and this disclosure may be modified and implemented as appropriate without departing from its essence. The processes and means described in this disclosure can be freely combined and implemented as long as no technical inconsistencies arise. Furthermore, processes described as being performed by one device may be divided and executed by multiple devices. Alternatively, processes described as being performed by different devices may be executed by a single device. In a computer system, the hardware configuration (server configuration) by which each function is implemented can be flexibly changed. For example, the processes in the embodiments described above may be executed by a computer outside the vehicle.

[0045] In one example, the vehicle 10 or ECU 100 according to the embodiment utilizes mobility This may be used to provide MaaS (Mobility as a Service). Furthermore, for example, the processing procedures shown in Figures 5 and 6 may be executed when providing a service (MaaS) using a vehicle 10 or ECU 100. In this case, the information processing method according to the above processing procedures is an example of a MaaS (Mobility as a Service) provision method.

[0046] The present disclosure can also be realized by supplying a computer program implementing the functions described in the embodiments above to a computer, and having one or more processors in the computer read and execute the program. Such a computer program may be provided to the computer by a non-temporary computer-readable storage medium that can be connected to the computer's system bus, or it may be provided to the computer via a network. Non-temporary computer-readable storage mediums include, for example, any type of disk such as magnetic disks (floppy disks, hard disk drives (HDDs), etc.), optical disks (CD-ROMs, DVDs, Blu-ray discs, etc.), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic cards, flash memory, optical cards, and any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0047] 10 vehicles 21 Cameras 100 ECU 101 Processors 102 Main memory 103 Auxiliary storage 104 Communications Department 110 Control Unit

Claims

1. This involves acquiring the actual image captured by the imaging unit, The imaging unit acquires a calibration image, which is an image previously captured and serves as a reference image for the background at the present time. Based on the aforementioned real image, a model is generated to estimate the areas where shadows exist and to correct the color tone. Based on the calibration image and the model, an estimated background image is generated, which is an image in which a shadow corresponding to the actual image is added to the calibration image and the color tone is corrected, and which is a background image corresponding to the actual image. Obtaining the ideal background image, which is the initial image, Based on the actual image, the estimated background image, and the ideal background image, a background replacement image is generated, which is an image in which the background portion of the actual image is replaced with the corresponding portion of the ideal background image. Based on the aforementioned background replacement image, the presence or absence of foreign matter is determined, An information processing device comprising a control unit configured to perform the following:

2. The system includes a storage unit that stores a trained machine learning model that determines the presence or absence of foreign objects based on the aforementioned ideal background image. The control unit determines the presence or absence of the foreign object by inputting the background replacement image into the trained machine learning model. The information processing apparatus according to claim 1.

3. The imaging unit is located inside the vehicle and captures images of at least the entrance and exit of the vehicle. The information processing apparatus according to claim 1.

4. The control unit, From the brightness distribution of the aforementioned real image, candidate shadow regions are extracted, The estimated shadow regions are extracted by comparing them with the previously generated estimated background image from among the candidate shadow regions. Correction parameters for bringing the color tone of the calibration image closer to the color tone of the actual image are determined by comparing the actual image and the calibration image. By executing this, the aforementioned model is generated. The information processing apparatus according to claim 1.

Images