Driver monitoring simulation method and system

Abstract

The present disclosure provides a driver monitoring simulation method performed by at least one processor. The method comprises the steps of: acquiring at least one image captured by a camera installed in a vehicle; generating data related to a driver; generating a mixed image in which the driver is seated in the driver's seat of the vehicle, on the basis of the at least one image and the data related to the driver; and performing simulation for a driver monitoring model on the basis of the mixed image.

Description

Driver monitoring simulation method and system

[0001] The present disclosure relates to a method and system for driver monitoring simulation, and more particularly, to a method and system for generating a composite image of a driver seated in a vehicle based on an interior image of the vehicle and data associated with the driver, and performing a simulation for a driver monitoring model based thereon.

[0002] Recently, with the advancement of automotive technologies such as IT, electricity, and electronics, autonomous driving technology, which utilizes all of these technologies, is attracting attention. However, because autonomous driving technology controls the vehicle without driver intervention, it poses numerous social issues, including safety regulations. Addressing these issues requires extensive testing and verification.

[0003] However, physically replicating a real-world traffic environment to evaluate autonomous driving technology requires significant time and expense, and overcoming spatial constraints is difficult. In particular, testing and verifying driver monitoring technology installed in autonomous vehicles requires restricting the driver's activities while seated in the driver's seat. This exposes the driver to the risk of accidents, compromising driver safety.

[0004] The present disclosure provides a driver monitoring simulation method and device (system) to solve the above problems.

[0005] The present disclosure can be implemented in various ways, including as a method, a device (system), or a computer program stored on a readable storage medium.

[0006] According to one embodiment of the present disclosure, a driver monitoring simulation method may include a step of acquiring at least one image captured by a camera installed in a vehicle, a step of generating data associated with a driver, a step of generating a composite image of a driver seated in a driver's seat of a vehicle based on the at least one image and the data associated with the driver, and a step of performing a simulation for a driver monitoring model based on the composite image.

[0007] According to one embodiment of the present disclosure, the at least one image may include at least one of a visible light image, an infrared image, or a thermal image.

[0008] According to one embodiment of the present disclosure, data associated with the driver may include data associated with at least one of the driver's gender, age, appearance, or worn equipment.

[0009] According to one embodiment of the present disclosure, the driver in the mixed image may be a virtual driver modeled based on data associated with the driver.

[0010] According to one embodiment of the present disclosure, the data associated with the driver may be a video taken of an actual person located at a remote location.

[0011] According to one embodiment of the present disclosure, the step of generating a mixed image may include the step of generating a mixed image based on at least one image, data associated with a driver, and data associated with an environmental condition of the vehicle.

[0012] According to one embodiment of the present disclosure, the step of generating a mixed image may include the step of receiving scenario data associated with a change in behavior of a driver seated in a driver's seat of a vehicle, and the step of generating a mixed image of the driver acting in the vehicle according to the scenario data based on at least one image, data associated with the driver, and the scenario data.

[0013] According to one embodiment of the present disclosure, the method may further include a step of evaluating the driver monitoring model based on at least one of whether the simulation results for the driver monitoring model normally output a warning associated with the scenario data or a result of detecting the driver's behavior.

[0014] A computer-readable non-transitory recording medium having recorded thereon commands for executing a method according to one embodiment of the present disclosure on a computer may be provided.

[0015] According to one embodiment of the present disclosure, a device is provided. The device includes a communication module, a memory, and at least one processor connected to the memory and configured to execute at least one computer-readable program contained in the memory, wherein the at least one program may include instructions for acquiring at least one image captured by a camera installed in a vehicle, generating data associated with a driver, generating a composite image of a driver seated in a vehicle based on the at least one image and the data associated with the driver, and performing a simulation for a driver monitoring model based on the composite image.

[0016] According to some embodiments of the present disclosure, simulations of driver monitoring models can be performed based on mixed images. This allows driver monitoring models to be evaluated without targeting actual drivers. Furthermore, accidents that may occur during the evaluation process of driver monitoring models can be prevented.

[0017] According to some embodiments of the present disclosure, simulations of a driver monitoring model can be performed using a virtual driver model. Accordingly, the performance of the driver monitoring model can be safely and easily evaluated even outside of an actual driving environment.

[0018] According to some embodiments of the present disclosure, simulations of a driver monitoring model can be performed remotely using real people. This allows the performance of the driver monitoring model to be safely and easily evaluated even outside of an actual driving environment.

[0019] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs (referred to as “one skilled in the art”) from the description of the claims.

[0020] Embodiments of the present disclosure will be described below with reference to the accompanying drawings, wherein like reference numerals represent similar elements, but are not limited thereto.

[0021] FIG. 1 illustrates an example of a driver monitoring simulation method according to one embodiment of the present disclosure.

[0022] FIG. 2 is a schematic diagram showing a configuration in which an information processing system according to one embodiment of the present disclosure is connected to enable communication with a plurality of evaluation target vehicles.

[0023] FIG. 3 is a diagram showing the internal configuration of an information processing system according to one embodiment of the present disclosure.

[0024] FIG. 4 is a diagram illustrating an example of a simulation performed on a driver monitoring model according to one embodiment of the present disclosure.

[0025] FIG. 5 is a diagram illustrating an example of a simulation performed on a driver monitoring model according to one embodiment of the present disclosure.

[0026] FIG. 6 is a diagram illustrating examples of various scenarios associated with a driver monitoring model according to one embodiment of the present disclosure.

[0027] FIG. 7 is a flowchart illustrating an example of a method according to one embodiment of the present disclosure.

[0028] Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, specific descriptions of widely known functions or configurations will be omitted if they may unnecessarily obscure the gist of the present disclosure.

[0029] In the attached drawings, identical or corresponding components are assigned the same reference numerals. Furthermore, in the description of the embodiments below, duplicate descriptions of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0030] The advantages and features of the disclosed embodiments, and methods for achieving them, will become clearer with reference to the embodiments described below, along with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided solely to ensure the completeness of the disclosure and to fully inform those skilled in the art of the scope of the invention.

[0031] The terms used in this specification will be briefly explained, followed by a detailed description of the disclosed embodiments. The terms used in this specification have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of engineers working in the relevant field, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0032] In this specification, singular expressions include plural expressions unless the context clearly indicates otherwise. Furthermore, plural expressions include singular expressions unless the context clearly indicates otherwise. When a part of the specification is said to include a component, this does not exclude other components, but rather implies that other components may be included, unless otherwise specifically stated.

[0033] Also, the term 'module' or 'part' used in the specification means a software or hardware component, and the 'module' or 'part' performs certain roles. However, the 'module' or 'part' is not limited to software or hardware. The 'module' or 'part' may be configured to reside on an addressable storage medium and may be configured to execute one or more processors. Thus, as an example, the 'module' or 'part' may include at least one of components such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. The functionality provided within the components and 'modules' or 'parts' may be combined into a smaller number of components and 'modules' or 'parts', or further separated into additional components and 'modules' or 'parts'.

[0034] According to one embodiment of the present disclosure, a 'module' or 'unit' may be implemented as a processor and a memory. 'Processor' should be broadly construed to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and the like. In some circumstances, a 'processor' may also refer to an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field-programmable gate array (FPGA), and the like. A 'processor' may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or any other such combination of configurations. In addition, 'memory' should be broadly construed to include any electronic component capable of storing electronic information. 'Memory' may refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, etc. Memory is said to be in electronic communication with the processor if the processor can read information from, and / or write information to, the memory. Memory integrated in a processor is in electronic communication with the processor.

[0035] In the present disclosure, the "system" may include, but is not limited to, at least one of a server device and a cloud device. For example, the system may be comprised of one or more server devices. As another example, the system may be comprised of one or more cloud devices. As yet another example, the system may be configured and operated by a combination of a server device and a cloud device.

[0036] In the present disclosure, 'display' may refer to any display device associated with a computing device, for example, any display device capable of displaying any information / data controlled by or provided from the computing device.

[0037] In the present disclosure, 'each of the plurality of As' or 'each of the plurality of As' may refer to each of all components included in the plurality of As, or may refer to each of some components included in the plurality of As.

[0038] In this disclosure, the term "driver monitoring model" refers to a model that detects specific driver behavior while driving a vehicle, and may include a driver monitoring system (DMS). Here, the driver monitoring system may refer to a system that monitors the driver using sensors such as cameras that monitor the driver, a processing device that determines the driver's status, and the like.

[0039] FIG. 1 illustrates an example of a driver monitoring simulation method according to one embodiment of the present disclosure. As illustrated, an information processing system (120) for driver monitoring simulation and a vehicle to be evaluated (110) can communicate with each other via a network. Specifically, the vehicle to be evaluated (110) can transmit at least one interior image (112) captured by a camera installed within the vehicle to be evaluated (110) to the information processing system (120). In addition, the information processing system (120) can transmit a composite image (122) to the vehicle to be evaluated (110). Here, the vehicle to be evaluated (110) may be an actual vehicle that performs driver monitoring based on the composite image (122).

[0040] In one embodiment, the information processing system (120) may receive an interior image (112) from a vehicle to be evaluated (110). Here, the interior image (112) may be an image captured by a camera installed inside the vehicle to be evaluated (110). Alternatively, the interior image (112) may be a composite image of the interior of the vehicle to be evaluated (110). In addition, the interior image (112) may include a visible light image, an infrared image, a thermal image, etc. The interior image (112) may be a single image or an image sequence.

[0041] In one embodiment, the information processing system (120) may receive or generate data associated with the driver. The data associated with the driver may include data related to the driver's gender, age, appearance, and equipment worn by the driver. For example, the data associated with the driver may include data regarding the driver's appearance, such as skin color, beard, and hair color, and data regarding equipment worn by the driver, such as glasses, sunglasses, or an eye mask.

[0042] In one embodiment, the information processing system (120) may generate a composite image (122) of a driver seated in the driver's seat of a vehicle to be evaluated (110) based on an interior image (112) and data associated with the driver. The driver in the composite image (122) may be a virtual driver modeled based on the data associated with the driver. Alternatively, the driver in the composite image (122) may be a driver corresponding to a real person located at a remote location. An example of generating the composite image (122) is described in detail below with reference to FIGS. 4 and 5 .

[0043] In one embodiment, the information processing system (120) may perform a simulation for a driver monitoring model based on a composite image (122). In this case, the information processing system (120) may transmit the composite image (122) to the vehicle to be evaluated (110). Accordingly, the vehicle to be evaluated (110) inputs the composite image (122) into the driver monitoring model, thereby enabling the information processing system (120) to perform a simulation for the driver monitoring model. Alternatively, the information processing system (120) may perform a simulation for the driver monitoring model by inputting the composite image (122) into the driver monitoring model within the information processing system (120) without transmitting it to the vehicle to be evaluated (110).

[0044] This configuration allows simulations of driver monitoring models based on mixed images. This allows driver monitoring models to be evaluated without targeting actual drivers. Furthermore, accidents that may occur during the evaluation process can be prevented.

[0045] FIG. 2 is a schematic diagram showing a configuration in which an information processing system (230) according to one embodiment of the present disclosure is connected to enable communication with a plurality of evaluation target vehicles (210_1, 210_2, 210_3). In FIG. 2, the evaluation target vehicles (210_1, 210_2, 210_3) are illustrated as communicating with the information processing system (230) via a network (220), but this is not limited thereto, and a communication unit mounted on the evaluation target vehicles (210_1, 210_2, 210_3) may communicate with other evaluation target vehicles and external devices. Here, the communication may include V2X (Vehicle to Everything) communication.

[0046] In one embodiment, the information processing system (230) may include one or more server devices and / or databases capable of storing, providing, and executing computer-executable programs (e.g., downloadable applications) and data related to the virtual driver and vehicle interior environment reproduction, and driver monitoring simulation, or one or more distributed computing devices and / or distributed databases based on cloud computing services. The information processing system (230) may provide information corresponding to signals input through applications (e.g., driver monitoring simulation-related applications, etc.) or perform corresponding processing. For example, the information processing system (230) may transmit a composite image of a virtual driver seated in a driver's seat of a plurality of evaluation target vehicles (210_1, 210_2, 210_3) through any application related to driver monitoring simulation.

[0047] The information processing system (230) can communicate with a plurality of evaluation target vehicles (210_1, 210_2, 210_3) via a network (220). The network (220) can be configured to enable communication between a plurality of evaluation target vehicles (210_1, 210_2, 210_3) and the information processing system (230). Depending on the installation environment, the network (220) can be configured as a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device, and RS-serial communication, a wireless network such as a mobile communication network, WLAN (Wireless LAN), Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and may include not only a communication method utilizing a communication network (e.g., a mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.) that the network (220) may include, but also short-range wireless communication between the vehicles to be evaluated (210_1, 210_2, 210_3).

[0048] In FIG. 2, the vehicles to be evaluated (210_1, 210_2, 210_3) may be vehicles equipped with a driver monitoring model. In addition, FIG. 2 illustrates three vehicles to be evaluated (210_1, 210_2, 210_3) communicating with an information processing system (230) via a network (220), but this is not limited thereto, and any number of vehicles to be evaluated may be configured to communicate with an information processing system (230) via a network (220).

[0049] In one embodiment, the vehicle being evaluated (210_1, 210_2, 210_3) may receive a composite image of the driver seated in the vehicle and scenario data related to changes in the driver's behavior from the information processing system (230) via a network (220). Furthermore, the results of the simulation for the driver monitoring model may be transmitted to the information processing system (230).

[0050] FIG. 3 is a diagram illustrating the internal configuration of an information processing system (230) according to one embodiment of the present disclosure. As illustrated, the information processing system (230) may include a driver model generation unit (310), a mixed image generation unit (320), a simulation execution unit (330), and a communication unit (340). Here, the information processing system (230) may be configured with at least one processor.

[0051] The driver model generation unit (310) can generate a driver model based on data associated with the driver. The driver-related data may include data related to various factors such as gender, age, appearance, and equipment worn. For example, the driver model generation unit (310) may generate a bearded man in his 40s as the driver model. As another example, the driver model generation unit (310) may generate a bespectacled woman in her 50s as the driver model. Data associated with the driver model generated by the driver model generation unit (310) may be transmitted to the mixed image generation unit (320) or stored in a database.

[0052] In one embodiment, the driver model generation unit (310) may generate a virtual driver model based on data associated with the driver stored in a database. Additionally or alternatively, the driver model generation unit (310) may generate a driver model based on a real person located at a remote location. Specifically, the driver model generation unit (310) may obtain an image of a real person captured at a remote location from an external device via the communication unit (340) and generate a driver model corresponding to the real person in the image. In this case, the behavior of the driver model may be identical to the behavior of the real person captured at a remote location. Additionally or alternatively, the driver model generation unit (310) may use an image of a real person captured at a remote location.

[0053] The mixed image generation unit (320) can generate a mixed image of a driver seated in the vehicle based on data associated with the vehicle's interior image and a driver model. Here, the interior image may be an image captured by a camera installed in the vehicle and received from the vehicle via the communication unit (340). Alternatively, the interior image may be an interior image of the vehicle synthesized by the mixed image generation unit (320).

[0054] In one embodiment, the mixed image generation unit (320) may generate a mixed image based on an interior image of the vehicle, data associated with the driver, and data associated with environmental conditions. The environmental conditions may include the interior / exterior environment of the vehicle, such as backlighting, humidity, and temperature. For example, the mixed image generation unit (320) may generate a mixed image that includes a driver in backlighting. As another example, the mixed image generation unit (320) may generate a mixed image that includes a fogged-up interior camera due to humidity, a mixed image that includes a driver wearing fogged-up glasses due to humidity, and the like.

[0055] In one embodiment, the mixed image generation unit (320) may generate a mixed image based on an interior image of the vehicle, data associated with the driver, and scenario data. Here, the scenario data may be associated with behavioral changes of a driver seated in the vehicle. For example, the scenario data may be data generated based on various scenarios for evaluating a driver monitoring model. Accordingly, the mixed image generation unit (320) may generate a mixed image of a driver acting in the vehicle according to the scenario data.

[0056] The simulation unit (330) can perform a simulation of a driver monitoring model based on a mixed image. Here, the driver monitoring model may refer to a model that detects specific driver behaviors while driving. Furthermore, the driver monitoring model may be installed in the vehicle being evaluated. Alternatively, the driver monitoring model may be included in the simulation unit (330) of the information processing system.

[0057] In one embodiment, the simulation performing unit (330) may perform a simulation on the driver monitoring model by inputting the mixed image generated by the mixed image generating unit (320) into the driver monitoring model. In this case, as a result of the simulation, a warning may be output for a driver who performs a specific action based on the scenario data. The simulation performing unit (330) may evaluate the driver monitoring model based on whether such warnings are output normally for various scenarios. An example of the simulation results is described in detail below with reference to FIG. 6.

[0058] The communication unit (340) can transmit and receive data with the vehicle being evaluated via a network. For example, the communication unit (340) can receive an interior image from the vehicle and transmit a composite image to the vehicle. Furthermore, the communication unit (340) can transmit and receive data with an external device via a network. For example, the communication unit (340) can receive images of a real person located remotely, scenario data, etc. from an external device. Furthermore, the communication unit (340) can transmit evaluation results for the driver monitoring model to a user terminal.

[0059] The internal configuration of the information processing system (230) illustrated in FIG. 3 is merely exemplary, and in some embodiments, additional configurations other than the illustrated internal configuration may be included, some configurations may be omitted, and some processes may be performed by other configurations or external systems. Furthermore, although the internal configurations of the information processing system (230) in FIG. 3 are described by dividing them by function, this does not necessarily mean that the internal configurations are physically distinct.

[0060] FIG. 4 is a diagram illustrating an example of a simulation performed on a driver monitoring model according to one embodiment of the present disclosure. In one embodiment, at least one processor of an information processing system (e.g., 230 of FIG. 2 ) may acquire an interior image of a vehicle (S410). Here, the interior image of the vehicle may be an interior image of the vehicle captured by a camera installed within the vehicle. Alternatively, the interior image of the vehicle may be an interior image of the vehicle synthesized by the processor.

[0061] Thereafter, the processor may generate a driver model (S420). Specifically, the processor may generate the driver model based on virtual driver data (462) stored in the database (460). Here, the virtual driver data (462) may be modeled based on data associated with the driver. Accordingly, the generated driver model may represent virtual drivers with various genders, ages, appearances, and equipment.

[0062] Thereafter, the processor can generate a composite image based on the vehicle interior image and the driver model (S430). Specifically, the processor can synthesize the vehicle interior image and the driver model to generate a composite image of a driver seated in the vehicle. Furthermore, the processor can further generate the composite image based on scenario data (464) stored in the database (460). In this case, the processor can generate a composite image of a driver seated in the vehicle acting according to the scenario data (464).

[0063] Thereafter, the processor can inject the generated mixed image into the driver monitoring model (S440). Here, the driver monitoring model can be installed in the vehicle. Alternatively, the driver monitoring model can be included in the information processing system. Accordingly, the processor can perform a simulation on the driver monitoring model based on the mixed image (S450).

[0064] This configuration allows simulations of the driver monitoring model using a virtual driver model. This allows the performance of the driver monitoring model to be safely and easily evaluated even outside of actual driving conditions.

[0065] FIG. 5 is a diagram illustrating an example of a simulation performed on a driver monitoring model according to one embodiment of the present disclosure. In one embodiment, at least one processor of an information processing system (e.g., 230 of FIG. 2 ) may acquire an interior image of a vehicle (S510). Here, the interior image of the vehicle may be an interior image of the vehicle captured by a camera installed within the vehicle. Alternatively, the interior image of the vehicle may be an interior image of the vehicle synthesized by the processor.

[0066] Thereafter, the processor can acquire a real-life image of a remote person (S520). Here, the processor can acquire the real-life image of a remote person from an external device. For example, the external device may include, but is not limited to, a Driver In the Loop Simulator (DILS), a virtual driver's seat, etc. Alternatively, the processor can acquire a real-life image of a remote person pre-stored from a database.

[0067] Thereafter, the processor can generate a composite image based on the vehicle's interior image and a remotely located real-person image (S530). Specifically, the processor can generate a driver model based on the remotely located real-person image. In this case, the driver model can synthesize the image with the vehicle's interior image to generate a composite image of the driver seated in the vehicle.

[0068] Thereafter, the processor can inject the generated mixed image into the driver monitoring model (S540). Here, the driver monitoring model can be installed in the vehicle. Alternatively, the driver monitoring model can be included in the information processing system. Accordingly, the processor can perform a simulation on the driver monitoring model based on the mixed image (S550).

[0069] In one embodiment, the driver's behavior within the composite video can be dynamically changed. For example, in this case, the processor can interface with an external device to generate a composite video that reflects the behavior of a real person and feed the generated composite video into the driver monitoring model in real time. This allows the driver's behavior within the composite video to be modified based on data from the real person acting in real time. A remote person can then act according to a given scenario.

[0070] This configuration allows simulations of driver monitoring models to be performed remotely using real people. This allows the performance of driver monitoring models to be safely and easily evaluated even outside of actual driving environments.

[0071] FIG. 6 is a diagram illustrating various scenarios (600) associated with a driver monitoring model according to one embodiment of the present disclosure. In one embodiment, the driver monitoring model can extract a driver within a moving vehicle from an input image or video and detect the driver's behavior. Furthermore, the driver monitoring model can output a warning based on the detected driver's behavior. For example, a composite image can be generated according to various scenarios (600). Each scenario can define the driver's behavior for a given period of time (e.g., 30 seconds, etc.) and the driver monitoring model's output ground truth for that scenario. Here, the output ground truth can include situations in which a warning is normally output for the driver's behavior, situations in which a specific driver behavior is normally output, etc. For example, if the driver satisfies two or more of the following conditions: no head / body movement for a predetermined period of time (e.g., 30 seconds, etc.), the driver monitoring model may be required to output a warning.

[0072] In the illustrated example, the first scenario may be a scenario in which the driver has no head / body movement, has his / her eyes closed, and does not blink for a given period of time. For the first scenario, a normal output may be for the driver monitoring model to generate an alert. The second scenario may be a scenario in which the driver has no head / body movement, has his / her eyes open, and blinks for a given period of time. For the second scenario, a normal output may be for the driver monitoring model to not generate an alert. The third scenario may be a scenario in which the driver has head / body movement, has his / her eyes closed, and does not blink for a given period of time. For the third scenario, a normal output may be for the driver monitoring model to not generate an alert.

[0073] In one embodiment, a driver monitoring model can be evaluated by performing a simulation based on the composite images associated with each scenario. In this case, the driver monitoring model can be evaluated based on whether the driver monitoring model properly issues warnings as a result of the simulation. For example, if the driver monitoring model does not issue a warning for a scenario that should issue a warning, a negative evaluation of the driver monitoring model can be generated. Furthermore, if the driver monitoring model issues a warning for a scenario that should not issue a warning, a negative evaluation of the driver monitoring model can be generated. Conversely, if the driver monitoring model issues and does not issue warnings for scenarios that should issue a warning and scenarios that should not issue a warning, respectively, a positive evaluation of the driver monitoring model can be generated.

[0074] In one embodiment, a driver monitoring model may be evaluated based on the results of the simulation in which the driver's behavior is detected. For example, if the driver's behavior (e.g., head / body movement, eye closure, eye blinking, etc.) is not detected in a specific scenario, a negative evaluation of the driver monitoring model may be generated. Additionally, the driver monitoring model may be evaluated based on whether it properly issues warnings. For example, a more negative evaluation may be generated if the driver monitoring model issues a warning despite not detecting the driver's behavior than if the driver monitoring model fails to detect the driver's behavior and therefore does not issue a warning.

[0075] While some scenarios are illustrated in Figure 6, this is not a limitation and various combinations of scenarios may be included. Furthermore, whether a warning is output normally may vary depending on the criteria and systems used to evaluate the performance of the driver monitoring system, even for the same input.

[0076] FIG. 7 is a flowchart illustrating an example of a method (700) according to one embodiment of the present disclosure. In one embodiment, the method (700) may be performed by at least one processor. The method (700) may begin with the processor acquiring at least one image captured by a camera installed in a vehicle (S710). Here, the at least one image may include at least one of a visible light image, an infrared image, or a thermal image.

[0077] Afterwards, the processor can generate data associated with the driver (S720). The data associated with the driver may include data related to at least one of the driver's gender, age, appearance, or equipment worn. Furthermore, the data associated with the driver may be a video captured of an actual person located remotely.

[0078] Thereafter, the processor may generate a composite image of a driver seated in the vehicle based on at least one image and data associated with the driver (S730). Here, the driver in the composite image may be a virtual driver modeled based on the data associated with the driver. Additionally, the processor may generate the composite image based on at least one image, data associated with the driver, and data associated with environmental conditions of the vehicle. Thereafter, the processor may perform a simulation for a driver monitoring model based on the composite image (S740).

[0079] In one embodiment, the processor may receive scenario data associated with changes in the behavior of a driver seated in the driver's seat of a vehicle. Furthermore, based on at least one image, data associated with the driver, and the scenario data, the processor may generate a composite image of the driver acting in the vehicle according to the scenario data. In this case, the processor may evaluate the driver monitoring model based on at least one of the following: whether the simulation results for the driver monitoring model normally output warnings associated with the scenario data, or the results of detecting the driver's behavior.

[0080] The above-described method may be provided as a computer program stored on a computer-readable recording medium for execution on a computer. The medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program instructions, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0081] The methods, operations, or techniques of the present disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will appreciate that the various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software will depend on the particular application and the design requirements imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementations should not be construed as departing from the scope of the present disclosure.

[0082] In a hardware implementation, the processing units used to perform the techniques may be implemented within one or more ASICs, DSPs, GPUs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, a computer, or a combination thereof.

[0083] Accordingly, the various exemplary logical blocks, modules, and circuits described in connection with the present disclosure may be implemented or performed by any combination of a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or those designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0084] In a firmware and / or software implementation, the techniques may be implemented as instructions stored on a computer-readable medium, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, a compact disc (CD), a magnetic or optical data storage device, etc. The instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described herein.

[0085] When implemented in software, the techniques may be stored on or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. In addition, any connection is suitably made to a computer-readable medium.

[0086] For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of media. Disk and disc, as used herein, includes compact discs, laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks usually reproduce data magnetically, whereas discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0087] A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. Alternatively, the processor and the storage medium may reside as discrete components in the user terminal.

[0088] While the embodiments described above have been described as utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the present disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or distributed computing environment. Furthermore, aspects of the present disclosure may be implemented in multiple processing chips or devices, and storage may be similarly affected across multiple devices. Such devices may include personal computers, network servers, and portable devices.

[0089] While the present disclosure has been described in connection with certain embodiments herein, various modifications and variations may be made without departing from the scope of the present disclosure, which would be apparent to those skilled in the art. Furthermore, such modifications and variations are intended to fall within the scope of the claims appended to this specification.

Claims

1. A driver monitoring simulation method performed by at least one processor, A step of acquiring at least one image captured by a camera installed in a vehicle; Step of generating data associated with the driver; A step of generating a composite image of the driver seated in the driver's seat of the vehicle based on at least one image and data associated with the driver; and Step of performing a simulation for a driver monitoring model based on the above mixed image A driver monitoring simulation method comprising:

2. In paragraph 1, A driver monitoring simulation method, wherein said at least one image comprises at least one of a visible light image, an infrared image or a thermal image.

3. In paragraph 1, A driver monitoring simulation method, wherein the data associated with the driver includes data associated with at least one of the driver's gender, age, appearance, or worn tools.

4. In paragraph 1, A driver monitoring simulation method, wherein the driver in the above mixed image is a virtual driver modeled based on data associated with the driver.

5. In paragraph 1, A driver monitoring simulation method, wherein the data associated with the above driver is a video captured of an actual person located at a remote location.

6. In paragraph 1, The step of generating the above mixed image is: A step of generating the mixed image based on at least one image, data associated with the driver, and data associated with environmental conditions of the vehicle. A driver monitoring simulation method comprising:

7. In paragraph 1, The step of generating the above mixed image is: A step of receiving scenario data related to behavioral changes of a driver seated in the driver's seat of the above vehicle; and A step of generating a composite image of the driver acting in accordance with the scenario data in the vehicle based on at least one image, data associated with the driver, and the scenario data. A driver monitoring simulation method comprising:

8. In paragraph 7, A step of evaluating the driver monitoring model based on at least one of whether the driver monitoring model normally outputs a warning associated with the scenario data as a result of the simulation for the driver monitoring model or whether the driver's behavior is detected. A driver monitoring simulation method further comprising:

9. A computer-readable, non-transitory recording medium recording commands for executing the method according to Article 1 on a computer.

10. As a device, Communication module; memory; and At least one processor coupled to said memory and configured to execute at least one computer-readable program contained in said memory, At least one of the above programs, Acquire at least one image captured by a camera installed in the vehicle, Generate data associated with the driver, Based on at least one image and data associated with the driver, a composite image is generated in which the driver is seated in the driver's seat of the vehicle, A device comprising commands for performing a simulation for a driver monitoring model based on the above mixed image.

Images