Method and system for generating mixed reality images related to driving environment

Abstract

The present disclosure relates to a method for generating mixed reality images related to a driving environment executed by at least one processor. [Solution] The mixed reality image generating method includes the steps of acquiring an actual image related to a driving environment, generating a first virtual image including objects of the driving environment, and generating a mixed reality image including objects of the driving environment based on the actual image and the first virtual image.

Description

[Technical field]

[0001] The present disclosure relates to a method and system for generating a mixed reality image related to a driving environment, and more particularly, to a method and system for generating a mixed reality image including objects of the driving environment based on an actual image related to the driving environment and a virtual image including the objects of the driving environment. [Background technology]

[0002] In recent years, with the advent of the Fourth Industrial Revolution, autonomous vehicle-related technologies have been attracting attention as the automobile of the future. Autonomous driving technology, which applies various cutting-edge technologies including IT technology, is gaining in prominence as a new growth engine for the automobile industry.

[0003] Meanwhile, autonomous vehicles generally travel to a given destination by recognizing the surrounding environment, determining the driving situation, and controlling the vehicle without the intervention of the driver, which leads to various social issues such as safety regulations, driving regulations, etc. Therefore, continuous performance testing is required to solve these social issues and commercialize autonomous driving technology.

[0004] However, as vehicle components and software become increasingly diverse and complex, there are problems such as space constraints, time and expense involved in actually creating and testing a driving environment to evaluate an autonomous vehicle. Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure provides a method for generating a mixed reality image related to a driving environment, a computer-readable non-transitory recording medium having instructions recorded thereon, and a system (apparatus) for solving the above problems. [Means for solving the problem]

[0006] The present disclosure can be implemented in numerous ways, including as a method, a system (apparatus), or a computer program stored on a computer readable storage medium.

[0007] According to one embodiment of the present disclosure, a method for generating a mixed reality image related to a driving environment, executed by at least one processor, includes a step of acquiring an actual image related to the driving environment, a step of generating a first virtual image including an object of the driving environment, and a step of generating a mixed reality image including the object of the driving environment based on the actual image and the first virtual image.

[0008] According to one embodiment of the present disclosure, the first virtual image includes an image generated from a simulation model that realizes a driving environment.

[0009] According to one embodiment of the present disclosure, the method further includes a step of generating a second virtual image related to the driving environment, and the step of generating a mixed reality image based on the real image and the first virtual image includes a step of generating a mixed reality image based on the real image, the first virtual image, and the second virtual image.

[0010] According to an embodiment of the present disclosure, the second virtual image includes a semantic image generated from a simulation model that realizes the driving environment.

[0011] According to an embodiment of the present disclosure, acquiring an actual image related to a driving environment includes acquiring the actual image using a first camera mounted on the vehicle.

[0012] According to one embodiment of the present disclosure, the first virtual image is generated from data acquired using a second camera mounted on a vehicle realized in the simulation model as the same vehicle model as the vehicle, and the second camera is a virtual camera realized in the simulation model as a camera having the same external and internal parameters as the first camera.

[0013] According to one embodiment of the present disclosure, the second virtual image is generated from data acquired using a third camera mounted on a vehicle realized in the simulation model as the same vehicle model as the vehicle, and the third camera is a virtual camera realized in the simulation model as a camera having the same external and internal parameters as the first camera.

[0014] According to one embodiment of the present disclosure, the step of generating a mixed reality image based on a real image and a first virtual image includes the steps of extracting a portion corresponding to an object from the first virtual image, and generating the mixed reality image based on the portion corresponding to the object and the real image.

[0015] According to an embodiment of the present disclosure, extracting a portion corresponding to the object includes extracting the portion corresponding to the object using a second virtual image.

[0016] According to one embodiment of the present disclosure, the step of extracting a portion corresponding to the object based on the second virtual image includes a step of filtering the second virtual image to generate a mask image for the object, and a step of extracting the portion corresponding to the object from the first virtual image using the mask image.

[0017] According to an embodiment of the present disclosure, generating a mixed reality image based on a portion corresponding to an object and the real image includes removing a portion of the real image corresponding to the portion corresponding to the object.

[0018] According to one embodiment of the present disclosure, the step of removing the portion of the actual image corresponding to the object includes the step of removing the portion of the actual image corresponding to the object using a second virtual image.

[0019] According to one embodiment of the present disclosure, the step of removing a portion of the actual image corresponding to the object using the second virtual image includes the steps of generating a mask inverse image from the second virtual image, and removing a portion of the actual image corresponding to the object using the mask inverse image.

[0020] According to an embodiment of the present disclosure, generating a mask inverse image includes generating a mask image for the object from the second virtual image, and inverting the mask image to generate the mask inverse image.

[0021] A non-transitory computer readable recording medium having instructions recorded thereon for executing the above-described method on a computer according to one embodiment of the present disclosure is provided.

[0022] An information processing system according to one embodiment of the present disclosure includes a communication module, a memory, and at least one processor coupled to the memory and configured to execute at least one computer-readable program contained in the memory, the at least one program including instructions for acquiring an actual image related to a driving environment, generating a first virtual image including objects of the driving environment, and generating a mixed reality image including the objects of the driving environment based on the actual image and the first virtual image. Effect of the Invention

[0023] According to one embodiment of the present disclosure, by integrating virtual objects into a real driving environment, it is possible to realize various test situations and test an autonomous vehicle without being restricted by spatial constraints.

[0024] According to one embodiment of the present disclosure, by visualizing and providing virtual objects in an actual driving environment, a tester on board an autonomous vehicle can visually recognize a virtual test situation in the actual driving environment, enabling smooth testing in various driving environments.

[0025] According to one embodiment of the present disclosure, instead of actually placing physical objects for testing an autonomous vehicle, images of virtual objects projected into the driving environment are provided during the autonomous vehicle testing process, thereby saving time and costs required for the testing.

[0026] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure belongs (referred to as a "person of ordinary skill in the art") from the description in the claims.

[0027] Non-limiting embodiments of the present disclosure will be described with reference to the accompanying drawings, in which like reference numerals indicate like elements, as follows: [Brief description of the drawings]

[0028] [Figure 1] FIG. 1 is an exemplary diagram in which a mixed reality image generation method related to a driving environment is used, according to one embodiment of the present disclosure. [Diagram 2] FIG. 2 is a block diagram illustrating an internal configuration of an information processing system according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 is a diagram illustrating an internal configuration of a processor in an information processing system according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is an exemplary diagram illustrating a mixed reality image generated from a real image and a virtual image according to one embodiment of the present disclosure. [Diagram 5] FIG. 5 is an exemplary diagram illustrating a process virtual image generated from a first virtual image and a second virtual image according to an embodiment of the present disclosure. [Figure 6] FIG. 6 is an exemplary diagram illustrating a process of generating an actual image from an actual image and a second virtual image according to an embodiment of the present disclosure. [Figure 7] FIG. 7 is a diagram illustrating an example of a virtual image according to an embodiment of the present disclosure. [Figure 8]FIG. 8 is a diagram illustrating an example of an actual image according to an embodiment of the present disclosure. [Figure 9] FIG. 9 is a diagram illustrating an example of a mixed reality image according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a flowchart illustrating a method for generating mixed reality images related to a driving environment according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Hereinafter, specific contents for implementing the present disclosure will be described in detail with reference to the accompanying drawings. Note that in the following description, detailed descriptions of well-known functions and configurations will be omitted if there is a risk of unnecessarily obscuring the gist of the present disclosure.

[0030] In the accompanying drawings, the same or corresponding components are given the same reference numerals. In addition, in the following description of the embodiments, duplicated descriptions of the same or corresponding components may be omitted. However, even if the description of a component is omitted, it is not intended that such a component is not included in a certain embodiment.

[0031] The advantages and features of the disclosed embodiments, and the methods for achieving the same, will become apparent from the following detailed description taken in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, and may be embodied in various different forms, and the embodiments are provided solely so that this disclosure will be complete and complete, and will fully convey the scope of the invention to those skilled in the art.

[0032] The terms used in this specification are briefly explained, and the disclosed embodiments are specifically explained. The terms used in this specification are selected from currently widely used general terms while taking into consideration the functions in this disclosure, but this may change depending on the intentions or precedents of engineers engaged in related fields, the emergence of new technologies, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in such cases, the meanings are described in detail in the description of the relevant invention. Therefore, the terms used in this disclosure should be defined based on the meanings that the terms have and the contents of the entire disclosure, rather than simply the names of the terms.

[0033] In this specification, the singular expression includes the plural expression unless the context clearly specifies otherwise. In addition, the plural expression includes the singular expression unless the context clearly specifies otherwise. When a part in the entire specification includes a certain element, it means that it may further include other elements, not excluding other elements, unless otherwise specified.

[0034] Furthermore, the term "module" or "unit" as used herein means a software or hardware component, and the "module" or "unit" performs some function. However, the "module" or "unit" is not limited to software or hardware. The "module" or "unit" may be configured to be in an addressable storage medium and may also be configured to implement one or more processors. Thus, by way of example, a "module" or "unit" may include components such as software components, object-oriented software components, class components, and task components, and at least one of a process, a function, an attribute, a procedure, a subroutine, a segment of program code, a driver, firmware, microcode, a circuit, data, a database, a data structure, a table, an array, or a variable. The components and "modules" or "units" may be combined into a smaller number of components and "modules" or "units" or the functions provided therein may be further separated into additional components and "modules" or "units".

[0035] According to an embodiment of the present disclosure, a "module" or a "unit" may be implemented with a processor and a memory. A "processor" should be broadly interpreted to include a general purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and the like. In some environments, a "processor" may refer to an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), and the like. A "processor" may refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or any other such combination. Additionally, a "memory" should be broadly interpreted to include any electronic component capable of storing electronic information. "Memory" may refer to various types of processor-readable media, such as random access memory (RAM), read only memory (ROM), non-volatile random access memory (NVRAM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage devices, registers, etc. Memory is said to be in electronic communication with a processor if the processor can read information from and / or write information to the memory. Memory that is integrated into a processor is in electronic communication with the processor.

[0036] In this disclosure, "mixed reality (MR)" may refer to the combination of the virtual world and the real world to create new environments, visualizations, and other new information, i.e., technology that splices virtual reality into the real world, allowing real physical objects to interact with virtual objects.

[0037] In this disclosure, "real world imagery" may refer to imagery of embodied physical objects or environments, such as images of vehicles, roadways, and surrounding objects that exist in reality.

[0038] In this disclosure, "virtual image" may refer to an image of a virtually realized object, rather than an image of a tangible physical object, for example, an image of an object in a virtual reality environment.

[0039] In this disclosure, a "semantic image" may refer to an image to which a semantic segmentation technique has been applied, where the semantic segmentation technique may refer to a technique that divides objects in an image into semantic units (e.g., objects) and predicts the class to which each pixel in the image belongs.

[0040] In this disclosure, a "mask image" may refer to an image used to mask a particular portion of an image, where masking may refer to the act of separating an area to which a particular effect is to be applied from the remaining area when applying a particular effect to an image.

[0041] 1 is an exemplary diagram showing an example of a mixed reality image generation method related to a driving environment according to an embodiment of the present disclosure. As shown in FIG. 1, the mixed reality image generation system includes a virtual image generation unit 120, a real image processing unit 150, a virtual image processing unit 160, and a mixed reality image generation unit 170, and a mixed reality image 190 can be generated from a real image 110 and virtual images 130 and 140.

[0042] According to an embodiment, the real image 110 may be generated from the first camera 100. In this case, the first camera 100 may be a camera mounted on a vehicle. The real image 110 may be an image related to the driving environment. For example, the first camera 100 mounted on a vehicle may generate the real image 110 capturing the driving road and surrounding objects. The generated real image 110 may then be transmitted to the real image processing unit 150.

[0043] According to an embodiment, the virtual image generator 120 may generate virtual images 130 and 140 and transmit them to the real image processor 150 and the virtual image processor 160. In this case, the first virtual image 130 may include an image generated from a simulation model that realizes a driving environment. The first virtual image 130 may be an image including objects of the driving environment (e.g., traffic cones located on a road, other vehicles, etc.). The second virtual image 140 may include a semantic image generated from a simulation model that realizes a driving environment.

[0044] According to an embodiment, the first virtual image 130 may be generated from the second camera 122. Also, the second virtual image 140 may be generated from the third camera 124. The second camera 122 may be a virtual camera mounted on a vehicle realized in a simulation model as the same vehicle model as the vehicle to which the first camera 100 is mounted. Similarly, the third camera 124 may also be a virtual camera mounted on a vehicle realized in a simulation model as the same vehicle model as the vehicle to which the first camera 100 is mounted. Also, the second camera 122 and the third camera 124 may be virtual cameras realized in a simulation model as cameras with external parameters (e.g., camera orientation, etc.) and internal parameters (e.g., focal length, optical center, etc.) set to be the same as those of the first camera.

[0045] According to an embodiment, a mixed reality image 190 including objects in a driving environment may be generated based on the real image 110 and the first virtual image 130. To this end, a real image 152 processed from the real image 110 may be generated by the real image editor 150. Also, a virtual image 162 processed from the first virtual image 130 may be generated by the virtual image editor 160. The processed real image 152 and the processed virtual image 162 may be blended by the mixed reality image generator 170 to generate the mixed reality image 190 (180). This will be described in detail below with reference to FIGS. 4 to 6.

[0046] According to an embodiment, the second virtual image 140 may be used to generate the processed real image 152 and the processed virtual image 162. For example, the second virtual image 140 may be used to extract a portion corresponding to an object from the first virtual image 130. Also, the second virtual image 140 may be used to remove an image of a portion where an object is located in the real image 110. This will be described in detail below with reference to FIGS. 5 and 6.

[0047] The above-described configuration can reduce the time and cost required for testing an autonomous vehicle by providing images of virtual objects projected into the driving environment during the autonomous vehicle testing process, rather than actually placing physical objects for the testing.

[0048] 2 is a block diagram showing an internal configuration of an information processing system 200 according to an embodiment of the present disclosure. The information processing system 200 corresponds to the mixed reality image generation system including the virtual image generation unit 120, the real image manipulation unit 150, the virtual image manipulation unit 160, and the mixed reality image generation unit 170 in FIG. 1, and may include a memory 210, a processor 220, a communication module 230, and an input / output interface 240. The information processing system 200 may be configured to communicate information and / or data with an external system via a network using the communication module 230.

[0049] The memory 210 may include any non-transitory computer-readable recording medium. According to an embodiment, the memory 210 may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. As another example, a permanent mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be included in the information processing system 200 as a separate permanent storage device distinct from the memory. In addition, the memory 210 may store an operating system and at least one program code (e.g., a code for generating a mixed reality image provided and operated in the information processing system 200, etc.).

[0050] Such software components may be loaded from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording medium may include a recording medium directly connectable to such information processing system 200, but may also include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. As another example, the software components may be loaded into the memory 210 via the communication module 230, which is not a computer-readable recording medium. For example, at least one program may be loaded into the memory 210 based on a computer program (e.g., a program for generating mixed reality images) installed by a file provided via the communication module 230 by a developer or a file distribution system that distributes an application installation file.

[0051] The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to a user terminal (not shown) or other external system by the memory 210 or the communication module 230. For example, the processor 220 may receive a plurality of pieces of information (e.g., actual images) required for mixed reality image generation, and generate a mixed reality image related to a driving environment based on the received plurality of pieces of information.

[0052] The communication module 230 may provide a configuration or function for a user terminal (not shown) and the information processing system 200 to communicate with each other via a network, and may provide a configuration or function for the information processing system 200 to communicate with an external system (such as a separate cloud system, for example). As an example, control signals, commands, data, etc. provided under the control of the processor 220 of the information processing system 200 may be transmitted to the user terminal and / or the external system via the communication module of the user terminal and / or the external system via the communication module 230 and the network. For example, the user terminal may receive a mixed reality image related to the generated driving environment.

[0053] Also, the input / output interface 240 of the information processing system 200 may be a means for connecting to the information processing system 200 or interfacing with a device (not shown) for input or output that the information processing system 200 may include. In Fig. 2, the input / output interface 240 is shown as an element configured separately from the processor 220, but is not limited thereto, and the input / output interface 240 may be configured to be included in the processor 220. The information processing system 200 may include more components than those shown in Fig. 2. However, it is not necessary to explicitly illustrate most of the conventional technical components.

[0054] The processor 220 of the information processing system 200 may be configured to manage, process, and / or store information and / or data received from a plurality of user terminals and / or a plurality of external systems. According to an embodiment, the processor 220 may receive a real image related to a driving environment. The processor 220 may then generate a mixed reality image including objects of the driving environment based on the received real image and the generated virtual image.

[0055] FIG. 3 is a diagram illustrating an internal configuration of a processor 220 of an information processing system 200 according to an embodiment of the present disclosure. According to an embodiment, the processor 220 may include a virtual image generating unit 120, a real image processing unit 150, a virtual image processing unit 160, and a mixed reality image generating unit 170. The internal configuration of the processor 220 of the information processing system illustrated in FIG. 3 is merely an example, and at least some of the configurations of the processor 220 may be omitted or other configurations may be added, and at least some of the operations or processes performed by the processor 220 may be implemented differently, such as being performed by a processor of a user terminal communicatively connected to the information processing system. Note that in FIG. 3, the configuration of the processor 220 is described by dividing it into its respective functions, but this does not necessarily mean that it is physically divided. For example, the real image processing unit 150 and the virtual image processing unit 160 are described separately, but this is for the purpose of helping to understand the invention, and is not limited thereto.

[0056] According to an embodiment, the virtual image generator 120 may generate a virtual image for generating a mixed reality image related to a driving environment. For example, the virtual image generator 120 may generate a first virtual image including an object of the driving environment (e.g., a traffic cone located on a road, another driving vehicle, etc.) and / or a second virtual image related to the driving environment. The first virtual image may include an image generated from a simulation model that realizes the driving environment. The second virtual image may include a semantic image generated from a simulation model that realizes the driving environment. In an embodiment, the first virtual image and the second virtual image may be generated from data acquired using a second camera and a third camera mounted on a vehicle realized in the simulation model. In this case, the vehicle realized in the simulation model may be the same model as a vehicle equipped with a first camera for acquiring an actual image. In addition, the second camera and the third camera may be virtual cameras realized in the simulation model as cameras having the same external parameters (e.g., camera orientation, etc.) and internal parameters (e.g., focal length, optical center, etc.) as the first camera 100.

[0057] According to an embodiment, the real image processing unit 150 may generate a processed real image for generating a mixed reality image from the real image. For example, the real image processing unit 150 may generate a processed real image by removing a part of the real image. Also, the real image processing unit 150 may use a second virtual image to generate the processed real image. For example, the real image processing unit 150 may generate a mask inverse image from the second virtual image, and remove a part of the real image using the mask inverse image to generate the processed real image. This will be described in detail later with reference to FIG. 6.

[0058] According to an embodiment, the virtual image processing unit 160 may generate a processed virtual image from a first virtual image. For example, the virtual image processing unit 160 may generate a processed virtual image by extracting a portion corresponding to an object from the first virtual image. The virtual image processing unit 160 may also use a second virtual image to generate the processed virtual image. For example, the virtual image processing unit 160 may generate a mask image from the second virtual image and extract a portion corresponding to the object from the first virtual image using the mask image to generate the processed virtual image. This will be described in detail below with reference to FIG. 5.

[0059] According to an embodiment, the mixed reality image generator 170 may generate a mixed reality image based on a processed virtual image and a processed real image. For example, the mixed reality image generator 170 may generate a mixed reality image by fusing a processed virtual image and a processed real image. This will be described in detail below with reference to FIG. 4.

[0060] 4 is an example diagram of a mixed reality image 190 being generated from a real image 110 and virtual images 130 and 140 according to one embodiment of the present disclosure. As shown in the figure, the mixed reality image 190 may be generated based on a first virtual image 130, a second virtual image 140, and the real image 110.

[0061] According to an embodiment, the real image 110 may be an image related to a driving environment. For example, the real image 110 may be an image captured of a driving road for testing an autonomous vehicle. The first virtual image 130 may include objects of the driving environment as an image generated from a simulation model that realizes the same driving environment as the real image. For example, the first virtual image 130 may be an image captured by a camera in the simulation model of surrounding vehicles or traffic cones existing in the simulation model that realizes the driving road of the real image. The second virtual image 140 may be a semantic image generated from a simulation model that realizes the same driving environment as the real image. For example, the second virtual image 140 may be an image segmented by object by applying a semantic segmentation technique to an image captured by a camera in the simulation model of surrounding vehicles or traffic cones existing in the simulation model that realizes the driving road of the real image.

[0062] According to an embodiment, the first virtual image 130 may be processed to generate a processed virtual image 162 (410). More specifically, the processed virtual image 162 may be generated by processing the first virtual image 130 in the virtual image processing unit 160 and extracting a portion corresponding to an object from the first virtual image 130. For example, a processed virtual image 162 in which only the extracted portion exists may be generated by extracting a portion corresponding to a vehicle and a traffic cone present in the first virtual image. In addition, the second virtual image 140 may be used in the process of processing the first virtual image 130. For example, a mask image for the vehicle and the traffic cone present in the first virtual image may be generated using the second virtual image 140, and an image in which the portion corresponding to the vehicle and the traffic cone is extracted may be generated using the generated mask image. This will be described in detail below with reference to FIG. 5.

[0063] According to an embodiment, the real image 110 may be processed to generate a processed real image 152 (420). More specifically, the processed real image 152 may be generated by removing a part of the real image 110 corresponding to a part of the real image 110 that corresponds to an object of the first virtual image, which is processed by the real image processing unit 150. For example, the processed real image 152 may be generated by removing a part of the real image 110 that corresponds to a part where the vehicle and traffic cone of the first virtual image are located. In addition, the second virtual image 140 may be used in the process of processing the real image 110. For example, a mask inverse image of the vehicle and traffic cone present in the first virtual image may be generated using the second virtual image 140, and an image in which a part of the real image 110 is removed may be generated using the generated mask inverse image. This will be described in detail below with reference to FIG. 6.

[0064] According to an embodiment, the processed virtual image 162 and the processed real image 152 may be fused to generate a mixed reality image 190 (180). More specifically, the mixed reality image 190 may be generated by projecting objects present in the processed virtual image 162 generated by the mixed reality image generator 170 onto the processed real image 152. For example, the mixed reality image 190 may be generated by projecting and fusing vehicles and traffic cones present in the processed virtual image 162 onto the removed portion of the processed real image 152.

[0065] 5 is an example diagram illustrating an example of generating an edited virtual image 162 from the first virtual image 130 and the second virtual image 140 according to an embodiment of the present disclosure. According to an embodiment, the edited virtual image 162 may be generated by extracting (530) portions corresponding to objects from the first virtual image 130. For example, by extracting portions corresponding to vehicles and traffic cones from the first virtual image 130, the edited virtual image 162 may be generated in which the remaining portions except for the portions corresponding to the vehicles and traffic cones are removed.

[0066] According to an embodiment, a mask image 520 may be used to extract a portion corresponding to an object from the first virtual image 130. For example, the mask image 520 is an image that may mask a portion corresponding to an object in the first virtual image 130, and by applying the mask image 520 to the first virtual image 130, the remaining portion except for the masked object portion may be removed. In this manner, the mask image 520 may be used to mask a portion corresponding to an object in the first virtual image 130, thereby extracting the portion corresponding to the object. In this case, the mask image 520 may be a binary type image.

[0067] According to an embodiment, the mask image 520 may be generated from the second virtual image 140. For example, the second virtual image 140 is an image separated into objects using a semantic segmentation technique, and the mask image 520 may be generated by filtering portions of the second virtual image corresponding to the objects (510).

[0068] 6 is an example diagram illustrating an example of generating an edited real image 152 from the real image 110 and the second virtual image 140 according to an embodiment of the present disclosure. According to an embodiment, the edited real image 152 may be generated by removing (630) a portion of the real image 110 that corresponds to a portion of the object in the first virtual image 130. For example, the edited real image 152 may be generated by removing a portion of the real image 110 that corresponds to a portion of the vehicle and traffic cone in the first virtual image.

[0069] According to an embodiment, a mask inverse image 620 may be used to remove a portion of the real image 110. The mask inverse image is an image that can mask the remaining portion of the first virtual image 110 except for the portion corresponding to the object, and by applying the mask inverse image 620 to the real image 110, only the portion corresponding to the object in the first virtual image 110 may be removed.

[0070] According to one embodiment, a mask inverse image 620 may be generated from the second virtual image 140. For example, a mask image 520 may be generated 510 by filtering the second virtual image 140, and the mask inverse image 620 may be generated 610 by inverting the mask image 520. The mask inverse image 620 may be a binary type image, similar to the mask image 520.

[0071] 7 is a diagram illustrating an example of a first virtual image 130 according to an embodiment of the present disclosure. According to an embodiment, the first virtual image 130 may be an image generated from a simulation model that realizes the same driving environment as the actual image. More specifically, the first virtual image 130 may be generated from data acquired using a camera mounted on a vehicle realized in the simulation model as the same vehicle model as the vehicle to which the camera that captured the actual image is attached. In addition, the camera used to generate the first virtual image 130 may be a virtual camera realized in the simulation model as a camera having the same external parameters (e.g., camera orientation, etc.) and internal parameters (e.g., focal length, optical center, etc.) as the camera that captured the actual image.

[0072] According to an embodiment, the first virtual image 130 may include objects of the driving environment. For example, the first virtual image 130 may be an image including a vehicle 720 driving around and traffic cones 710_1, 710_2, 710_3, and 710_4 placed around the vehicle. The objects of the first virtual image 130 may be extracted and processed by a mask image and merged with the actual image to generate a mixed reality image.

[0073] 8 is a diagram illustrating an example of an actual image 110 according to an embodiment of the present disclosure. According to an embodiment, the actual image 110 may be an image related to a driving environment. For example, the actual image 110 may be an image including a driving road and objects (e.g., surrounding vehicles, etc.) located in the vicinity. The actual image 110 may be acquired by a camera mounted on a vehicle.

[0074] According to an embodiment, the real image 110 may be processed to generate a mixed reality image. More specifically, a portion of the real image 110 that corresponds to an object of the first virtual image may be removed. For example, a portion 820 corresponding to a portion where a vehicle is located in the first virtual image and portions 810_1, 810_2, 810_3, and 810_4 corresponding to a portion where a traffic cone is located may be removed by a mask inverse image.

[0075] 9 is a diagram illustrating an example of a mixed reality image 190 according to an embodiment of the present disclosure. According to an embodiment, the mixed reality image 190 may be generated by fusing a processed virtual image with a processed real image. The mixed reality image 190 may be generated by projecting a portion corresponding to an object of the first virtual image extracted onto the real image. For example, the mixed reality image 190 in which the virtual object and the real image are fusing may be generated by combining the surrounding vehicle 920 and traffic cones 910_1, 910_2, 910_3, and 910_4 of the first virtual image with the real image including the road on which the vehicle is traveling.

[0076] In this way, by visualizing and providing virtual objects in the actual driving environment, the tester in the autonomous vehicle can visually recognize the test situation and perform the test smoothly. Furthermore, by providing an image in which virtual objects are projected in the driving environment during the autonomous vehicle testing process, instead of actually placing physical objects for the autonomous vehicle testing, it is possible to save time and costs required for the test.

[0077] 10 is a flowchart illustrating a method 1000 for generating mixed reality images related to a driving environment according to one embodiment of the present disclosure. The method 1000 may be executed by at least one processor (e.g., processor 220) of an information processing system. As shown in the figure, the method 1000 may begin by acquiring an actual image related to the driving environment (S1010). In one embodiment, the actual image related to the driving environment may be acquired using a first camera mounted on a vehicle.

[0078] The processor may then generate a first virtual image including objects of the driving environment (S1020). In one embodiment, the first virtual image may include an image generated from a simulation model that realizes the driving environment. The first virtual image may also be generated from data acquired using a second camera mounted on a vehicle that is realized in the simulation model as the same vehicle model as the vehicle in which the camera used to acquire the actual image is mounted. The second camera may be a virtual camera that is realized in the simulation model as a camera having the same external and internal parameters as the first camera.

[0079] Finally, the processor may generate a mixed reality image including an object of the driving environment based on the real image and the first virtual image (S1030). In one embodiment, the processor may extract a portion corresponding to the object from the first virtual image, and generate the mixed reality image based on the portion corresponding to the object and the real image. The processor may also remove a portion of the real image corresponding to the portion corresponding to the object.

[0080] In one embodiment, the processor may generate a second virtual image related to the driving environment and generate a mixed reality image based on the actual image, the first virtual image, and the second virtual image. The second virtual image may include a semantic image generated from a simulation model that realizes the driving environment. The second virtual image may be generated from data acquired using a third camera attached to a vehicle realized in the simulation model as the same vehicle model as the vehicle in which the camera used to acquire the actual image is installed, and the third camera may be a virtual camera realized in the simulation model as a camera having the same external parameters and internal parameters as the first camera.

[0081] In one embodiment, the processor may extract a portion corresponding to the object from the first virtual image using the second virtual image, filter the second virtual image to generate a mask image for the object, and extract the portion corresponding to the object from the first virtual image using the mask image.

[0082] In one embodiment, the processor may use the second virtual image to remove a portion of the actual image corresponding to a portion of the first virtual image corresponding to an object. The processor may generate a mask inverse image from the second virtual image and remove a portion of the actual image corresponding to a portion of the object using the mask image. The mask inverse image may be generated by generating a mask image for the object from the second virtual image and inverting the mask image to generate the mask inverse image.

[0083] 10 and the above description are merely examples and may be implemented differently in some embodiments. For example, in some embodiments, the order of steps may be changed, some steps may be repeated, some steps may be omitted, or some steps may be added.

[0084] The above method may be provided as a computer program stored in a computer-readable recording medium for execution by a computer. The medium may be one that continuously stores a computer-executable program or one that temporarily stores the program for execution or download. The medium may be various recording or storage means in the form of a single or multiple hardware components combined, but is not limited to a medium directly connected to any computer system, and may be one that is distributed over a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto optical media such as floptical disks, and ROMs, RAMs, flash memories, etc., configured to store program instructions. Other examples of the medium include recording media or storage media managed by app stores that distribute applications, and sites, servers, etc. that supply or distribute various other software.

[0085] The methods, operations, or techniques of the present disclosure may be realized by various means. For example, such techniques may be realized by hardware, firmware, software, or a combination thereof. Those of ordinary skill in the art will appreciate that the various exemplary logic blocks, modules, circuits, and algorithmic steps described in connection with the disclosure of this application may also be realized by electronic hardware, computer software, or a combination of both. To clearly explain such interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is realized as hardware or software depends on the particular application and design requirements imposed on the overall system. Those of ordinary skill in the art may realize the described functionality in various ways for each particular application, but such implementation should not be interpreted as departing from the scope of the present disclosure.

[0086] In a hardware implementation, the processing units used to perform the techniques may be implemented within one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described in this disclosure, computers, or combinations thereof.

[0087] Thus, the various example logic blocks, modules, and circuits described in connection with this disclosure may be implemented or performed by a general purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination designed to perform the functions described herein. A general purpose processor may be a microprocessor, but the processor may also be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

[0088] In a firmware and / or software implementation, the techniques may be implemented with instructions stored on a computer-readable medium, such as random access memory (RAM), read-only memory (ROM), nonvolatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. The instructions may also be executable by one or more processors to cause the processors to perform certain aspects of the functions described in this disclosure.

[0089] Although the embodiments described above are described as utilizing aspects of the presently disclosed subject matter in one or more stand-alone computer systems, the present disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or distributed computing environment. Furthermore, aspects of the subject matter in the present disclosure may be implemented in multiple processing chips or devices, and storage may be affected similarly across multiple devices. Such devices may include PCs, network servers, and even portable devices.

[0090] Although the present disclosure has been described herein with reference to some embodiments, various modifications and changes may be made without departing from the scope of the present disclosure that would be understood by a person of ordinary skill in the art to which the present disclosure pertains, and such modifications and changes should be considered to fall within the scope of the claims appended hereto.

Claims

1. A method for generating mixed reality images relating to a driving environment, the method being executed by at least one processor, comprising: acquiring actual video related to a driving environment; generating a first virtual image including an object of the driving environment; generating a mixed reality image including an object of the driving environment based on the real image and the first virtual image; A mixed reality image generation method comprising:

2. The method of claim 1 , wherein the first virtual image includes an image generated from a simulation model that realizes the driving environment.

3. generating a second virtual image related to the driving environment; generating a mixed reality image based on the real image and the first virtual image, The method of claim 1 , further comprising: generating a mixed reality image based on the real image, the first virtual image, and the second virtual image.

4. The method of claim 3 , wherein the second virtual image includes a semantic image generated from a simulation model that realizes the driving environment.

5. The step of acquiring an actual image related to a driving environment includes: The method of claim 3 , further comprising the step of acquiring the real image using a first camera mounted on a vehicle.

6. The first virtual image is generated from data acquired using a second camera mounted on a vehicle realized in a simulation model as the same vehicle model as the vehicle; The second camera is The method of claim 5 , wherein the first camera is a virtual camera realized in a simulation model as a camera having the same external parameters and internal parameters as the first camera.

7. The second virtual image is generated from data acquired using a third camera mounted on a vehicle realized in a simulation model as the same vehicle model as the vehicle; The third camera is The method of claim 5 , wherein the first camera is a virtual camera realized in a simulation model as a camera having the same external parameters and internal parameters as the first camera.

8. generating a mixed reality image based on the real image and the first virtual image, extracting a portion corresponding to an object from the first virtual image; generating a mixed reality image based on a portion corresponding to the object and the actual image; The mixed reality image generating method according to claim 3 , comprising:

9. The step of extracting a portion corresponding to the object includes: The method of claim 8 , further comprising: extracting a portion corresponding to an object using the second virtual image.

10. The step of extracting a portion corresponding to an object using the second virtual image includes: filtering the second virtual image to generate a mask image for the object; extracting a portion corresponding to an object from the first virtual image using the mask image; The mixed reality image generating method according to claim 9 , comprising:

11. generating a mixed reality image based on the portion corresponding to the object and the actual image, The method of claim 8 , further comprising removing a portion of the real image corresponding to the object.

12. The step of removing a portion of the actual image corresponding to the object includes: The method of claim 11 , further comprising removing a portion of the real image corresponding to a portion of the object using the second virtual image.

13. removing a portion of the actual image corresponding to a portion of the object using the second virtual image, generating a mask inverse image from the second virtual image; removing a portion of the actual image corresponding to the object using the mask inverse image; The mixed reality image generating method according to claim 12, comprising:

14. The step of generating a mask inverse image includes: generating a mask image for the object from the second virtual image; inverting the mask image to generate the mask inverse image; The mixed reality image generating method according to claim 13, comprising:

15. A non-transitory computer readable recording medium having instructions recorded thereon for executing the method according to claim 1 on a computer.

16. As an information processing system, A communication module; Memory, at least one processor coupled to the memory and configured to execute at least one computer readable program contained in the memory; The at least one program comprises: Obtaining actual footage related to the driving environment, generating a first virtual image including an object of the driving environment; an information processing system comprising: a command for generating a mixed reality image including an object of the driving environment based on the real image and the first virtual image;

Images