Method for updating machine learning model in real time

Abstract

The present invention comprises: a step in which an object recognition machine learning application or a driving scenario machine learning application receives an event for an important object and a danger signal; a step in which the object recognition machine learning application receives a video image stored for a predetermined period of time at a time when an event occurs; a step in which a first object recognition embedding vector is generated, an object recognition probability value is extracted, and a second object recognition embedding vector satisfying a probability value higher than a predetermined level is transmitted to the driving scenario machine learning application; a step in which the driving scenario machine learning application processes the second object recognition embedding vector as an input value to generate a first driving scenario embedding vector, and calculates a probability value of the first driving scenario embedding vector; and a step in which a second driving scenario embedding vector having a probability value lower than a predetermined level is extracted and a feedback signal is transmitted to the object recognition machine learning application with reference to object information included in the second driving scenario.

Description

How to update machine learning models in real time

[0001] The present invention relates to a real-time machine learning model update method, and more particularly, to a real-time machine learning model update method in which object recognition and driving scenarios are learned through separate machine learning, but the reliability of the machine learning model of the driving scenario is increased, and object recognition errors can be verified using semantic relations extracted by the machine learning model of the driving scenario.

[0002] Combining image information and 3D height information is being applied to facility management, security technology, and autonomous driving technology development. Recently, there have been active attempts to develop automated models by generating and learning virtual information based on user-desired scenarios using artificial intelligence and machine learning technology.

[0003] 3D object information acquired from various sensors is stored to support resolution according to distance (level of detail: LoD), and the method of storing 3D data is divided into a method of finding image areas with high similarity within the same image (intra-frame prediction) and a method of finding similarity in different image frames (inter-frame prediction). Recently, technologies that provide 3D video images using video point cloud compression technology (V-PCC: video point cloud compression) and geometry-based point cloud compression technology (G-PCC: geometry-based point cloud compression) are being developed and applied.

[0004] V-PCC technology classifies 3D objects, stores the classified 3D objects as patch information projected onto a hexahedron of a 3D grid (voxel grid), converts them into 2D video images, and transmits them using H.265 and HEVC compression technologies.

[0005] A point cloud is composed of R / G / B or Y / Cb / Cr image pixel values ​​in 3D space, and in G-PCC, one object is stored in a voxel structure consisting of an occupancy map image for a 3D / 2D patch, a geometry image of the 3D patch, and a texture (color) image of the 3D patch, and the 3D object can be reconstructed and rendered by adjusting the image viewpoint and the photometric normal vector in a specific direction. However, errors occur when the coordinates of the 3D point used for the 3D patch projected on each plane are changed to 2D information, and distortion also occurs in the geometric information of the 3D point due to the difference between the resolution supported by the grid defined in the system and the actual expressed resolution. The luminance normal vector is a normal vector for a surface used in the photometric stereo method used to obtain information on the surface of an object when there are images obtained by observing the same object from at least three different light sources, and obtaining the normal vector allows one to know the surface shape of the object.

[0006] If the user's viewpoint coordinates are managed in 3D and the LoD and orientation information of the viewpoint of the provided video image are stored and managed, it is possible to analyze images previously transmitted to the user. To calculate the area of ​​a previously transmitted image, the image area is calculated based on the 3D grid coordinates of the viewpoint. At this time, the image area provided is determined according to the LoD. In addition, pixel values ​​change depending on the orientation angle of the user's viewpoint.

[0007] Creating automated models by analyzing video images requires defining driving situations and road characteristics. The Association for Standardization of Automation and Measuring Systems (ASAM) defines openODD (Operational Domain Design: ODD), which categorizes object attribute information into relations, contexts, actions, events, and objects. It also provides an ontology structure between objects, enabling the creation of standardized object information.

[0008] The standard for simulation-based testing to ensure the safety of autonomous vehicles (ISO / DIS 34503) configures object information into scenery elements, environmental conditions, and dynamic elements to define driving situation locations, weather, intersections, and traffic conditions.

[0009] Therefore, for autonomous driving and automation of various devices, video images and 3D height information acquired from various devices can be analyzed to classify and store object information as standardized data, and scenario-based simulation data can be generated to apply artificial intelligence and machine learning methods to create a machine learning model.

[0010] Accordingly, the problem to be solved by the present invention is to provide a real-time machine learning model update method that can verify object recognition errors by using semantic relations extracted by the machine learning model of the driving scenario, thereby increasing the reliability of the machine learning model of the driving scenario, even though object recognition and driving scenarios are learned through separate machine learning.

[0011] We propose an optimization method for object recognition and virtual object generation for object recognition. We also aim to effectively merge images collected from different environments and generate video images containing virtual objects. Furthermore, we present machine learning methods and their application for implementing driving scenarios for object recognition and collision avoidance.

[0012] The present invention provides a method for supporting the update of a real-time machine learning model, in order to achieve the above object, comprising: a step in which an object recognition machine learning application or a driving scenario machine learning application receives an event regarding an important object and a danger signal; a step in which the object recognition machine learning application receives a video image stored for a certain period of time at a time when the event occurs; a step in which the object recognition machine learning application generates a first object recognition embedding vector, extracts an object recognition probability value, and transmits a second object recognition embedding vector satisfying a probability value higher than a certain level to the driving scenario machine learning application; a step in which the driving scenario machine learning application processes the second object recognition embedding vector as an input value to generate a first driving scenario embedding vector and calculates a probability value of the first driving scenario embedding vector; A method for supporting updating of a real-time machine learning model is provided, characterized in that the driving scenario machine learning application extracts the second driving scenario embedding vector having a probability value below a certain level, and transmits a feedback signal to the object recognition machine learning application with reference to object information included in the second driving scenario, and the object recognition machine learning application optimizes the first object recognition embedding vector with reference to the object information included in the feedback signal.

[0013] According to one embodiment of the present invention, it is preferable that the object recognition machine learning model is a machine learning model generated by convolutional neural network (CNN) learning, and the driving scenario machine learning model is a machine learning model generated by graph neural network (GNN) learning.

[0014] In addition, it is preferable that the driving scenario machine learning model is a machine learning model trained using driving scenario learning data provided by a generative adversarial network (GAN) machine learning model.

[0015] According to another embodiment of the present invention, it is preferable that the driving scenario learning data provided by the adversarial generative machine learning model is updated by adding a new driving scenario generated by combining the reference driving scenario learning data and an ontology reflecting hierarchical information between objects to the reference driving scenario learning data.

[0016] In addition, the learning rate of the object recognition machine learning application can be increased for a certain period of time from the time of occurrence of the received event, or the learning rate of the driving scenario machine learning application can be increased for training.

[0017] According to another embodiment of the present invention, when the object recognition machine learning application is retrained by optimizing the first object recognition embedding vector, the driving scenario machine learning application can be retrained.

[0018] In addition, when the object recognition machine learning application transmits an object recognition machine learning model updated through retraining of the object recognition machine learning application to the device that transmitted the event, the driving scenario machine learning application can also transmit the driving scenario machine learning model updated through retraining of the driving scenario machine learning application to the device that transmitted the event.

[0019] By using image blur and representative photometric normal vector, object recognition area can be quickly extracted from video image, and differential resolution can be defined at the object recognition stage according to the importance of the object, so that after object recognition, variable image resolution can be provided according to the importance of the object when rendering the image.

[0020] Recognized objects can be generated by extracting (A) 3D object information, (B) distance-based layer and alpha channel information of recognized objects, (C) stereo area, (D) pixel similarity information by image recognition area, and (E) luminance normal vector, and managing object information by group (subset) and effectively arranging virtual objects.

[0021] By utilizing 3D information generated from high-resolution images and object recognition information of a subset, low-resolution images can be converted into high-resolution images and texturing can be performed to combine the images.

[0022] A machine learning model is created by combining various machine learning methods to extract object recognition and driving scenarios, and driving scenario machine learning (ML) is performed by modifying and augmenting (adding virtual objects) object recognition information (object recognition embedding vector), and then the generated driving scenario ML model is applied to verify the object recognition embedding vector with added virtual objects, and if the model is satisfied, an actual virtual object image is created.

[0023] According to the present invention, data processing can be improved by quickly selecting an object recognition area by changing a high-resolution image into a low-resolution image.

[0024] In addition, according to the present invention, by managing object recognition image information by group, object recognition, virtual objects, and backgrounds can be easily added, and machine learning learning data can be effectively constructed, and after ML learning, a model is created and applied, and the virtual object embedding vector is augmented to verify data reliability, and then the virtual object is inserted into an actual image, thereby increasing data reliability.

[0025] Furthermore, according to the present invention, although object recognition and driving scenarios are learned through separate machine learning, the reliability of the ML model of the driving scenario can be increased, and object recognition errors can be verified using the semantic relation information between objects extracted by the model.

[0026] FIG. 1 is a configuration diagram of an object recognition and virtual object generation system according to one embodiment of the present invention.

[0027] FIG. 2 is a configuration diagram of an object recognition and virtual object generation system according to another embodiment of the present invention.

[0028] Figure 3 is a flowchart illustrating a method for determining an object recognition area according to one embodiment of the present invention.

[0029] Figure 4 is an example of an image in which only objects with blurring noise below a certain level are extracted after applying blurring.

[0030] Figure 5 is a flowchart of a method for monitoring overlapping objects by managing information in the object recognition area in subset units.

[0031] FIG. 6 is a flowchart showing a method for combining images by an image combining unit (300) according to another embodiment of the present invention.

[0032] Figure 7 is a flowchart of a method for a virtual object creation unit (400) to create a virtual object.

[0033] Figure 8 is a flowchart of a method for a virtual object creation unit (400) to create a virtual background.

[0034] Figure 9 is a diagram illustrating the relationship between a data acquisition device, a driving scenario machine learning learning and model generation server, and an object recognition machine learning learning and model generation server.

[0035] Figure 10 is a detailed description of the driving scenario verification unit (500), and is a flowchart showing a method for creating a driving scenario machine learning model.

[0036] Figure 11 is a flowchart illustrating a method for creating a virtual object and verifying a driving scenario.

[0037] Figure 12 is a configuration diagram of a device that learns a driving scenario using a GAN machine learning method.

[0038] Figure 13 is a process of creating and applying a driving scenario ML model to extract virtual / reference objects that do not satisfy a certain level of driving scenarios.

[0039] Figure 14 is a sequence diagram showing data transmission between a data acquisition device (810), an object recognition machine learning learning and model generation server (820), and a driving scenario machine learning learning and model generation server (830).

[0040] The present invention relates to a method for supporting the update of a real-time machine learning model, and the method for supporting the update of a real-time machine learning model comprises the steps of: receiving an event for an important object and a danger signal by an object recognition machine learning application or a driving scenario machine learning application; receiving a video image stored for a predetermined period of time at the time of occurrence of the event by the object recognition machine learning application; generating a first object recognition embedding vector, extracting an object recognition probability value, and transmitting a second object recognition embedding vector satisfying a probability value higher than a predetermined level to the driving scenario machine learning application; generating a first driving scenario embedding vector by the driving scenario machine learning application by processing the second object recognition embedding vector as an input value, and calculating a probability value of the first driving scenario embedding vector; extracting the second driving scenario embedding vector having a probability value lower than a predetermined level by the driving scenario machine learning application, and transmitting a feedback signal to the object recognition machine learning application with reference to object information included in the second driving scenario; and The machine learning application is characterized in that it optimizes the first object recognition embedding vector by referring to the object information included in the feedback signal.

[0041] Hereinafter, with reference to the attached drawings, preferred embodiments of the present invention will be described in detail, enabling those skilled in the art to easily implement the invention. However, these embodiments are intended to illustrate the invention more specifically, and it will be apparent to those skilled in the art that the scope of the present invention is not limited thereby.

[0042] In order to clearly solve the problem to be solved by the present invention, the composition of the invention will be described in detail based on a preferred embodiment of the present invention with reference to the accompanying drawings. In assigning reference numbers to components in the drawings, the same reference numbers are assigned to the same components even if they are in different drawings, and it is made clear in advance that components in other drawings may be cited when necessary when describing the drawings. In addition, in explaining the operating principle of the preferred embodiment of the present invention in detail, if it is determined that a specific description of a known function or configuration related to the present invention and other matters may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

[0043] Terms including ordinal numbers, such as "second," "first," etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a second component may be referred to as a "first component," and similarly, a first component may also be referred to as a "second component." The term "and / or" includes a combination of multiple related items described herein or any of multiple related items described herein.

[0044]

[0045] FIG. 1 is a configuration diagram of an object recognition and virtual object generation system according to one embodiment of the present invention.

[0046] An object recognition and virtual object generation system according to one embodiment of the present invention comprises an object recognition area determination unit (100), an object-specific image subset management unit (200), an image combination unit (300), and a virtual object generation unit (400).

[0047] The object recognition area determination unit (100) extracts the object area by tracking the shadow area of ​​the video image.

[0048] The shadow region analysis unit (110) extracts the object region by tracking the shadow region at a low resolution. The shadow region analysis unit (110) converts a high-resolution image to a low resolution, and then quickly classifies only the shadow region, predicts the object region based on the shadow region, and analyzes the high-resolution region for the predicted region. In order to quickly extract the object recognition region, the blurring analysis unit (120) applies blurring to two or more consecutive images when the video image is converted to a low-resolution image, and selects only objects with blurring noise below a certain level, thereby enabling quick object region selection.

[0049] The object-specific image subset management unit (200) configures a subset based on objects recognized from the entire image video and manages attribute information such as object type, color, and importance.

[0050] The object geometry information acquisition unit (210) acquires height information and edge information of the recognized object.

[0051] The stereoscopic image analysis unit (220) analyzes objects in the stereoscopic image area by considering both the camera viewpoint and the light normal vector. At this time, in image areas other than the stereo area, only the light normal vector is considered.

[0052] Additionally, the light normal vector unit (230) analyzes light and intensity direction information using the representative light normal vector of the object area.

[0053] Additionally, the pixel similarity analysis unit (240) analyzes the similarity information of pixel values ​​in the object recognition area. When creating a virtual object, the pixel similarity analysis unit (240) performs texturing by considering the pixel value similarity information of an existing object.

[0054] In addition, the layer information analysis unit (250) determines the presence / absence of an object by adjusting the distance-based object layer information and alpha channel information when creating a virtual object and performing machine learning. The information analyzed by the layer information analysis unit (250) is used to add the necessary virtual object. The alpha channel information is auxiliary data information separated from the color expression data for each pixel.

[0055] The image combining unit (300) uses high-resolution geometry to correct a low-resolution image, converts it back into a high-resolution image, and then combines various images into an image in the same coordinate system.

[0056] The object integration unit (310) converts objects of various images into absolute coordinates and integrates them into the same coordinate system.

[0057] The adjustment unit (320) determines a subset of each object having the same coordinate system in the voxel structure by position, and selects an object in each subset that has a high image resolution, exists in the stereo area of ​​the camera, and has the smallest blur noise between consecutive images of the same subset as a reference object.

[0058] The adjustment unit (320) determines the luminance normal vector and pixel statistics of the reference object of each subset, and determines the objects as overlapping if the recognition areas between the objects have an overlapping area of ​​a certain level or more. In addition, the adjustment unit (320) corrects the spatial pose of the objects included in the subset group to remove the parallax between images captured from different camera viewpoints, and modifies the texture of the same subset based on the statistics of the reference object.

[0059] The virtual object creation unit (400) uses a 3D virtual object to add a virtual object to a reference image.

[0060] Additionally, the virtual object placement unit (410) adds virtual objects using the scale of the object determined by distance to the reference image, alpha channel information, and nearby object information.

[0061] The virtual object posture determination unit (420) determines the virtual object posture using the camera viewpoint and stereoscopic image area according to the arrangement of the virtual object.

[0062] The texturing unit (430) performs texturing using the intensity and direction of light, shadow information, pixel similarity information of nearby objects, etc.

[0063]

[0064] FIG. 2 is a configuration diagram of an object recognition and virtual object generation system according to another embodiment of the present invention.

[0065] Fig. 2 is different from Fig. 1 in that a driving scenario verification unit (500) is added between the object-specific image subset management unit (200) and the virtual object creation unit (400).

[0066] The driving scenario verification unit (500) performs the function of adding a virtual object to an actual image if, before generating an image of a virtual object, it recognizes an existing object as a virtual object or satisfies the driving scenario machine learning model by updating only the object recognition embedding vector for the virtual object without modifying the image, and generates a driving scenario model by applying a GNN (graph neural network) learning method.

[0067] The object-specific recognition feature generation unit (510) defines object-specific recognition features based on OpenODD attribute information.

[0068] The driving scenario definition section (520) defines the driving scenario applied to the system, reflecting the ISO / DIS 34503 standard.

[0069] The graph embedding vector generation unit (530) generates a graph embedding vector using the recognized object recognition embedding vector and OpenODD ontology information. At this time, for efficient machine learning, the graph embedding vector can be generated by generating virtual driving scenario vector information.

[0070] The driving scenario machine learning model generation unit (540) generates a driving scenario machine learning model by applying the GNN learning method.

[0071] The driving scenario verification unit (550) verifies the applied driving scenario ML module, extracts abnormal links from among the driving scenarios defined by the user, and generates a feedback signal.

[0072] During the video image collection phase, key event information such as camera speed changes, ambient sound changes, and ambient illumination changes are stored, and the collected event information is utilized when determining the object recognition area. For fast object recognition, the image is converted to a low-resolution grayscale image, and then the shadow area or pixels with a brightness value above a certain level are determined to extract important light direction and brightness information. Based on the shadow area and important luminance normal vector area, the primary recognition area of ​​each object is selected, and then, according to the importance of each object, the primary recognition area of ​​the object is determined by variably changing the resolution of the RGB image (VRS), applying a blurring kernel, and removing noise. The determined pixel area of ​​each object is organized into subsets of object recognition areas of consecutive images using the pixel similarity information of each area and managed, and each object information is managed for each subset.

[0073] Figure 3 is a flowchart illustrating a method for determining an object recognition area according to one embodiment of the present invention.

[0074] Step 21 is the step where changes in ambient light brightness, changes in ambient noise, and changes in speed above a certain level occur on the edge device, and when a major object is recognized, it is saved as a major event.

[0075] Step 22 is a step in which the edge device transmits video images including the stored main events and events to the cloud server at regular intervals.

[0076] In step 101, the cloud server determines the sampling time of the video image by considering the main event information stored in 22 (the time of change in the brightness of the ambient light and the change in the ambient sound and the camera movement speed). For example, if the camera movement speed is 80 km / h, it moves approximately 22 m / s, so if there is an object with a possibility of collision in the vicinity, the sampling time can be determined at a cycle of 1 / 10 second or shorter to determine the object recognition area. Therefore, using the camera movement speed event stored in step 21, if the movement speed is high, the sampling interval of the video image in the time domain should be shortened, and if the movement speed is low, the sampling interval should be relatively long. In addition, if the brightness of the ambient light and noise change beyond a certain level, it can be used as a factor for adjusting the sampling time.

[0077] Step 102 is the step of converting a high-resolution image into a low-resolution image and adjusting it to grayscale for fast processing of video images.

[0078] Step 110 determines the region of interest where an object exists by selecting pixel areas with brightness values ​​above a certain level and pixel areas with brightness values ​​below a certain level. It is desirable to determine pixel areas with brightness values ​​below a certain level as shadow areas and extract the luminance normal vector.

[0079] At this time, for pixel areas of identical object candidates existing within a certain distance from two or more consecutive video images determined by the above-determined sampling time, depth information (depth) can be extracted by extracting 3D normal vectors and diffuse albedo, and the motion vector of each object can be determined. The motion vector is determined by selecting identical pixels from two or more consecutive image areas within the shadow area or important luminance normal vector area of ​​each object.

[0080] Step 111 determines variable resolution for each major shadow area.

[0081] In step 120, the image blur method is applied to the region of interest extracted in step 110 to more accurately extract the recognition region of the target object. In addition, depending on the importance of the object recognition region extracted in step 111, a resolution higher than the image resolution applied in step 110 is applied to convert the image into an RGB image, and then a blur kernel is applied to the region of each object in two or more consecutive images. In the region of an object with a motion vector within a certain level, blurring noise below a certain level occurs. Therefore, after applying the blur kernel, the object region with a blurring noise above a certain level is removed, and then a plurality of recognized object regions can be extracted and updated. In addition, the updated object region and camera viewpoint information can be used to extract a more accurate motion vector of the object. In step 111 or 120, the method of adjusting the resolution for each object region can be a variable rate shading (VRS) method, so that a variable image resolution can be applied according to the importance of each object. In addition, the above importance can be determined according to the purpose of video image analysis, such as collision prevention between surrounding objects, movement pattern analysis, and special situation analysis. In addition, since the importance of the object can be applied with variable resolution according to the importance during simulation analysis using a vehicle driving scenario, efficient simulation testing can be performed. For example, in step 21, if there is a vehicle with its emergency lights on and driving in an emergency situation as a data collection event, a high importance can be applied to an object area where the light change is very large in a short period, and the area can be converted into a high-resolution RGB image, and then step 111 or 120 can be performed. In addition, if an ambient sound event is collected in step 21, the importance of each object can be determined by considering the occurrence and direction of the vehicle horn, collision sound, siren, etc.Also, in these cases, the important object area is selected by calculating the vectors for the light and surrounding sound areas in advance at step 110.

[0082] The object recognition area subsets are managed based on the similarity of the pixel values ​​of the object area extracted in step 130.

[0083]

[0084] Figure 4 is an example of an image in which only objects with blurring noise below a certain level are extracted after blurring has been applied. Figure 4 is an example of objects extracted after blurring has been applied over time, and an example in which background elements are artificially excluded from the object recognition target. Blurring noise is determined based on the speed of the image capturing device and the difference in the movement speed of surrounding objects, and selective extraction is possible only for objects with blurring noise below a certain level.

[0085] Object tracking typically involves tracking objects across consecutive images. However, when two objects overlap, continuous tracking is difficult. However, object tracking is possible by classifying identical objects into identical subsets and tracking them over a period of time. Furthermore, by monitoring overlapping and obscured objects in real time for a set period of time, the presence of overlapping objects can be inferred.

[0086]

[0087] Figure 5 is a flowchart of a method for monitoring overlapping objects by managing information in the object recognition area in subset units.

[0088] In step 271, when the time and distance for monitoring overlapping objects are set, the object image subset management unit (200) selects monitoring objects by reflecting distance-based layer information. The distance-based layer information is stored as a layer by classifying recognized objects by a certain distance based on the camera viewpoint.

[0089] At step 272, a subset of the previously stored object recognition regions is called.

[0090] The method of updating the object recognition region subset in step 273 is performed as in step 130 of Fig. 3. However, the monitoring distance reflects the movement speed of the camera and the relative movement speed of surrounding objects, and is determined as a sampling time that satisfies the monitoring distance set in step 271.

[0091] Step 274 compares the recognition area of ​​the previous object, determined from the same subset, with the recognition area of ​​the current object, which has changed. Step 275 determines the object group that contains objects that overlap to a certain degree or more, based on the results of Step 274. Step 276 displays or provides a message regarding the location of the extracted overlapping objects in the system.

[0092] In step 274, the comparison of the changed object recognition area is determined through an affine transformation. Specifically, the affine transformation is applied to the previously recognized object recognition area using the center of the previous luminance normal vector and the center of the luminance normal vector of the current object recognition area. At this time, the scale is calculated by comparing the actual recognized current object area with the previous object. Furthermore, the scale of the target object is compared with the scale of other nearby objects. If the scale difference exceeds a certain level, the current object is determined to overlap with another object in step 274. The scale difference threshold in step 274 can vary depending on the driving conditions. For example, in rainy or snowy environments with poor visibility, the scale change value of the object can be set larger. Furthermore, the object to be compared for scale change can be selected from objects with the same layer information, classified as the closest object to the target object or at a similar distance. Therefore, the monitoring results for overlapping objects are output in step 276, and overlapping objects are monitored in the same object recognition subset for the monitoring time set in step 271.

[0093] The affine transformation of the object applied in Fig. 5 is as shown in the mathematical expression 1 below. If a and b are defined as shown below, x' y' can be defined, and it can be calculated by changing it into a determinant as shown below.

[0094]

[0095] In a method for integrating video and geometry data captured by various devices, the texture of objects in other images can be adjusted based on the texture information of a reference image. At this time, the reference image is selected as the image with the lowest image resolution, geometric distortion, and camera distortion. In order to combine various images, the stored camera viewpoint and distance layer of the subset are used, and the subset containing each object is determined by matching it with the subset of the reference video image based on the object's location information, and the object with the highest resolution and the lowest blur noise in the continuous video images is determined as the reference object based on the subset of the reference video image, and the pixel statistics (the average pixel value and variance value by distance) of the reference object and the object recognition area are used to adjust the pixel values ​​of objects captured at different times and included in the same integrated subset to have similar statistics to the pixel statistics of the reference object. At this time, the camera viewpoint of the reference image containing the reference object is also considered, and after parallax adjustment for the objects included in the stereo area, the textures of the objects included in the mapped subset are changed.

[0096]

[0097] FIG. 6 is a flowchart showing a method for combining images by an image combining unit (300) according to another embodiment of the present invention.

[0098]

[0099] Step 310 converts objects from various images into absolute coordinates and integrates them into a single coordinate system. Objects for 3D modeling are stored as voxel structures containing 3D point cloud information. The voxel structure is hierarchical, and as you move from low resolution to high resolution, the amount of voxel information supporting LoD increases.

[0100] Step 311 determines a position-specific subset of each object with the same coordinate system in the voxel structure. Each subset group includes a reference object, and the texture of the subset group is modified using the luminance normal vector and pixel statistics of the reference object extracted in step 322.

[0101] In step 321, the reference object is selected as the object with the highest image resolution in each subset, existing in the stereo area of ​​the camera, and having the smallest blur noise between consecutive images of the same subset. Small blur noise means that the distortion of the camera and the distortion information of the height value acquired by the 3D depth sensor in consecutive time periods can be assumed to be the smallest, and also that the change in momentum can be small, so that it can be assumed that it is moving in a translational direction (maintaining the same direction of movement). In addition, since the reference object is determined on a subset basis, the reference frame of one subset is the frame containing the object with the smallest blur noise in two consecutive frames, and once the reference object of the reference frame is selected, the same object can be selected as the reference object in all other image frames of the subset. In addition, different reference objects may exist in each frame of a subset corresponding to a group of consecutive images.

[0102] Step 322 determines the luminance normal vector and pixel statistics of the reference object in each subset. The pixel statistics correspond to the mean or variance of the luminance normal vector. Therefore, it provides information that can determine the degree of change in the luminance normal vector of the object surface.

[0103] In step 323, if the recognition area between each object in each video frame of the subset has a certain level of overlap with respect to the camera viewpoint containing the reference object, it is determined to be an overlapping object and classified as an object that does not require rendering.

[0104] Step 324 is a step for correcting the spatial pose of objects included in the subset group to remove parallax between images captured from different camera viewpoints. The correction amount is based on the viewpoint of the reference image containing the reference object, and the 3D pose of the new object is corrected by referring to the motion vector of the closest object within the matched subset. The objects of each subset are stored in a 3D voxel structure, so that 3D rotation can be easily performed using the 3D information of the object. The image distortion of the object caused by the rotation amount correction can be corrected by using the texture of the object using the geometric information (height information) included in the voxel. When correcting the pose of each object, the light reflected on the surface of the object is removed and then rendered.

[0105] Step 325 is a step of modifying the texture of the same subset based on the statistics of the reference object, and is a step of adjusting the pixel statistics of each object included in the subset to be similar to the pixel statistics of the reference object. Therefore, it corresponds to a step of increasing the pixel similarity of all objects included in the same subset. Step 325 is a step of modifying the luminosity normal vector of a new object to be similar within a certain level by referring to the luminosity normal vector of the nearby object determined in step 324, thereby modifying the light reflection direction and the shadow position of each object. At this time, when selecting the shadow position, the stereo area can generate natural shadows between objects by considering the camera viewpoint. In addition, the luminosity normal vector according to the position of the objects can be adjusted in detail within a certain level by considering the relative distance from the reference object.

[0106] Using a 3D virtual volumetric video object (VVO) in a reference image, the virtual object can be inserted into a 2D image. This requires natural adjustment of the virtual object's position, pose, and luminance normal vector. Therefore, a subset of the 2D / 3D image that will contain the virtual object is selected, and the luminance normal vector and statistical information of the reference object in that subset are used for texturing. Afterwards, the 3D virtual object is projected onto a 2D plane and rendered using the camera viewpoint of the reference subset.

[0107]

[0108] Figure 7 is a flowchart of a method for a virtual object creation unit (400) to create a virtual object.

[0109] Step 411 is a step for determining a 3D virtual object to be inserted into a 2D image. The virtual object is a 3D virtual object (virtual volumetric video object), and free pose control and texture modification of the virtual object are easy.

[0110] Step 412 is the step for determining the location where the virtual object will be created. Since the objects in the 2D image are analyzed as stereoscopic images, and the recognition area and geometry information of each object are stored in a subset, once the location of the virtual object is determined, the information necessary for configuring the subset of the virtual object is stored through the object-specific image subset management unit (200). Furthermore, once the layer information is determined along with the location of the virtual object, layer information for each distance in the subset is stored, thereby determining the size of the object area of ​​the virtual object.

[0111] Step 413 is a step for selecting a reference subset to include virtual objects and determining a reference object for the subset. Once the virtual object creation location is determined in step 412, the reference object of the subset containing the closest object is determined as the reference object for the virtual object. Alternatively, as in step 321 of FIG. 6, an object with less photo distortion and geometry distortion among objects within a certain distance from the virtual object location can be determined as the reference object.

[0112] Step 431 is the step of determining the luminance normal vector and pixel statistics of the reference object. The pixel statistics correspond to the mean or variance of the luminance normal. Therefore, it is information that can determine the degree of change in the luminance normal vector according to the surface of the object.

[0113] Step 432 determines the 3D rotation amount of the virtual object by considering the change in the motion vector of the reference object or the closest object in the same subset, and corrects and renders the pose of the virtual object. If the nearby object exists in the stereo area of ​​the reference image, it has a motion vector value with the parallax corrected in the video image, enabling more accurate pose correction of the virtual object.

[0114] Step 433 is a step for modifying the texture of the virtual object based on the statistical values ​​of the reference object. This step is to adjust the pixel statistics of the virtual object to be similar to the pixel statistics of the reference subset, thereby increasing the overall pixel similarity of the 2D image. After adjusting the texturing of the virtual object, the luminance normal vector of the virtual object is adjusted based on the luminance normal vector value of the reference object.

[0115] Step 434 is a step for projecting the textured 3D virtual object onto a 2D plane and rendering it. Rendering considers the layer information of a subset of multiple objects existing in the image, starting with the object that exists farthest away, and then considering the layer information stored in the subset of virtual objects, rendering is performed in the area of ​​the virtual object determined in step 412. At this time, the position of the shadow of the virtual object can be created by rendering the stereo area considering the camera viewpoint, so that a natural shadow can be generated. In addition, as in step 323 of Fig. 6, overlapping objects can be classified and excluded in advance during rendering.

[0116] To insert a virtual background into a reference image, the background can be divided into cases where the background has an infinite focal length of the camera or a fixed focal length. In the former case, the reference background can be deleted and a new background can be inserted. In the latter case, the background is inserted by projecting it into 2D using a 3D virtual object. In addition, the pixel statistics of all objects in the image can be adjusted using the reference object information of the subset that includes the modified virtual background. In addition, since a light environment matching the environment of the virtual background must be created, the luminosity normal vector of each subset must be artificially modified. To overcome the difficulty, the distance and layer information between each object are used to group them, and then they are integrated into sub-group voxels within the subset. Then, the luminosity normal vector of the reference object of the sub-group is used to apply the luminosity normal vector to all objects included in the sub-group voxels. Therefore, among the objects included in one subset, objects having a luminance normal above a certain level are selected as reference objects, and objects close to the reference objects within a certain distance are classified into sub-subsets, thereby adjusting the luminance normal vector on a sub-subset basis.

[0117]

[0118] Figure 8 is a flowchart of a method for a virtual object creation unit (400) to create a virtual background.

[0119] Step 411 is the step of selecting a virtual background to be inserted into the captured 2D image. The virtual background is a 3D virtual object (virtual volumetric video object), and it is easy to freely control the pose of the virtual background and modify its texture.

[0120] Step 442 is the step for determining the location for generating the virtual background. Since the object of the 2D image is analyzed as a stereoscopic image, and the recognition area and geometry information of each object are stored in a subset, when the location of the layer where the virtual background is to be placed is determined, the information necessary for configuring the subset of the virtual background is stored through the object-specific image subset management unit (200). In addition, the size of the relative object area of ​​the virtual background is determined using the layer information for each distance of the subset. In addition, the posture correction of the virtual background is performed by considering the posture information of the 3D viewpoint of each video image frame to correct the posture of the virtual background.

[0121] Step 443 determines the pixel statistics of the virtual background. This corresponds to the average or variance of the pixels that make up the virtual background.

[0122] Step 444 is similar to step 413 of Fig. 7. Like step 321 of Fig. 6, blur noise is calculated on consecutive video images, and a reference object candidate having blurring noise below a certain level is determined. Then, the object with the smallest photo distortion and geometric distortion in each subset is determined as the reference object.

[0123] Step 445 is the step of determining the luminance normal vector and pixel statistics of the reference object. The pixel statistics correspond to the mean or variance of the luminance normal vector. Therefore, it is information that can determine the degree of change in the luminance normal vector according to the surface of the object.

[0124] Step 446 calculates the difference between the pixel statistics of the virtual background and the reference object of each subset. Step 447 uses the difference in the pixel statistics of each subset to modify the pixel statistics of the objects included in each subset so that all objects in each subset have similar statistics to the virtual background. In this step, the luminance normal vector of the objects included in each sub-subset is adjusted using the luminance normal vector of the reference object in each sub-subset.

[0125] Step 448 is the step for projecting the textured virtual object onto a 2D plane and rendering it. Rendering considers layer information of multiple object subsets within the image, starting with the farthest object, and then considering layer information stored in the subset of virtual objects. At this time, the position of the shadow of the virtual object can be created by rendering the stereo area while considering the camera viewpoint, thereby creating natural shadows. In addition, as in step 323 of Fig. 6, overlapping objects can be classified and excluded in advance during rendering.

[0126]

[0127] The ISO / DIS 34503 standard provides an ontology for object configuration and attribute information for designing driving scenarios. Furthermore, when a user designs a driving scenario to prevent collisions between vehicles, the semantic relationships between customized objects are defined. The driving scenario simulator determines the likelihood of a collision based on a defined set of driving scenario rules, then displays user actions and executes system action events to prevent accidents. However, image machine learning models primarily use convolutional neural network (CNN) learning to extract object features for object recognition. However, driving scenarios require defining a series of relationships based on object features, requiring vector information in the form of a graph (relationships between objects). Graph neural network (GNN) learning is an appropriate machine learning method for this purpose. Furthermore, analyzing real-world data to define relationships between objects and then generating driving scenarios is expensive. Therefore, to address these issues, object recognition embedding vectors can be virtually generated and used as driving scenario machine learning training data. Furthermore, the driving scenario data can be augmented to perform machine learning training. At this time, GNN can be applied as the driving scenario training method, and GAN training can be applied as a method for augmenting the driving scenarios. Therefore, after virtually generating object recognition information and driving scenario training data, the user can select the final driving scenario and then retrain the final driving scenario using the GNN method, thereby increasing the reliability of the driving scenario ML model.

[0128] Figure 9 is a diagram illustrating the relationship between a data acquisition device, a driving scenario machine learning learning and model generation server, and an object recognition machine learning learning and model generation server.

[0129] The data acquisition device (810), or edge device, is a device that acquires data. The data acquisition device (810) may be a device equipped with a camera, lidar, or other sensor, and may be configured as a vehicle, robot, surveillance camera, drone, etc. The edge device applies the machine learning model generated by the object recognition machine learning learning and model generation server (820) and the driving scenario machine learning learning and model generation server (830) to recognize objects and operate by reflecting the driving scenario required for autonomous driving. Therefore, each device is configured with a communication network via the Internet or Wi-Fi. The object recognition machine learning learning and model generation server (820) and the driving scenario machine learning learning and model generation server (830) improve the ML model as a data server or application server and transmit the improved machine learning model to the edge device. The model transmission applies OTA (Over The Air) technology.

[0130] Data (841, 842) transmitted and received between the data acquisition device (810) and the object recognition machine learning learning and model generation server (820) or between the data acquisition device (810) and the driving scenario machine learning learning and model generation server (830) is transmitted when each server (820, 830) receives a video image or 3D point information (point cloud), sensor information and important events (step 22 of FIG. 3), the type of object, the association information between objects matching the recognized driving scenario and the classified importance from the data acquisition device (810), or when each server transmits a machine learning model to multiple edge devices by the OTA method.

[0131] When each server (820, 830) receives a video image, if an object is recognized through an object recognition model on an edge device, the video image or a previous video image is transmitted to each server.

[0132] Data (843) transmitted from the object recognition machine learning learning and model generation server (820) to the driving scenario machine learning learning and model generation server (830) is a case where the object recognition machine learning learning result or recognized object information is transmitted.

[0133] The data (845) transmitted from the driving scenario machine learning training and model generation server (830) to the object recognition machine learning training and model generation server (820) is a feedback signal. Since the feedback signal includes information on objects that do not satisfy the driving scenario as a result of the driving scenario training, the object recognition machine learning training and model generation server (820) can use the information to modify the object recognition embedding vector and perform retraining.

[0134] When the machine learning driving scenario is augmented and trained using the GAN learning method in the driving scenario machine learning learning and model generation server (830), artificial noise is generated in the association information (link information) between objects according to the weights applied to the generator of the driving scenario machine learning learning and model generation server (830) for training. For example, when the driving scenario machine learning learning and model generation server (830) performs GAN learning, if the weight of the generator is large, noise can be increased when generating the object association information embedding vector, thereby reducing the result value of the generator's loss function. In addition, if the objective function of the driving scenario learning is to prevent collisions between vehicles, false link information can be generated so as not to recognize cases with a high possibility of collision. Therefore, in order to perform an effective vehicle collision avoidance simulation, information between objects with a high probability of collision is required, and the information is generated as a virtual object recognition embedding vector by the object recognition machine learning learning and model generation server (820), and information on sections with a high probability of collision is transmitted from the object recognition machine learning learning and model generation server (820) to the driving scenario machine learning learning and model generation server (830), and the driving scenario machine learning learning and model generation server (830) augments the driving scenario data so that, during GAN learning, the learning rate of the discriminator can be increased for sections with a high probability of collision in order to determine collision avoidance and false driving information.

[0135] Edge devices update their object recognition machine learning models and driving scenario machine learning models via over-the-air (OTA) from their respective service servers. Edge devices provide real-time generated object IDs and driving scenario IDs to multiple servers, allowing each server to determine and verify the machine learning learning rate.

[0136] Each service server modifies the embedding vector and updates the machine learning model through feedback signals between each other, and then transmits it to the edge device.

[0137]

[0138] Figure 10 is a detailed description of the driving scenario verification unit (500), and is a flowchart showing a method for creating a driving scenario machine learning model.

[0139] Step 511 defines the features needed to recognize objects. For example, this step defines image features needed to recognize objects such as cars, people, lanes, guardrails, streetlights, and sidewalks.

[0140] Step 512 is a step of configuring and storing the object recognition area as a subset in order to quickly recognize an object in a captured video image. When an object is recognized by a data acquisition device (810), i.e., an edge device, the image is transmitted to an object recognition machine learning learning and model generation server (820) and analyzed in real time. In FIG. 9, this is data (841) transmitted and received between the data acquisition device (810) and the object recognition machine learning learning and model generation server (820).

[0141] Step 513 is the step of matching the subset and object recognition features to create machine learning training data and saving it as an object recognition embedding vector.

[0142] Step 514 is a step for creating an object recognition machine learning (ML) model by applying convolutional neural network (CNN) learning. Accordingly, the acquired video image is processed through steps 511 to 513, and the object recognition machine learning model created in step 514 is applied, thereby performing a step (515) for automating object recognition using the object recognition machine learning model on an edge device. As in the data (841) transmitted and received between the data acquisition device (810) of FIG. 9 and the object recognition machine learning training and model generation server (820), when an object recognition ML model is transmitted OTA from the server (820) to the edge device (810), the edge device (810) operates the ML model to recognize an object in real time, and transmits the corresponding video image and event information to the server (820). As in step 512, the server (820) assigns an ID to an object extracted from a series of images, stores the object group of the series of images as one subset, and thereby manages object information in units of subsets.

[0143] Step 521 defines the driving scenarios to be used in machine learning training. For example, if a collision avoidance scenario is defined, a vehicle to the left of the current vehicle may cross the lane and enter the current vehicle's path. In this case, the vehicle can slow down to prevent an accident. This series of situations for collision avoidance is defined as a driving scenario. Video images are analyzed to identify objects, and association information between the identified objects is matched to the driving scenario. This association information between the recognized objects is then stored in a graph database.

[0144] Step 531 constructs a graph embedding vector by storing a series of situations matching the defined driving scenario in a graph database. To effectively build a graph database, objects and their attribute information, the ontology structure between objects, and driving scenario rules that satisfy various driving situations must be defined in advance.

[0145] Step 541 is a step for generating an ML model capable of recognizing driving scenarios by performing graph neural network (GNN) learning using the above-mentioned graph embedding vector. In addition, after augmenting various driving scenarios using ontology information between objects and a GAN machine learning learning method, a user selects the driving scenarios extracted by the GAN model, performs GNN machine learning learning using the selected driving scenarios, generates a driving scenario ML model, and transmits the generated driving scenario ML model to an edge device so that an autonomous driving function that can prevent vehicle collisions of driving vehicles can be applied.

[0146] Figure 11 is a flowchart illustrating a method for creating a virtual object and verifying a driving scenario.

[0147] FIG. 11 is a flowchart illustrating a method of simulating before generating an image with a virtual object inserted by inserting a virtual object into an actual image when the object recognition ML model and the driving scenario ML model are satisfied by inputting virtual object information into the object recognition embedding vector information in step 513 of FIG. 10.

[0148] Step 516 is the step of virtually constructing an embedding vector for a virtual object that does not exist in the actual image. Objects with defined characteristics can be selected and converted into virtual object vector information, or randomly input virtual object information to augment object information. In the latter case, by reflecting subset information of the object, the distance, area, and layer information of the recognized object are considered. Even though the image information of the virtual object does not exist in the actual image, the location and subset information of the virtual object can be generated, and then the virtual object recognition embedding vector can be defined.

[0149] Step 517 applies an object recognition machine learning (ML) model with a convolutional neural network (CNN) learning, and then step 518 checks the object recognition result to configure an optimal virtual object recognition embedding vector. After applying the object recognition machine learning model in step 517, a feedback signal is provided to step 516, which configures a virtual object recognition embedding vector in step 518, using the calculated object recognition probability value, so that the virtual object recognition embedding vector configuration can be optimized.

[0150] When the driving scenario to be applied is determined in step 522, step 532 is a step of constructing a graph embedding vector by extracting information between objects matching the driving scenario using the driving scenario rule and object ontology information selected in step 522 for objects that satisfy the object recognition ML model in step 518.

[0151] Step 542 applies a driving scenario recognition ML model generated using a graph neural network (GNN) machine learning method to process the graph embedding vector generated in Step 532. Step 543 verifies the driving scenario recognition results. Furthermore, the driving scenario recognition probability value calculated in Step 542 can be used to provide a feedback signal to Step 532 to optimize the configuration of the graph embedding vector.

[0152] Therefore, for the virtual object recognition embedding vector virtually input in step 516, for the virtual object information that satisfies the object recognition ML model (518) and the driving scenario recognition ML model (542), the virtual object is inserted into the actual captured image video in step 544.

[0153] The method for performing rendering for inserting a virtual object is as shown in Fig. 7 or Fig. 8. Therefore, by simulating the reliability of virtual object insertion using an object recognition and driving scenario ML model and then inserting the virtual object, a video image with a highly reliable virtual object inserted can be generated.

[0154] When training driving scenarios for collision avoidance by inserting virtual objects into video images, defining the desired baseline driving scenario requires the user to understand and define the relationships between all objects, creating a challenge. To address this issue, random driving scenarios can be generated based on ontology information defining inter-object relationships and defined driving scenario rules, and valid driving scenarios can be extracted through machine learning.

[0155] Therefore, by applying CGNN (convolutional GNN) machine learning learning using driving scenario data extracted from actual image data and driving scenarios virtually generated and extracted through machine learning learning, an ML model can be created. At this time, a generative adversarial network (GAN) learning method can be used to generate various driving scenarios, and the GAN is trained by dividing it into a generator and a discriminator. The generator can generate true or false links by reflecting the driving scenario links with predefined ontology information and the rules of the reference driving scenario extracted from actual data, and the discriminator learns to determine the false links generated by the generator so that the result of each loss function (LF) satisfies the reference value. At this time, when the objective function of the driving scenario is to prevent collisions between vehicles, the weight of the generator is increased to lower the safety of vehicle driving, and conversely, to increase safety, the weight of the generator is decreased and applied. In addition, since driving scenarios with a high probability of collision require user action of the reference vehicle (e.g., vehicle speed control), the learning rate (training epoch) of the discriminator can be increased for training in the above section. Various driving scenarios can be randomly generated using the GAN (generative adversarial network) method, and new driving scenario rules that can reduce the probability of collision can be extracted using the analysis results of true and fake driving scenarios through the discriminator. In addition, it is necessary to generate virtual object recognition embedding vectors (distance information and related attribute information between objects) by considering the weights applied to the generator, and for situations with a high probability of collision (situations requiring vehicle control), the learning rate of the discriminator is increased to conduct effective training.

[0156] Mathematical formula 2 is the formula for the GAN learning method.

[0157]

[0158] The meaning of each function symbol is as follows.

[0159] D(x): The probability value that the discriminator can infer the actual link data x as the true value.

[0160] Ex: Expected value of all actual link data

[0161] G(z): Output value of the generator when noise is z

[0162] D(G(z)): The probability value that the discriminator can infer a false link as a true value.

[0163] Ez: Expected values ​​for all false link data (all expected values ​​that will be fed into the generator)

[0164] The generator is a process that minimizes the function of the above mathematical expression 2, while the discriminator is a process that maximizes it.

[0165] Minimizing the loss function of the generator is specifically minimizing log(1-D(G(z)). Alternatively, a modified method minimizes log D(G(z)).

[0166] Figure 12 is a configuration diagram of a device that learns a driving scenario using a GAN machine learning method.

[0167] The GAN machine learning method consists of a generator (620) and a discriminator (650), and the learning data provided to the discriminator consists of an embedding vector of a reference driving scenario and a false or semi-driving scenario embedding vector generated by the generator (620). The semi-driving scenario embedding vector is data generated by reflecting ontology information between objects and the reference driving scenario, and is thus distinguished from false driving scenario data.

[0168] Accordingly, if the semi-driving scenario generated by the generator (620) is classified as a true value by the discriminator (650), and the loss function value of the discriminator (650) satisfies a certain level, the semi-driving scenario can be updated to a true driving scenario. The attribute map of the standard / false / semi of each driving scenario generated by the generator (620) is used to calculate the loss function of the loss function calculation unit (660).

[0169] The discriminator (650) guesses the attribute information of standard / false / semi for the provided driving scenario.

[0170] The loss function calculation unit (660) compares the predicted value of the discriminator (650) with the attribute map provided by the generator (620) to calculate the loss function of the discriminator (650). The attribute map can be classified into true / false / neutral, and the loss function is determined by comparing the predicted value of the discriminator (650) with the attribute map provided by the generator (620), and the machine learning parameters of the discriminator (650) are updated based on the result.

[0171] When training the discriminator (650), the above attribute map can be used to retrain the prediction error of the driving scenario, and in cases where the driving scenario is different from the reference driving scenario, different from the driving scenario rule provided by the user, or in a vehicle collision-prone section defined by the user, the learning rate of the discriminator (650) can be increased for training. When the loss function value of the discriminator (650) is determined, the loss function value of the generator (620) is also determined. At this time, the final loss function value can be determined by applying a scale value to the loss function value of the generator (620) or the discriminator (650). The role of the scale value is to change the weights of the generator (620) and the discriminator (650), thereby inducing machine learning training in a desired direction. For example, if the scale value of the generator (620) is increased, the generation of false data of the generator (620) may increase, and the discrimination ability of the discriminator (650) may decrease.

[0172] The loss function calculation unit (660) transmits a feedback signal (backpropagation) to the generator (620) for adjusting the generator parameter values. The value of the false / semi-driving scenario of the generator (620) can be changed or increased / decreased based on the loss function result value of the generator (620). For example, if the loss function value of the generator (620) is greater than a certain level, in order to lower the loss function value, a semi-driving scenario vector can be additionally generated or the vector value of the false driving scenario can be decreased / excluded. In addition, in order to control the noise of the generator (620) at a certain level, the generator (620) can apply a conditional data generation method, such as the generation of a semi-driving scenario that determines whether to generate false link information within a certain level by a rule defined in advance based on the information of the graph embedding vector.

[0173] In addition, if the loss function result of the discriminator (650) is below a certain level and the discriminator (650) determines that the semi-driving scenario provided as an input value is a true value, the semi-driving scenario determined to be a true value can be updated as the reference driving scenario. The update can update the reference driving scenario by reflecting the user's feedback.

[0174]

[0175] Figure 13 is a process of creating and applying a driving scenario ML model to extract virtual / reference objects that do not satisfy a certain level of driving scenarios.

[0176] Steps 516, 517, and 518 are identical to those in FIG. 11. Step 516 constructs an object recognition embedding vector or a virtual object recognition embedding vector by reflecting object information present in the virtual object and video image after recognizing the object. Step 517 calculates the probability value of the object recognition machine learning model for the reference / virtual object recognition embedding vector provided as an input value.

[0177] In step 535, using driving scenario machine learning, the object recognition embedding vector information constructed in step 516, the ontology information between objects, and the user-defined driving scenario rules (sematic relation) are reflected to determine links between objects that satisfy the driving scenario, and a graph embedding vector is constructed.

[0178] Step 546 is a step of calculating the probability value of the generated graph embedding vector by applying the driving scenario machine learning model generated by the GAN / GNN machine learning method in FIG. 11. In step 546, if the GAN machine learning method is applied, the machine learning model is a machine learning model generated by automatically generating and training various driving scenarios. Therefore, after selecting the driving scenarios extracted from the model results, the driving scenarios are retrained using the GNN learning method to generate and apply the final driving scenario model, so that a highly reliable machine learning model can be applied. It can be inferred that the links of the driving scenarios that do not satisfy the final driving scenario model results include information on virtual objects with a high probability of object recognition errors and collisions. Therefore, if the link information including the object recognition errors and collision possibilities is object information generated for an intentional vehicle collision simulation, it can be interpreted as intentional error data being recognized, and if it is an object generated for a collision avoidance simulation, it can be classified as an object requiring data modification and fed back to the object recognition embedding vector generation module. Therefore, by providing a feedback signal including link / object information that does not satisfy the driving scenario ML model of step 546 in step 547 to the reference / virtual object recognition embedding vector construction module, object recognition errors and driving scenario determination errors can be corrected. The feedback signal can be composed of a feedback signal between different servers (820, 830), like signal 845 of FIG. 9, and can also be composed of object recognition embedding vector information transmission between servers (820, 830), like signal 843.

[0179]

[0180] Figure 14 is a sequence diagram showing data transmission between a data acquisition device (810), an object recognition machine learning learning and model generation server (820), and a driving scenario machine learning learning and model generation server (830).

[0181] The data acquisition device (810) includes a machine learning model for object recognition and autonomous vehicle driving. To continuously update the machine learning model and improve learning efficiency, data transmission and feedback signals are appropriately provided between each device (810, 820, 830). Furthermore, a service is provided that enables testing of driving scenarios based on actual vehicle driving equipped with the data acquisition device (810).

[0182] As shown in Fig. 9, the data acquisition device (810) updates the object recognition machine learning model in real time via OTA (over-the-air) from the object recognition machine learning learning and model generation server (820), and the driving scenario machine learning model in the driving scenario machine learning learning and model generation server (830). The data acquisition device (810) provides key video images and event information related to object recognition and driving scenarios to each server (820, 830) in real time.

[0183] In step 710, the data acquisition device (810) generates an event using an object recognition machine learning model or a driving scenario machine learning model and transmits it to the object recognition machine learning training and model generation server (820).

[0184] In step 711, when a main object is recognized or a situation that matches the driving scenario rule is recognized by the machine learning model operating in the data acquisition device (810), the corresponding video image is transmitted to the object recognition machine learning training and model generation server (820). At this time, since the driving scenario must define connection information between at least two objects, the two machine learning models must be able to share object IDs. Sharing of object IDs can be processed so that different machine learning model threads can input / output data when the object ID is generated at the OS level, or data can be processed through a module that wraps the threads of the two machine learning models in the data acquisition device (810). In this case, in step 710, the data acquisition device (810) assigns an object ID to the recognized object, transmits the video image and the recognized object ID to the object recognition machine learning learning and model generation server (820), and transmits the object ID and the recognized driving scenario ID to the driving scenario machine learning learning and model generation server (830) to server-A / B.

[0185] In step 711, the data acquisition device (810) transmits a video image to the object recognition machine learning and model generation server (820).

[0186] Based on the video image and object information received from the data acquisition device (810) at step 720, the object recognition machine learning learning and model generation server (820) generates an object recognition embedding vector, and also processes the object recognition embedding vector by applying an object recognition machine learning model at step 721, and transmits the refined object recognition result to the driving scenario machine learning learning and model generation server (830) as at step 722.

[0187] In step 730, the driving scenario machine learning learning and model generation server (830) uses the object recognition embedding vector information received from the object recognition machine learning learning and model generation server (820) and the driving scenario ID and object information received from the data acquisition device (810) to generate a graph embedding vector containing connection information between objects matching the driving scenario rules, and then generates a new model using a GAN or GNN machine learning learning method. At this time, the driving scenario ID information received from the data acquisition device (810) can be used to increase the machine learning learning rate in the relevant section to effectively perform learning. The reason for increasing the learning rate is to efficiently relearn and verify the driving scenario information acquired using the driving scenario machine learning model in the data acquisition device (810).

[0188] In step 731, if the driving scenario ID received from the data acquisition device (810) is excluded from the learning data in order to satisfy the loss function of the newly generated machine learning model, the object information included in the excluded driving scenario is fed back to the object recognition machine learning learning and model generation server (820). The object recognition machine learning learning and model generation server (820) reflects the feedback signal, reconstructs the object recognition embedding vector, performs object recognition machine learning learning, and updates the newly generated machine learning model to the data acquisition device (810) through the OTA method in step 741. At this time, in step 742, the driving scenario machine learning learning and model generation server (830) also transmits the newly generated machine learning model to the data acquisition device (810).

[0189]

[0190] Embodiments of the present invention may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specifically designed and configured for the present invention, or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0191] As described above, the present invention has been described with specific details such as specific components and limited embodiments and drawings, but these are provided only to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments, and those with ordinary knowledge in the field to which the present invention pertains can make various modifications and variations from this description.

[0192] Therefore, the idea of ​​the present invention should not be limited to the described embodiments, and all things that are equivalent or equivalent to the claims described below as well as the claims are considered to fall within the scope of the idea of ​​the present invention.

[0193] The term '~ unit' used in this embodiment means a software or hardware component such as an FPGA (field-programmable gate array) or an ASIC, and the '~ unit' performs certain roles. However, the '~ unit' is not limited to software or hardware. The '~ unit' may be configured to be on an addressable storage medium and may be configured to play one or more processors. Thus, as an example, the '~ unit' includes components such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and '~ units' may be combined into a smaller number of components and '~ units' or further separated into additional components and '~ units'. Additionally, components and '~parts' may be implemented to regenerate one or more CPUs within a device or secure multimedia card.

[0194] All of the functions described above can be performed by a processor such as a microprocessor, controller, microcontroller, ASIC (Application Specific Integrated Circuit), etc., according to software or program code coded to perform the functions described above. The design, development, and implementation of the code will be apparent to those skilled in the art based on the description of the present invention.

Claims

1. In a method to support real-time machine learning model updates, A step in which an object recognition machine learning application or a driving scenario machine learning application receives events for important objects and hazard signals; A step in which the object recognition machine learning application receives a video image stored for a certain period of time from the time of occurrence of the event; A step in which the object recognition machine learning application generates a first object recognition embedding vector, extracts an object recognition probability value, and transmits a second object recognition embedding vector that satisfies a probability value higher than a certain level to the driving scenario machine learning application; A step in which the driving scenario machine learning application processes the second object recognition embedding vector as an input value to generate a first driving scenario embedding vector and calculates a probability value of the first driving scenario embedding vector; The driving scenario machine learning application includes a step of extracting the second driving scenario embedding vector having a probability value below a certain level, and transmitting a feedback signal to the object recognition machine learning application by referencing object information included in the second driving scenario. A method for supporting updating of a real-time machine learning model, characterized in that the object recognition machine learning application optimizes the first object recognition embedding vector by referring to the object information included in the feedback signal.

2. In paragraph 1, A method for supporting real-time machine learning model updates, characterized in that the object recognition machine learning model is a machine learning model generated through convolutional neural network (CNN) learning, and the driving scenario machine learning model is a machine learning model generated through graph neural network (GNN) learning.

3. In paragraph 1, A method for supporting real-time machine learning model updates, characterized in that the driving scenario machine learning model is a machine learning model trained using driving scenario training data provided by a generative adversarial network (GAN) machine learning model.

4. In paragraph 3, A method for supporting real-time machine learning model updates, characterized in that the driving scenario learning data provided by the above adversarial generative machine learning model is updated by adding a new driving scenario generated by combining the reference driving scenario learning data and an ontology reflecting hierarchical information between objects to the reference driving scenario learning data.

5. In paragraph 1, A method for supporting the update of a real-time machine learning model, characterized in that the learning rate of the object recognition machine learning application is increased for a certain period of time from the time of occurrence of the received event, and the learning rate of the driving scenario machine learning application is increased for training.

6. In paragraph 1, A method for supporting an update of a real-time machine learning model, characterized in that when retraining of the object recognition machine learning application is performed by optimizing the first object recognition embedding vector, retraining of the driving scenario machine learning application is performed.

7. In paragraph 6, A method for supporting real-time machine learning model updates, characterized in that when the object recognition machine learning application transmits an object recognition machine learning model updated through retraining of the object recognition machine learning application to the device that transmitted the event, the driving scenario machine learning application also transmits the driving scenario machine learning model updated through retraining of the driving scenario machine learning application to the device that transmitted the event.

Images