Method for generating machine learning model for vehicle driving scenarios

Abstract

The present invention relates to a method for generating a machine learning model for vehicle driving scenarios, the method comprising the steps of: recognizing features of an object in an image to generate a first object recognition embedding vector for machine learning training; performing, by using the first object recognition embedding vector as an input value, convolutional neural network (CNN) machine learning training and generating an object recognition machine learning model; generating a first graph embedding vector by using a second object recognition embedding vector used to generate the object recognition machine learning model and driving scenario rules that define the semantic relation between predetermined objects; and performing, by using the first graph embedding vector as an input value, graph neural network (GNN) machine learning training and generating a driving scenario machine learning model.

Description

How to Create a Machine Learning Model for Vehicle Driving Scenarios

[0001] The present invention relates to a method for generating a vehicle driving scenario machine learning model, and more particularly, to a method for generating a vehicle driving scenario machine learning model in which object recognition and driving scenarios are learned through separate machine learning, but the reliability of the machine learning model of the driving scenario is increased, and object recognition errors can be verified using semantic relations extracted by the model.

[0002] Combining image information and 3D height information is being applied to facility management, security technology, and autonomous driving technology development. Recently, there have been active attempts to develop automated models by generating and learning virtual information based on user-desired scenarios using artificial intelligence and machine learning technology.

[0003] 3D object information acquired from various sensors is stored to support resolution according to distance (level of detail: LoD), and the method of storing 3D data is divided into a method of finding image areas with high similarity within the same image (intra-frame prediction) and a method of finding similarity in different image frames (inter-frame prediction). Recently, technologies that provide 3D video images using video point cloud compression technology (V-PCC: video point cloud compression) and geometry-based point cloud compression technology (G-PCC: geometry-based point cloud compression) are being developed and applied.

[0004] V-PCC technology classifies 3D objects, stores the classified 3D objects as patch information projected onto a hexahedron of a 3D grid (voxel grid), converts them into 2D video images, and transmits them using H.265 and HEVC compression technologies.

[0005] A point cloud is composed of R / G / B or Y / Cb / Cr image pixel values ​​in 3D space, and in G-PCC, one object is stored in a voxel structure consisting of an occupancy map image for a 3D / 2D patch, a geometry image of the 3D patch, and a texture (color) image of the 3D patch, and the 3D object can be reconstructed and rendered by adjusting the image viewpoint and the photometric normal vector in a specific direction. However, errors occur when the coordinates of the 3D point used for the 3D patch projected on each plane are changed to 2D information, and distortion also occurs in the geometric information of the 3D point due to the difference between the resolution supported by the grid defined in the system and the actual expressed resolution. The luminance normal vector is a normal vector for a surface used in the photometric stereo method used to obtain information on the surface of an object when there are images obtained by observing the same object from at least three different light sources, and obtaining the normal vector allows one to know the surface shape of the object.

[0006] If the user's viewpoint coordinates are managed in 3D and the LoD and orientation information of the viewpoint of the provided video image are stored and managed, it is possible to analyze images previously transmitted to the user. To calculate the area of ​​a previously transmitted image, the image area is calculated based on the 3D grid coordinates of the viewpoint. At this time, the image area provided is determined according to the LoD. In addition, pixel values ​​change depending on the orientation angle of the user's viewpoint.

[0007] Creating automated models by analyzing video images requires defining driving situations and road characteristics. The Association for Standardization of Automation and Measuring Systems (ASAM) defines openODD (Operational Domain Design: ODD), which categorizes object attribute information into relations, contexts, actions, events, and objects. It also provides an ontology structure between objects, enabling the creation of standardized object information.

[0008] The standard for simulation-based testing to ensure the safety of autonomous vehicles (ISO / DIS 34503) configures object information into scenery elements, environmental conditions, and dynamic elements to define driving situation locations, weather, intersections, and traffic conditions.

[0009] Therefore, for autonomous driving and automation of various devices, video images and 3D height information acquired from various devices can be analyzed to classify and store object information as standardized data, and scenario-based simulation data can be generated to apply artificial intelligence and machine learning methods to create a machine learning model.

[0010] Accordingly, the problem to be solved by the present invention is to provide a method for generating a vehicle driving scenario machine learning model that can verify object recognition errors by using semantic relation information between objects extracted by the model, while object recognition and driving scenarios are learned through separate machine learning, thereby increasing the reliability of the machine learning model of the driving scenario.

[0011] In high-resolution video images, object recognition and analysis require significant time, and image rendering also requires high-performance systems. Therefore, the present invention proposes a method for rapidly recognizing objects in high-resolution video images and efficiently rendering images on low-performance systems.

[0012] The present invention provides a method for generating an object recognition machine learning model and a driving scenario machine learning model from a video image to achieve the above object recognition problem, the method comprising: recognizing an object feature in a video image to generate a first object recognition embedding vector for machine learning; performing convolutional neural network (CNN) machine learning using the first object recognition embedding vector as an input value, and generating an object recognition machine learning model; generating a first graph embedding vector using a second object recognition embedding vector used in generating the object recognition machine learning model and a driving scenario rule defining semantic relations between predetermined objects; and performing graph neural network (GNN) machine learning using the first graph embedding vector as an input value, and generating a driving scenario machine learning model.

[0013] According to one embodiment of the present invention, it is preferable that the graph neural network machine learning training generates a generative adversarial network (GAN) machine learning model using the first graph embedding vector as an input value, and generates the driving scenario machine learning model using the second graph embedding vector used to generate the adversarial generative model as an input value.

[0014] In addition, in the step of generating the second graph embedding vector, it is preferable to generate new graph information by combining hierarchical information (ontology) between predefined objects with some graphs of the first graph embedding vector and add it to the first graph embedding vector.

[0015] According to another embodiment of the present invention, it is preferable that the object recognition model generation is performed in a first server, the first server transmits the second object recognition embedding vector to a second server, and the second server generates the driving scenario machine learning model.

[0016] In addition, after generating the object recognition machine learning model and the driving scenario machine learning model, it is desirable to transmit the machine learning models to an edge device.

[0017] By using image blur and representative photometric normal vector, object recognition area can be quickly extracted from video image, and differential resolution can be defined at the object recognition stage according to the importance of the object, so that after object recognition, variable image resolution can be provided according to the importance of the object when rendering the image.

[0018] Recognized objects can be generated by extracting (A) 3D object information, (B) distance-based layer and alpha channel information of recognized objects, (C) stereo area, (D) pixel similarity information by image recognition area, and (E) luminance normal vector, and managing object information by group (subset) and effectively arranging virtual objects.

[0019] By utilizing 3D information generated from high-resolution images and object recognition information of a subset, low-resolution images can be converted into high-resolution images and texturing can be performed to combine the images.

[0020] A machine learning model is created by combining various machine learning methods to extract object recognition and driving scenarios, and driving scenario machine learning (ML) is performed by modifying and augmenting (adding virtual objects) object recognition information (object recognition embedding vector), and then the generated driving scenario ML model is applied to verify the object recognition embedding vector with added virtual objects, and if the model is satisfied, an actual virtual object image is created.

[0021] According to the present invention, by changing a high-resolution image into a low-resolution image and quickly selecting an object recognition area, it is possible to quickly determine an object recognition area, and by determining the resolution of an object to be rendered in advance according to the importance of the recognized object, the rendering speed can be improved.

[0022] In addition, according to the present invention, by managing object recognition image information by group, object recognition, virtual objects, and backgrounds can be easily added, and machine learning learning data can be effectively constructed, and after machine learning, a model is created and applied to augment a virtual object embedding vector to verify data reliability, and then data reliability can be increased by inserting a virtual object into an actual image.

[0023] Furthermore, according to the present invention, although object recognition and driving scenarios are learned through separate machine learning, the reliability of the machine learning model of the driving scenario can be increased, and object recognition errors can be verified using the sematic relation information between objects extracted by the model.

[0024] FIG. 1 is a configuration diagram of an object recognition and virtual object generation system according to one embodiment of the present invention.

[0025] FIG. 2 is a configuration diagram of an object recognition and virtual object generation system according to another embodiment of the present invention.

[0026] Figure 3 is a flowchart illustrating a method for determining an object recognition area according to one embodiment of the present invention.

[0027] Figure 4 is an example of an image in which only objects with blurring noise below a certain level are extracted after applying blurring.

[0028] [Correction pursuant to Rule 91 18.04.2024] Figure 5 is a flowchart showing the method by which the virtual object creation unit (400) creates a virtual object.

[0029] [Correction under Rule 91 18.04.2024] Figure 6 is a flowchart regarding the method by which the virtual object creation unit (400) creates a virtual background.

[0030] [Revised 18.04.2024 under Rule 91] Figure 7 is a diagram illustrating the relationship between a data acquisition device, a driving scenario machine learning learning and model generation server, and an object recognition machine learning learning and model generation server.

[0031] [Revised 18.04.2024 under Rule 91] Fig. 8 is a detailed description of the driving scenario verification unit (500), and is a flowchart showing a method for creating a driving scenario machine learning model.

[0032] The present invention relates to a method for generating an object recognition machine learning model and a driving scenario machine learning model from a video image, and is characterized by comprising the steps of: recognizing features of an object in a video image to generate a first object recognition embedding vector for machine learning; performing convolutional neural network (CNN) machine learning using the first object recognition embedding vector as an input value and generating an object recognition machine learning model; generating a first graph embedding vector using a second object recognition embedding vector used for generating the object recognition machine learning model and a driving scenario rule defining semantic relations between predetermined objects; and performing graph neural network (GNN) machine learning using the first graph embedding vector as an input value and generating a driving scenario machine learning model.

[0033] Hereinafter, with reference to the attached drawings, preferred embodiments of the present invention will be described in detail, enabling those skilled in the art to easily implement the invention. However, these embodiments are intended to illustrate the invention more specifically, and it will be apparent to those skilled in the art that the scope of the present invention is not limited thereby.

[0034] In order to clearly solve the problem to be solved by the present invention, the composition of the invention will be described in detail based on a preferred embodiment of the present invention with reference to the accompanying drawings. In assigning reference numbers to components in the drawings, the same reference numbers are assigned to the same components even if they are in different drawings, and it is made clear in advance that components in other drawings may be cited when necessary when describing the drawings. In addition, in explaining the operating principle of the preferred embodiment of the present invention in detail, if it is determined that a specific description of a known function or configuration related to the present invention and other matters may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

[0035] Terms including ordinal numbers, such as "second," "first," etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a second component may be referred to as a "first component," and similarly, a first component may also be referred to as a "second component." The term "and / or" includes a combination of multiple related items described herein or any of multiple related items described herein.

[0036]

[0037] FIG. 1 is a configuration diagram of an object recognition and virtual object generation system according to one embodiment of the present invention.

[0038] An object recognition and virtual object generation system according to one embodiment of the present invention comprises an object recognition area determination unit (100), an object-specific image subset management unit (200), an image combination unit (300), and a virtual object generation unit (400).

[0039] The object recognition area determination unit (100) extracts the object area by tracking the shadow area of ​​the video image.

[0040] The shadow region analysis unit (110) extracts the object region by tracking the shadow region at a low resolution. The shadow region analysis unit (110) converts a high-resolution image to a low resolution, and then quickly classifies only the shadow region, predicts the object region based on the shadow region, and analyzes the high-resolution region for the predicted region. In order to quickly extract the object recognition region, when the video image is converted to a low-resolution image, the blurring analysis unit (120) applies blurring to two or more consecutive images to select only objects with blurring noise below a certain level, thereby enabling quick object region selection.

[0041] The object-specific image subset management unit (200) configures a subset based on objects recognized from the entire image video and manages attribute information such as object type, color, and importance.

[0042] The object geometry information acquisition unit (210) acquires height information and edge information of the recognized object.

[0043] The stereoscopic image analysis unit (220) analyzes objects in the stereoscopic image area by considering both the camera viewpoint and the light normal vector. At this time, in image areas other than the stereo area, only the light normal vector is considered.

[0044] Additionally, the light normal vector unit (230) analyzes light and intensity direction information using the representative light normal vector of the object area.

[0045] Additionally, the pixel similarity analysis unit (240) analyzes the similarity information of pixel values ​​in the object recognition area. When creating a virtual object, the pixel similarity analysis unit (240) performs texturing by considering the pixel value similarity information of an existing object.

[0046] In addition, the layer information analysis unit (250) determines the presence / absence of an object by adjusting the distance-based object layer information and alpha channel information when creating a virtual object and performing machine learning. The information analyzed by the layer information analysis unit (250) is used to add the necessary virtual object. The alpha channel information is auxiliary data information separated from the color expression data for each pixel.

[0047] The image combining unit (300) uses high-resolution geometry to correct a low-resolution image, converts it back into a high-resolution image, and then combines various images into an image in the same coordinate system.

[0048] The object integration unit (310) converts objects of various images into absolute coordinates and integrates them into the same coordinate system.

[0049] The adjustment unit (320) determines a subset of each object having the same coordinate system in the voxel structure by position, and selects an object in each subset that has a high image resolution, exists in the stereo area of ​​the camera, and has the smallest blur noise between consecutive images of the same subset as a reference object.

[0050] The adjustment unit (320) determines the luminance normal vector and pixel statistics of the reference object of each subset, and determines the objects as overlapping if the recognition areas between the objects have an overlapping area of ​​a certain level or more. In addition, the adjustment unit (320) corrects the spatial pose of the objects included in the subset group to remove the parallax between images captured from different camera viewpoints, and modifies the texture of the same subset based on the statistics of the reference object.

[0051] The virtual object creation unit (400) uses a 3D virtual object to add a virtual object to a reference image.

[0052] Additionally, the virtual object placement unit (410) adds virtual objects using the scale of the object determined by distance to the reference image, alpha channel information, and nearby object information.

[0053] The virtual object posture determination unit (420) determines the virtual object posture using the camera viewpoint and stereoscopic image area according to the arrangement of the virtual object.

[0054] The texturing unit (430) performs texturing using the intensity and direction of light, shadow information, pixel similarity information of nearby objects, etc.

[0055]

[0056] FIG. 2 is a configuration diagram of an object recognition and virtual object generation system according to another embodiment of the present invention.

[0057] Fig. 2 is different from Fig. 1 in that a driving scenario verification unit (500) is added between the object-specific image subset management unit (200) and the virtual object creation unit (400).

[0058] The driving scenario verification unit (500) performs the function of adding a virtual object to an actual image if, before generating an image of a virtual object, it recognizes an existing object as a virtual object or satisfies the driving scenario machine learning model by updating only the object recognition embedding vector for the virtual object without modifying the image, and generates a driving scenario model by applying a GNN (graph neural network) learning method.

[0059] The object-specific recognition feature generation unit (510) defines object-specific recognition features based on OpenODD attribute information.

[0060] The driving scenario definition section (520) defines the driving scenario applied to the system, reflecting the ISO / DIS 34503 standard.

[0061] The graph embedding vector generation unit (530) generates a graph embedding vector using the recognized object recognition embedding vector and OpenODD ontology information. At this time, for efficient machine learning, the graph embedding vector can be generated by generating virtual driving scenario vector information.

[0062] The driving scenario machine learning model generation unit (540) generates a driving scenario machine learning model by applying the GNN learning method.

[0063] The driving scenario verification unit (550) verifies the applied driving scenario ML module, extracts abnormal links from among the driving scenarios defined by the user, and generates a feedback signal.

[0064] During the video image collection phase, key event information such as camera speed changes, ambient sound changes, and ambient illumination changes are stored, and the collected event information is utilized when determining the object recognition area. For fast object recognition, the image is converted to a low-resolution grayscale image, and then the shadow area or pixels with a brightness value above a certain level are determined to extract important light direction and brightness information. Based on the shadow area and important luminance normal vector area, the primary recognition area of ​​each object is selected, and then, according to the importance of each object, the primary recognition area of ​​the object is determined by variably changing the resolution of the RGB image (VRS), applying a blurring kernel, and removing noise. The determined pixel area of ​​each object is organized into subsets of object recognition areas of consecutive images using pixel similarity information of each area and managed, and each object information is managed for each subset.

[0065] Figure 3 is a flowchart illustrating a method for determining an object recognition area according to one embodiment of the present invention.

[0066] Step 21 is the step where changes in ambient light brightness, changes in ambient noise, and changes in speed above a certain level occur on the edge device, and when a major object is recognized, it is saved as a major event.

[0067] Step 22 is a step in which the edge device transmits video images including the stored main events and events to the cloud server at regular intervals.

[0068] In step 101, the cloud server determines the sampling time of the video image by considering the main event information stored in 22 (the time of change in the brightness of the ambient light and the change in the ambient sound and the camera movement speed). For example, if the camera movement speed is 80 km / h, it moves approximately 22 m / s, so if there is an object with a possibility of collision in the vicinity, the sampling time can be determined at a cycle of 1 / 10 second or shorter to determine the object recognition area. Therefore, using the camera movement speed event stored in step 21, if the movement speed is high, the sampling interval of the video image in the time domain should be shortened, and if the movement speed is low, the sampling interval should be relatively long. In addition, if the brightness of the ambient light and noise change beyond a certain level, it can be used as a factor for adjusting the sampling time.

[0069] Step 102 is the step of converting a high-resolution image into a low-resolution image and adjusting it to grayscale for fast processing of video images.

[0070] Step 110 determines the region of interest where an object exists by selecting pixel areas with brightness values ​​above a certain level and pixel areas with brightness values ​​below a certain level. It is desirable to determine pixel areas with brightness values ​​below a certain level as shadow areas and extract the luminance normal vector.

[0071] At this time, for pixel areas of identical object candidates existing within a certain distance from two or more consecutive video images determined by the above-determined sampling time, depth information (depth) can be extracted by extracting 3D normal vectors and diffuse albedo, and the motion vector of each object can be determined. The motion vector is determined by selecting identical pixels from two or more consecutive image areas within the shadow area or important luminance normal vector area of ​​each object.

[0072] Step 111 determines variable resolution for each major shadow area.

[0073] In step 120, the image blur method is applied to the region of interest extracted in step 110 to more accurately extract the recognition region of the target object. In addition, depending on the importance of the object recognition region extracted in step 111, a resolution higher than the image resolution applied in step 110 is applied to convert the image into an RGB image, and then a blur kernel is applied to the region of each object in two or more consecutive images. In the region of an object with a motion vector within a certain level, blurring noise below a certain level occurs. Therefore, after applying the blur kernel, the object region with a blurring noise above a certain level is removed, and then a plurality of recognized object regions can be extracted and updated. In addition, the updated object region and camera viewpoint information can be used to extract a more accurate motion vector of the object. In step 111 or 120, the method of adjusting the resolution for each object region can be a variable rate shading (VRS) method, so that a variable image resolution can be applied according to the importance of each object. In addition, the above importance can be determined according to the purpose of video image analysis, such as collision prevention between surrounding objects, movement pattern analysis, and special situation analysis. In addition, since the importance of the object can be applied with variable resolution according to the importance during simulation analysis using a vehicle driving scenario, efficient simulation testing can be performed. For example, in step 21, if there is a vehicle with its emergency lights on and driving in an emergency situation as a data collection event, a high importance can be applied to an object area where the light change is very large in a short period, and the area can be converted into a high-resolution RGB image, and then step 111 or 120 can be performed. In addition, if an ambient sound event is collected in step 21, the importance of each object can be determined by considering the occurrence and direction of the vehicle horn, collision sound, siren, etc.Also, in these cases, the important object area is selected by calculating the vectors for the light and surrounding sound areas in advance at step 110.

[0074] The object recognition area subsets are managed based on the similarity of the pixel values ​​of the object area extracted in step 130.

[0075]

[0076] Figure 4 is an example of an image in which only objects with blurring noise below a certain level are extracted after blurring has been applied. Figure 4 is an example of objects extracted after blurring has been applied over time, and an example in which background elements are artificially excluded from the object recognition target. Blurring noise is determined based on the speed of the image capturing device and the difference in the movement speed of surrounding objects, and selective extraction is possible only for objects with blurring noise below a certain level.

[0077] Object tracking typically involves tracking objects across consecutive images. However, when two objects overlap, continuous tracking is difficult. However, object tracking is possible by classifying identical objects into identical subsets and tracking them over a period of time. Furthermore, by monitoring overlapping and obscured objects in real time for a set period of time, the presence of overlapping objects can be inferred.

[0078]

[0079] [Correction pursuant to Rule 91 18.04.2024] Figure 5 is a flowchart showing the method by which the virtual object creation unit (400) creates a virtual object.

[0080] Step 411 is a step for determining a 3D virtual object to be inserted into a 2D image. The virtual object is a 3D virtual object (virtual volumetric video object), and free pose control and texture modification of the virtual object are easy.

[0081] Step 412 is the step for determining the location where the virtual object will be created. Since the objects in the 2D image are analyzed as stereoscopic images, and the recognition area and geometry information of each object are stored in a subset, once the location of the virtual object is determined, the information necessary for configuring the subset of the virtual object is stored through the object-specific image subset management unit (200). Furthermore, once the layer information is determined along with the location of the virtual object, layer information for each distance in the subset is stored, thereby determining the size of the object area of ​​the virtual object.

[0082] Step 413 is the step of selecting a reference subset to include virtual objects and determining the reference object of the subset. Once the virtual object creation location is determined in step 412, the reference object of the subset containing the closest object is determined as the reference object of the virtual object. Alternatively, among objects existing within a certain distance from the virtual object location, an object with less photo distortion and geometric distortion can be determined as the reference object.

[0083] Step 431 is the step of determining the luminance normal vector and pixel statistics of the reference object. The pixel statistics correspond to the mean or variance of the luminance normal. Therefore, it is information that can determine the degree of change in the luminance normal vector according to the surface of the object.

[0084] Step 432 determines the 3D rotation amount of the virtual object by considering the change in the motion vector of the reference object or the closest object in the same subset, and corrects and renders the pose of the virtual object. If the nearby object exists in the stereo area of ​​the reference image, it has a motion vector value with the parallax corrected in the video image, enabling more accurate pose correction of the virtual object.

[0085] Step 433 is a step for modifying the texture of the virtual object based on the statistical values ​​of the reference object. This step is to adjust the pixel statistics of the virtual object to be similar to the pixel statistics of the reference subset, thereby increasing the overall pixel similarity of the 2D image. After adjusting the texturing of the virtual object, the luminance normal vector of the virtual object is adjusted based on the luminance normal vector value of the reference object.

[0086] Step 434 is the step of projecting the textured 3D virtual object onto a 2D plane and rendering it. Rendering considers the layer information of a subset of multiple objects existing in the image, starting with the object that exists farthest away, and then considering the layer information stored in the subset of virtual objects, rendering is performed in the area of ​​the virtual object determined in step 412. At this time, the position of the shadow of the virtual object can be created by rendering the stereo area considering the camera viewpoint, so that natural shadows can be generated. In addition, overlapping objects can be classified in advance and excluded during rendering.

[0087] To insert a virtual background into a reference image, the background can be divided into cases where the background has an infinite focal length of the camera or a fixed focal length. In the former case, the reference background can be deleted and a new background can be inserted. In the latter case, the background is inserted by projecting it into 2D using a 3D virtual object. In addition, the pixel statistics of all objects in the image can be adjusted using the reference object information of the subset that includes the modified virtual background. In addition, since a light environment matching the environment of the virtual background must be created, the luminosity normal vector of each subset must be artificially modified. To overcome the difficulty, the distance and layer information between each object are used to group them, and then they are integrated into sub-group voxels within the subset. Then, the luminosity normal vector of the reference object of the sub-group is used to apply the luminosity normal vector to all objects included in the sub-group voxels. Therefore, among the objects included in one subset, objects having a luminance normal above a certain level are selected as reference objects, and objects close to the reference objects within a certain distance are classified into sub-subsets, thereby adjusting the luminance normal vector on a sub-subset basis.

[0088]

[0089] [Correction under Rule 91 18.04.2024] Figure 6 is a flowchart regarding the method by which the virtual object creation unit (400) creates a virtual background.

[0090] Step 411 is the step of selecting a virtual background to be inserted into the captured 2D image. The virtual background is a 3D virtual object (virtual volumetric video object), and it is easy to freely control the pose of the virtual background and modify its texture.

[0091] Step 442 is the step for determining the location for generating the virtual background. Since the object of the 2D image is analyzed as a stereoscopic image, and the recognition area and geometry information of each object are stored in a subset, when the location of the layer where the virtual background is to be placed is determined, the information necessary for configuring the subset of the virtual background is stored through the object-specific image subset management unit (200). In addition, the size of the relative object area of ​​the virtual background is determined using the layer information for each distance of the subset. In addition, the posture correction of the virtual background is performed by considering the posture information of the 3D viewpoint of each video image frame to correct the posture of the virtual background.

[0092] Step 443 determines the pixel statistics of the virtual background. This corresponds to the average or variance of the pixels that make up the virtual background.

[0093] [Revised 18.04.2024 under Rule 91] Step 444 is similar to step 413 of Fig. 5. In addition, blur noise is calculated on consecutive video images, and reference object candidates having blurring noise below a certain level are determined, and then the object with the smallest photo distortion and geometric distortion in each subset is determined as the reference object.

[0094] Step 445 is the step of determining the luminance normal vector and pixel statistics of the reference object. The pixel statistics correspond to the mean or variance of the luminance normal vector. Therefore, it is information that can determine the degree of change in the luminance normal vector according to the surface of the object.

[0095] Step 446 calculates the difference between the pixel statistics of the virtual background and the reference object of each subset. Step 447 uses the difference in the pixel statistics of each subset to modify the pixel statistics of the objects included in each subset so that all objects in each subset have similar statistics to the virtual background. In this step, the luminance normal vector of the objects included in each sub-subset is adjusted using the luminance normal vector of the reference object in each sub-subset.

[0096] Step 448 is the rendering step for projecting the textured virtual object onto a 2D plane. Rendering considers layer information from multiple subsets of objects within the image, starting with the farthest object, and then considering layer information stored in the subset of virtual objects. At this time, the position of the virtual object's shadow is rendered in the stereo area, taking into account the camera viewpoint, thereby creating natural shadows. Furthermore, overlapping objects can be classified and excluded in advance during rendering.

[0097]

[0098] The ISO / DIS 34503 standard provides an ontology for object configuration and attribute information for designing driving scenarios. Furthermore, when a user designs a driving scenario to prevent collisions between vehicles, the semantic relationships between customized objects are defined. The driving scenario simulator determines the likelihood of a collision based on a defined set of driving scenario rules, then displays user actions and executes system action events to prevent accidents. However, image machine learning models primarily use convolutional neural network (CNN) learning to extract object features for object recognition. However, driving scenarios require defining a series of relationships based on object features, requiring vector information in the form of a graph (relationships between objects). Graph neural network (GNN) learning is an appropriate machine learning method for this purpose. Furthermore, analyzing real-world data to define relationships between objects and then generating driving scenarios is expensive. Therefore, to address these issues, object recognition embedding vectors can be virtually generated and used as driving scenario machine learning training data. Furthermore, the driving scenario data can be augmented to perform machine learning training. At this time, GNN can be applied as the driving scenario training method, and GAN training can be applied as a method for augmenting the driving scenarios. Therefore, after virtually generating object recognition information and driving scenario training data, the user can select the final driving scenario and then retrain the final driving scenario using the GNN method, thereby increasing the reliability of the driving scenario ML model.

[0099] [Revised 18.04.2024 under Rule 91] Figure 7 is a diagram illustrating the relationship between a data acquisition device, a driving scenario machine learning learning and model generation server, and an object recognition machine learning learning and model generation server.

[0100] The data acquisition device (810), or edge device, is a device that acquires data. The data acquisition device (810) may be a device equipped with a camera, lidar, or other sensor, and may be configured as a vehicle, robot, surveillance camera, drone, etc. The edge device applies the machine learning model generated by the object recognition machine learning learning and model generation server (820) and the driving scenario machine learning learning and model generation server (830) to recognize objects and operate by reflecting the driving scenario required for autonomous driving. Therefore, each device is configured with a communication network via the Internet or Wi-Fi. The object recognition machine learning learning and model generation server (820) and the driving scenario machine learning learning and model generation server (830) improve the ML model as a data server or application server and transmit the improved machine learning model to the edge device. The model transmission applies OTA (Over The Air) technology.

[0101] Data (841, 842) transmitted and received between the data acquisition device (810) and the object recognition machine learning learning and model generation server (820) or between the data acquisition device (810) and the driving scenario machine learning learning and model generation server (830) is transmitted when each server (820, 830) receives a video image or 3D point information (point cloud), sensor information and important events (step 22 of FIG. 3), the type of object, the association information between objects matching the recognized driving scenario and the classified importance from the data acquisition device (810), or when each server transmits a machine learning model to multiple edge devices by the OTA method.

[0102] When each server (820, 830) receives a video image, if an object is recognized through an object recognition model on the edge device, the video image or the previous video image is transmitted to each server.

[0103] Data (843) transmitted from the object recognition machine learning learning and model generation server (820) to the driving scenario machine learning learning and model generation server (830) is a case where the object recognition machine learning learning result or recognized object information is transmitted.

[0104] The data (845) transmitted from the driving scenario machine learning training and model generation server (830) to the object recognition machine learning training and model generation server (820) is a feedback signal. Since the feedback signal includes information on objects that do not satisfy the driving scenario as a result of the driving scenario training, the object recognition machine learning training and model generation server (820) can use the information to modify the object recognition embedding vector and perform retraining.

[0105] When the machine learning driving scenario is augmented and trained using the GAN learning method in the driving scenario machine learning learning and model generation server (830), artificial noise is generated in the association information (link information) between objects according to the weights applied to the generator of the driving scenario machine learning learning and model generation server (830) for training. For example, when the driving scenario machine learning learning and model generation server (830) performs GAN learning, if the weight of the generator is large, noise can be increased when generating the object association information embedding vector, thereby reducing the result value of the generator's loss function. In addition, if the objective function of the driving scenario learning is to prevent collisions between vehicles, false link information can be generated so as not to recognize cases with a high possibility of collision. Therefore, in order to perform an effective vehicle collision avoidance simulation, information between objects with a high probability of collision is required, and the information is generated as a virtual object recognition embedding vector by the object recognition machine learning learning and model generation server (820), and information on sections with a high probability of collision is transmitted from the object recognition machine learning learning and model generation server (820) to the driving scenario machine learning learning and model generation server (830), and the driving scenario machine learning learning and model generation server (830) augments the driving scenario data so that, during GAN learning, the learning rate of the discriminator can be increased for sections with a high probability of collision in order to determine collision avoidance and false driving information.

[0106] Edge devices update their object recognition machine learning models and driving scenario machine learning models via over-the-air (OTA) from their respective service servers. Edge devices provide real-time generated object IDs and driving scenario IDs to multiple servers, allowing each server to determine and verify the machine learning learning rate.

[0107] Each service server modifies the embedding vector and updates the machine learning model through feedback signals between each other, and then transmits it to the edge device.

[0108]

[0109] [Revised 18.04.2024 under Rule 91] Fig. 8 is a detailed description of the driving scenario verification unit (500), and is a flowchart showing a method for creating a driving scenario machine learning model.

[0110] Before generating a video image with a virtual object inserted, a virtual object recognition embedding vector is generated, and after verifying the satisfaction of the object recognition machine learning model and the driving scenario recognition model, a video image with the virtual object inserted into the actual video is generated. In the object recognition step, a convolution neural network (CNN) machine learning method is applied, and for the driving scenario machine learning method, a generative adversarial network (GAN) or graphical neural network (GNN) learning method is applied.

[0111] Step 511 defines features for object recognition. For example, this step defines image features needed to recognize objects such as cars, people, lanes, guardrails, streetlights, and sidewalks.

[0112] [Revised on 18.04.2024 by Article 91 of the Rules] Step 512 is a step of configuring and storing the object recognition area as a subset in order to quickly recognize an object in a captured video image. When an object is recognized by a data acquisition device (810), i.e., an edge device, the image is transmitted to an object recognition machine learning learning and model generation server (820) and analyzed in real time. In FIG. 7, this is data (841) transmitted and received between the data acquisition device (810) and the object recognition machine learning learning and model generation server (820).

[0113] Step 513 is the step of matching the subset and object recognition features to create machine learning training data and saving it as an object recognition embedding vector.

[0114] [Revised on 18.04.2024 under Rule 91] Step 514 is a step for generating an object recognition machine learning (ML) model by applying convolutional neural network (CNN) learning. Accordingly, the acquired video image is processed through steps 511 to 513, and the object recognition machine learning model generated in step 514 is applied, thereby performing a step (515) for automating object recognition using the object recognition machine learning model on an edge device. As in the data (841) transmitted and received between the data acquisition device (810) of FIG. 7 and the object recognition machine learning training and model generation server (820), when an object recognition ML model is transmitted OTA from the server (820) to the edge device (810), the edge device (810) operates the ML model to recognize an object in real time, and transmits the corresponding video image and event information to the server (820), and as in step 512, the server (820) assigns an ID to an object extracted from a series of images, stores the object group of the series of images as one subset, thereby managing object information in units of subsets.

[0115] Step 521 defines the driving scenarios to be used in machine learning training. For example, if a collision avoidance scenario is defined, a vehicle to the left of the current vehicle may cross the lane and enter the current vehicle's path. In this case, the vehicle can slow down to prevent an accident. This series of situations for collision avoidance is defined as a driving scenario. Video images are analyzed to identify objects, and association information between the identified objects is matched to the driving scenario. This association information between the recognized objects is then stored in a graph database.

[0116] Step 531 constructs a graph embedding vector by storing a series of situations matching the defined driving scenario in a graph database. To effectively build a graph database, objects and their attribute information, the ontology structure between objects, and driving scenario rules that satisfy various driving situations must be defined in advance.

[0117] Step 541 is a step for generating an ML model capable of recognizing driving scenarios by performing graph neural network (GNN) learning using the above-mentioned graph embedding vector. In addition, after augmenting various driving scenarios using ontology information between objects and a GAN machine learning learning method, a user selects the driving scenarios extracted by the GAN model, performs GNN machine learning learning using the selected driving scenarios, generates a driving scenario ML model, and transmits the generated driving scenario ML model to an edge device so that an autonomous driving function that can prevent vehicle collisions of driving vehicles can be applied.

[0118]

[0119] Embodiments of the present invention may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specifically designed and configured for the present invention, or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0120] As described above, the present invention has been described with specific details such as specific components and limited embodiments and drawings, but these are provided only to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments, and those with ordinary knowledge in the field to which the present invention pertains can make various modifications and variations from this description.

[0121] Therefore, the idea of ​​the present invention should not be limited to the described embodiments, and all things that are equivalent or equivalent to the claims described below as well as the claims are considered to fall within the scope of the idea of ​​the present invention.

[0122] The term '~ unit' used in this embodiment means a software or hardware component such as an FPGA (field-programmable gate array) or an ASIC, and the '~ unit' performs certain roles. However, the '~ unit' is not limited to software or hardware. The '~ unit' may be configured to be on an addressable storage medium and may be configured to play one or more processors. Thus, as an example, the '~ unit' includes components such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and '~ units' may be combined into a smaller number of components and '~ units' or further separated into additional components and '~ units'. Additionally, components and '~parts' may be implemented to regenerate one or more CPUs within a device or secure multimedia card.

[0123] All of the functions described above can be performed by a processor such as a microprocessor, controller, microcontroller, ASIC (Application Specific Integrated Circuit), etc., according to software or program code coded to perform the functions described above. The design, development, and implementation of the code will be apparent to those skilled in the art based on the description of the present invention.

Claims

1. A method for generating an object recognition machine learning model and a driving scenario machine learning model from a video image, A step of recognizing the features of an object in a video image and generating a first object recognition embedding vector for machine learning; A step of performing convolutional neural network (CNN) machine learning training using the first object recognition embedding vector as an input value and generating an object recognition machine learning model; A step of generating a first graph embedding vector using a second object recognition embedding vector used to generate the object recognition machine learning model and a driving scenario rule defining semantic relation information between predetermined objects; A method for generating an object recognition machine learning model and a driving scenario machine learning model, including a step of performing graph neural network (GNN) learning using the first graph embedding vector as an input value and generating a driving scenario machine learning model.

2. In paragraph 1, The above graph neural network machine learning learning is, A method for generating an object recognition machine learning model and a driving scenario machine learning model, characterized in that a generative adversarial network (GAN) machine learning model is generated using the first graph embedding vector as an input value, and a driving scenario machine learning model is generated using the second graph embedding vector used to generate the adversarial generative model as an input value.

3. In paragraph 2, In the step of generating the second graph embedding vector, A method for generating an object recognition machine learning model and a driving scenario machine learning model, characterized in that new graph information is generated by combining hierarchical information (ontology) between predefined objects with a portion of the graph of the first graph embedding vector and adding it to the first graph embedding vector.

4. In paragraph 2, A method for generating an object recognition machine learning model and a driving scenario machine learning model, characterized in that the object recognition model generation is performed on a first server, the first server transmits the second object recognition embedding vector to a second server, and the second server generates the driving scenario machine learning model.

5. In paragraph 1, A method for generating an object recognition machine learning model and a driving scenario machine learning model, characterized in that after generating the object recognition machine learning model and the driving scenario machine learning model, the machine learning models are transmitted to an edge device.

Images