Method and apparatus for predicting three-dimensional occupancy state by using neural network model, and computer-readable recording medium

Abstract

The present disclosure relates to a method and an apparatus for predicting an occupancy state by using a neural network model, and a computer-readable recording medium. The method according to one embodiment of the present disclosure may comprise obtaining multi-view images obtained from a camera. In addition, the method may comprise using the multi-view images as input data for a neural network model, and using the multi-view images so as to train the neural network model such that the neural network model outputs a three-dimensional prediction result related to an occupancy state. Particularly, the method may comprise training the neural network model to perform a first task for extracting a two-dimensional image feature for each of the multi-view images, a second task for converting same into a three-dimensional space by using the two-dimensional image feature, and a third task for predicting an occupancy state for the three-dimensional space.

Description

Method, device and computer-readable recording medium for predicting three-dimensional occupancy status using a neural network model

[0001] The present disclosure relates to a method, device, and computer-readable recording medium for predicting a three-dimensional occupancy state using a neural network model.

[0002] In autonomous driving systems, predicting the occupancy status of a vehicle's surroundings based on images acquired from one or more sensors can be a critical problem.

[0003] Camera sensors offer the advantage of low installation costs, and the high-resolution images acquired from cameras can include detailed features such as object color and texture. However, because camera images do not directly provide distance information, predicting occupancy based on camera images can be a challenging task.

[0004] Therefore, in order to accurately predict 3D occupancy status from camera images acquired in real time, a robust neural network model is required that accurately extracts 3D information from 2D images and predicts occupancy status based on 3D information with low error.

[0005] The background technology described above is technical information that the inventor possessed for the purpose of deriving the present invention or acquired in the process of deriving the present invention, and cannot necessarily be considered as publicly known technology disclosed to the general public prior to the application for the present invention.

[0006] Some embodiments of the present disclosure provide a method, device, and computer-readable recording medium for predicting three-dimensional occupancy using a neural network model. The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems and advantages of the present invention not mentioned above can be understood through the following description and will be more clearly understood through embodiments of the present invention. Furthermore, it will be appreciated that the problems and advantages to be solved by the present invention can be realized by the means and combinations thereof set forth in the claims.

[0007] As a technical means for achieving the above-described technical problem, a first aspect of the present disclosure may provide a method for predicting an occupancy state using a neural network model, comprising: a step of acquiring multi-view images acquired from a camera; and a step of training a neural network model to use the multi-view images as input data for the neural network model and output a three-dimensional prediction result regarding the occupancy state using the multi-view images; wherein the step of training the neural network model includes a step of training the neural network model to perform a first task of extracting a two-dimensional image feature for each of the multi-view images, a second task of transforming the two-dimensional image feature into a three-dimensional space, and a third task of predicting the occupancy state for the three-dimensional space.

[0008] A second aspect of the present disclosure provides a device for predicting an occupancy state using a neural network model, comprising: a memory storing at least one program; and a processor executing the at least one program, wherein the processor acquires multi-view images acquired from a camera, uses the multi-view images as input data for the neural network model, and trains the neural network model so that the neural network model outputs a three-dimensional prediction result regarding the occupancy state using the multi-view images.

[0009] A third aspect of the present disclosure can provide a computer-readable recording medium having recorded thereon a program for executing the method according to the first aspect on a computer.

[0010] In addition, other methods for implementing the present invention, other systems, and computer-readable recording media storing a computer program for executing the method may be further provided.

[0011] Other aspects, features and advantages other than those described above will become apparent from the following drawings, claims and detailed description of the invention.

[0012] According to one embodiment of the present disclosure, a robust neural network model can be trained to accurately extract three-dimensional information from a two-dimensional image and predict an occupancy state based on the three-dimensional information with a low error.

[0013] According to one embodiment of the present disclosure, a three-dimensional occupancy state can be accurately predicted from a camera image acquired in real time using a neural network model.

[0014] FIG. 1 is a diagram illustrating an example of a method for predicting an occupancy state around a self-driving vehicle in an autonomous driving system according to one embodiment.

[0015] FIGS. 2 and 3 are drawings for explaining an autonomous driving method according to one embodiment.

[0016] FIG. 4 is a diagram illustrating an example of a method for learning a neural network model according to one embodiment.

[0017] FIG. 5 is a diagram illustrating another example of a method for learning a neural network model according to one embodiment.

[0018] FIG. 6 is a diagram illustrating another example of a method for learning a neural network model according to one embodiment.

[0019] FIG. 7 is a diagram illustrating another example of a method for learning a neural network model according to one embodiment.

[0020] FIG. 8 is a diagram illustrating another example of a method for learning a neural network model according to one embodiment.

[0021] FIG. 9 is a diagram illustrating an example of a method for predicting occupancy status using a neural network model according to one embodiment.

[0022] FIG. 10 is a flowchart illustrating an example of a method for predicting occupancy status using a neural network model according to one embodiment.

[0023] FIG. 11 is a schematic diagram illustrating an example of the internal configuration of a device for predicting occupancy status using a neural network model according to one embodiment.

[0024] According to one embodiment of the present disclosure, a method for predicting an occupancy state using a neural network model may include acquiring multi-view images acquired from a camera, using the multi-view images as input data for the neural network model, and training the neural network model so that the neural network model outputs a three-dimensional prediction result regarding the occupancy state using the multi-view images, and the step of training the neural network model may include a step of training the neural network model to perform a first task of extracting a two-dimensional image feature for each of the multi-view images, a second task of converting the two-dimensional image feature into a three-dimensional space, and a third task of predicting the occupancy state for the three-dimensional space.

[0025] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments presented below, but may be implemented in various different forms, and it should be understood that it includes all transformations, equivalents, and substitutes included in the spirit and technical scope of the present invention. The embodiments presented below are provided to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention. In describing the present invention, if a detailed description of a related known technology is judged to obscure the gist of the present invention, the detailed description thereof will be omitted.

[0026] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0027] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a given function. Furthermore, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented by algorithms that execute on one or more processors. Furthermore, the present disclosure may employ conventional techniques for electronic configuration, signal processing, and / or data processing. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations.

[0028] Additionally, the connecting lines or connecting members between components depicted in the drawings are merely exemplary representations of functional connections and / or physical or circuit connections. In an actual device, connections between components may be represented by various functional connections, physical connections, or circuit connections that may be replaced or added.

[0029] In the present disclosure, 'vehicle' may mean any type of transportation means used to move people or things with a motor such as a car, bus, motorcycle, kickboard or truck.

[0030] In the present disclosure, a "neural network model" is an algorithm technology that classifies / learns the characteristics of input data on its own, and the element technology is a technology that simulates functions of the human brain, such as cognition and judgment, by utilizing machine learning algorithms such as deep learning. The neural network model of the embodiments of the present disclosure may have, for example, a deep neural network structure. The neural network model may be learned using training data based on one or more nodes and operation rules between the nodes. The structure of the nodes, the structure of the layers, and the operation rules between the nodes may be determined in various ways depending on the embodiment. The neural network model includes hardware resources such as one or more processors, memories, registers, summation processing units, parallel processing units, or multiplication processing units, and operates the hardware resources based on a parameter set applied to each hardware resource. To this end, the processor that operates the neural network model may perform a task or resource management process that allocates hardware resources for each operation of the neural network model. The neural network model may have a structure such as a convolutional neural network (CNN), a recurrent neural network (RNN), or a long short-term memory (LSTM).

[0031] Below, embodiments are described in detail with reference to the attached drawings. However, the embodiments may be implemented in various different forms and are not limited to the examples described herein.

[0032] FIG. 1 is a diagram illustrating an example of a method for predicting an occupancy state around a self-driving vehicle in an autonomous driving system according to one embodiment.

[0033] Referring to FIG. 1, the autonomous driving system (100) may include a magnetic vehicle (110) equipped with an autonomous driving device (not shown).

[0034] An autonomous driving device may include various sensors that the autonomous driving system (100) uses to recognize the surrounding environment of the vehicle (110). For example, the autonomous driving device may include various sensors such as Lidar, cameras, and ultrasonic sensors. The sensors may collect information necessary for autonomous driving, such as the location of the vehicle (110), the location and speed of surrounding vehicles, the driving path, and the locations of surrounding objects.

[0035] An autonomous driving device may include computing hardware that runs software for controlling the vehicle (110). For example, the autonomous driving device may recognize the environment or situation surrounding the vehicle (110) based on data collected from sensors, determine the behavior of the vehicle (110), and run software for controlling the vehicle (110).

[0036] In the present disclosure, the autonomous driving device can predict the occupancy status of the surroundings of the self-vehicle (110) based on images acquired from one or more sensors.

[0037] In more detail, the autonomous driving device according to the embodiment of the present disclosure can obtain a three-dimensional prediction result (130) regarding an occupancy state from multi-view images (120) obtained from a camera using a neural network model.

[0038] Camera sensors have the advantage of low installation costs, and the high-resolution images they capture can include detailed features such as object color and texture. However, because camera images do not directly provide distance information, predicting occupancy based on camera images can be a challenging task.

[0039] Therefore, in order to accurately predict 3D occupancy status from camera images acquired in real time, a robust neural network model is required that accurately extracts 3D information from 2D images and predicts occupancy status based on 3D information with low error.

[0040] For the purpose described above, the present disclosure discloses a method for training a neural network model according to a multi-task learning method and obtaining a three-dimensional prediction result (130) regarding an occupancy state using the trained neural network model.

[0041] For example, according to a multi-task learning method, multi-view images acquired from a camera can be used as input data for learning a neural network model, and the neural network model can be learned to perform multiple tasks using the multi-view images.

[0042] The neural network model according to the embodiment of the present disclosure can learn general expressions by sharing and utilizing information obtained during the learning process by learning to perform multiple tasks simultaneously, and can prevent overfitting for each task.

[0043] Meanwhile, a specific method for learning a neural network model according to an embodiment of the present disclosure will be described with reference to drawings 4 to 8 below.

[0044] FIGS. 2 and 3 are drawings for explaining an autonomous driving method according to one embodiment.

[0045] Referring to FIG. 2, an autonomous driving device according to an embodiment of the present invention can be mounted on a vehicle to implement an autonomous vehicle (10). The autonomous driving device mounted on the autonomous vehicle (10) can include various sensors for collecting information about the surroundings. For example, the autonomous driving device can detect the movement of a preceding vehicle (20) driving in front through an image sensor and / or an event sensor mounted on the front of the autonomous vehicle (10). The autonomous driving device can further include sensors for detecting not only the front of the autonomous vehicle (10), but also other vehicles (30) driving in the adjacent lane, and pedestrians around the autonomous vehicle (10).

[0046] According to one embodiment, the autonomous driving device may include multiple cameras that capture images of the surroundings of the autonomous vehicle (10) from various angles. For example, the multiple cameras may be positioned at the front, sides, and rear of the autonomous vehicle (10) to generate images of the surroundings of the autonomous vehicle (10) captured from different viewpoints or angles. In addition, the autonomous driving device may acquire multi-view images of the surroundings of the autonomous vehicle (10) based on the images captured by the multiple cameras.

[0047] Meanwhile, at least one of the sensors for collecting situational information surrounding the autonomous vehicle may have a predetermined field of view (FoV) as illustrated in FIG. 1. For example, when a sensor mounted on the front of the autonomous vehicle (10) has a field of view (FoV) as illustrated in FIG. 1, information detected at the center of the sensor may have relatively high importance. This may be because the information detected at the center of the sensor mostly includes information corresponding to the movement of the preceding vehicle (20).

[0048] The autonomous driving device processes information collected by sensors of the autonomous vehicle (10) in real time to control the movement of the autonomous vehicle (10), and at least some of the information collected by the sensors can be stored in a memory device.

[0049] Referring to FIG. 3, the autonomous driving device (40) may include a sensor unit (41), a processor (46), a memory system (47), and a body control module (48). The sensor unit (41) includes a plurality of sensors (42-45), and the plurality of sensors (42-45) may include an image sensor, an event sensor, an illuminance sensor, a GPS device, an acceleration sensor, and the like.

[0050] Data collected by the sensors (42-45) can be transmitted to the processor (46). The processor (46) stores the data collected by the sensors (42-45) in the memory system (47) and controls the body control module (48) based on the data collected by the sensors (42-45) to determine the movement of the vehicle. The memory system (47) can include two or more memory devices and a system controller for controlling the memory devices. Each of the memory devices can be provided as a single semiconductor chip.

[0051] In addition to the system controller of the memory system (47), each of the memory devices included in the memory system (47) may include a memory controller, and the memory controller may include an artificial intelligence (AI) operation circuit such as a neural network. The memory controller may generate operation data by assigning a predetermined weight to data received from sensors (42-45) or a processor (46) and store the operation data in a memory chip.

[0052] FIG. 4 is a diagram illustrating an example of a method for learning a neural network model according to one embodiment.

[0053] A device (hereinafter referred to as “device”) for predicting occupancy status using a neural network model according to an embodiment of the present disclosure can train a neural network model to output a three-dimensional prediction result regarding occupancy status using multi-view images.

[0054] According to one embodiment, the device may acquire multi-view images (410) acquired from a camera, and use the multi-view images (410) to construct a learning data set for training a neural network model (420). For example, the device may use the multi-view images (410) as input data for the neural network model (420), and may use data to be output for each task using the neural network model (420) as correct answer data (Ground Truth) corresponding to the input data.

[0055] Meanwhile, although the device for predicting occupancy using a neural network model in FIG. 1 is described as being an autonomous driving device, it is not limited thereto. For example, the device in FIG. 4 may be identical to the autonomous driving device in FIG. 1 or may be a device included in the autonomous driving device. As another example, the device in FIG. 4 may be an external server or external device that has its own computing capabilities and is installed outside the vehicle (110) to exchange information with the autonomous driving device. For example, the device may acquire multi-view images from a camera installed in the vehicle (110), predict the occupancy status, and transmit the prediction result to the autonomous driving device.

[0056] Referring to FIG. 4, the device can train the neural network model (420) to perform a first task of extracting a two-dimensional image feature (430) for each of the multi-view images (410).

[0057] According to one embodiment, the device may be configured such that the neural network model (420) includes an image encoder module. For example, the image encoder may use at least one image among the multi-view images as input data. In addition, the image encoder may extract image features from the two-dimensional multi-view images and generate a two-dimensional feature map. For example, the image encoder may be configured using a Residual Network (Resnet) and a Feature Pyramid Network (FPN).

[0058] FIG. 5 is a diagram illustrating another example of a method for learning a neural network model according to one embodiment.

[0059] Referring to FIG. 5, the first task may include receiving a Lidar point cloud (520) acquired from a Lidar sensor in addition to multi-view images (510) and outputting a two-dimensional prediction result (540) regarding the occupancy state.

[0060] In other words, the device can use the lidar point cloud (520) in addition to the multi-view images (510) as input data for training the neural network model (530), and can train the neural network model (530) to perform a first task of outputting a two-dimensional prediction result (540) regarding the occupancy state using the multi-view images (510) and the lidar point cloud (520).

[0061] Here, the lidar point cloud (520) may include information acquired by the lidar sensor emitting light toward an object. For example, the lidar point cloud (520) may include three-dimensional information about the surrounding environment, such as distance information to the object, height information of the object, and reflection intensity, which can be acquired using light reflected from the object.

[0062] In one embodiment, the device may be configured to include a neural network model (530) comprising a predictor module. For example, the predictor module may receive at least one image among multi-view images and a lidar point cloud (520) as input and output a two-dimensional prediction result (540) regarding the occupancy state.

[0063] According to one embodiment, the correct answer data corresponding to the two-dimensional prediction result (540) may be a projection image that projects the lidar point cloud (520) onto a multi-view image. For example, the device may calculate a loss between the two-dimensional prediction result (540) and the projection image as the loss for the first task described above.

[0064] FIG. 6 is a diagram illustrating another example of a method for learning a neural network model according to one embodiment.

[0065] Referring to FIG. 6, the device can train the neural network model (620) to perform a second task of transforming multi-view images (610) into a three-dimensional space (630).

[0066] In one embodiment, the second task may include measuring depth information (640) for a two-dimensional image. In other words, the device may train the neural network model (620) to perform the second task of measuring depth information (640) for a two-dimensional image in addition to transforming the two-dimensional image (610) into a three-dimensional space (630).

[0067] According to one embodiment, the device may be configured such that the neural network model (620) includes a view transformer module. For example, the view transformer may refer to a module that measures depth information (640) for a two-dimensional image and estimates context information using two-dimensional image features (e.g., image features for multi-view images) obtained from an image encoder and camera parameter information. In addition, the view transformer may perform a voxel pooling operation based on the depth information (640) and the context information, thereby transforming two-dimensional multi-view images into a three-dimensional space (630).

[0068] According to one embodiment, the correct answer data for the depth information (640) measured by the neural network model (620) may be the correct depth information (ground truth depth). For example, the device may calculate the loss between the measured depth information (640) and the correct depth information as the loss for the second task described above.

[0069] FIG. 7 is a diagram illustrating another example of a method for learning a neural network model according to one embodiment.

[0070] Referring to FIG. 7, the device can train the neural network model (720) to perform a third task of predicting (730) an occupancy state for a three-dimensional space based on multi-view images (710).

[0071] According to one embodiment, the third task may include extracting a three-dimensional feature for a three-dimensional space, predicting whether a given object is occupied based on the three-dimensional feature, and classifying a class for the three-dimensional space based on a label of the given object.

[0072] In other words, the neural network model (720) performing the third task may mean determining whether a given object occupies a certain area in a three-dimensional space and classifying the corresponding area into a class. For example, if no object occupies a certain area in a three-dimensional space, the neural network model (720) may classify the corresponding area into a class of the first group. As another example, if a given object occupies a certain area in a three-dimensional space, the neural network model (720) may classify the corresponding area into a class of the second group based on the label of the given object. For example, the class of the first group may include a 'Free Class'. In addition, the class of the second group may include 'Classes of barrier, bicycle, bus, car, motorcycle, sidewalk, pedestrian, drivable road, non-drivable road, vegetation, etc.'

[0073] According to one embodiment, the device may be configured such that the neural network model (720) includes a 3D encoder module and an occupancy prediction head. For example, the 3D encoder module may refer to a module that extracts 3D features for a 3D space acquired from a view transformer. For example, the 3D encoder module may include a Resnet 3D module and an FPN 3D module. In addition, the occupancy prediction head may refer to a module that predicts whether a given object is occupied based on 3D features and classifies a class for the 3D space based on a label of the given object.

[0074] According to one embodiment, the correct data for a class classified by the neural network model (720) in a three-dimensional space may be a ground truth class. For example, the device may calculate the loss between the classified class and the correct class as a loss for the third task described above.

[0075] FIG. 8 is a diagram illustrating another example of a method for learning a neural network model according to one embodiment.

[0076] As described above, the device according to one embodiment of the present disclosure can train a neural network model to perform a first task, a second task, and a third task, and calculate a loss with respect to the output of each of the first task, the second task, and the third task.

[0077] In one embodiment, the device may train a neural network model based on the first loss, the second loss, and the third loss calculated for each of the first, second, and third tasks, respectively. For example, the device may calculate the overall loss based on the first, second, and third losses, and train the neural network model to reduce the overall loss. For example, the device may weight and sum the first, second, and third losses, respectively, to calculate the overall loss.

[0078] Meanwhile, referring to FIG. 8, the first loss may include a loss calculated based on a two-dimensional prediction result (830) regarding the occupancy state. In other words, the device may input multi-view images (810) as input data to a neural network model (820), obtain a two-dimensional prediction result (830) regarding the occupancy state as output data of the neural network model (820), and calculate the first loss based on the two-dimensional prediction result (830).

[0079] Additionally, the second loss may include a loss calculated based on depth measurements (840) for the multi-view images. For example, the device may input multi-view images (810) as input data to a neural network model (820), obtain depth measurements (840) for the multi-view images (810) as output data of the neural network model (820), and calculate the second loss based on the depth measurements (840).

[0080] According to one embodiment, the third loss may include a loss (850) that is calculated in a class-balanced manner. More specifically, the loss for the third task may include a loss calculated based on the classification of classes in a three-dimensional space and the frequency of the classes. For example, the training data set used when training the neural network model (820) to perform the third task may be a class-imbalanced data set. In other words, the areas included in the multi-view images (810) may generally be more classified as “road class” or “free class” than areas classified as “bicycle class” or “motorcycle class.” Therefore, when calculating the loss for the third task, the device may calculate the third loss by taking into account the frequency of the classes more, and assigning a low weight to a high weight to a class with a high frequency. For example, the third loss may be calculated using Weighted Cross-Entropy or Dice Loss.

[0081] Continuing with reference to FIG. 8, the device can calculate the overall loss by weighting and summing the first loss calculated based on the two-dimensional prediction result (830), the second loss calculated based on the depth measurement (840), and the third loss calculated considering the class balance (850). Furthermore, the device can train the neural network model (820) to output the three-dimensional prediction result (870) regarding the occupancy state in a way that reduces the overall loss.

[0082] FIG. 9 is a diagram illustrating an example of a method for predicting occupancy status using a neural network model according to one embodiment.

[0083] Referring to FIG. 9, the device can input multi-view images (910) as input data for a neural network model (920) learned in the manner described above with reference to FIGS. 4 to 8.

[0084] Additionally, the device can obtain a three-dimensional prediction result (940) regarding the occupancy state as output data of the neural network model (920).

[0085] More specifically, the device can extract two-dimensional image features for multi-view images and convert the two-dimensional image features into a three-dimensional space. In addition, the device can extract three-dimensional features for the three-dimensional space, predict whether a given object is occupied for each region of the three-dimensional space based on the three-dimensional features, and classify a class for each region based on a label of the given object, thereby obtaining a three-dimensional prediction result (940) regarding the occupancy state.

[0086] However, as described above in FIG. 1, images acquired from a camera do not contain distance information, and therefore ambiguity issues may arise depending on the distance of objects included in the image. Therefore, in the inference step of the neural network model according to one embodiment of the present disclosure, an ambiguity suppression process may be further performed.

[0087] In one embodiment, the device may determine a class based on the reliability of the class classified for each region of the three-dimensional space. Here, the reliability of the class may be calculated based on the distance of each region of the three-dimensional space from the ego car. For example, if a region of the three-dimensional space is classified as a second group, the device may determine a class for the region as a first group based on the reliability of the class. For example, if a region of the three-dimensional space is classified as a 'car class', but the reliability of the region is calculated to be low because the region is far from the ego car, the region may be determined as a 'free class'.

[0088] FIG. 10 is a flowchart illustrating an example of a method for predicting occupancy status using a neural network model according to one embodiment.

[0089] Referring to FIG. 10, a method for predicting occupancy status using a neural network model may include steps 1010 to 1050. However, even if omitted below, the contents described with reference to FIGS. 1 to 9 may also be applied to the method for predicting occupancy status using the neural network model of FIG. 10.

[0090] At step 1010, the device can acquire multi-view images acquired from a camera.

[0091] In step 1030, the device can use the multi-view images as input data for a neural network model, and train the neural network model to output a three-dimensional prediction result regarding the occupancy state using the multi-view images.

[0092] For example, the device can train a neural network model to perform a first task of extracting two-dimensional image features for each of the multi-view images, a second task of transforming the two-dimensional image features into a three-dimensional space, and a third task of predicting an occupancy state for the three-dimensional space.

[0093] According to one embodiment, the first task may further include receiving a lidar point cloud acquired from a lidar sensor in addition to multi-view images, and further outputting a two-dimensional prediction result regarding the occupancy state.

[0094] According to one embodiment, the second task may further comprise measuring depth information for the multi-view images.

[0095] According to one embodiment, the third task may include extracting three-dimensional features for a three-dimensional space, predicting whether each region of the three-dimensional space is occupied by a given object based on the three-dimensional features, and classifying a class for each region based on a label of the given object.

[0096] According to one embodiment, the device may train a neural network model based on a first loss, a second loss, and a third loss calculated for each of the first task, the second task, and the third task, respectively. Here, the first loss may include a loss calculated based on a two-dimensional prediction result regarding an occupancy state, the second loss may include a loss calculated based on a depth measurement for multi-view images, and the third loss may include a loss calculated based on a classification of classes and a frequency of the classes for a three-dimensional space.

[0097] In step 1050, the device can input multi-view images as input data for a learned neural network model, and obtain a 3D prediction result regarding the occupancy state as output data of the neural network model.

[0098] In one embodiment, the device may determine a class based on the confidence level of the class classified for each region in the three-dimensional space. Here, the confidence level may be calculated based on the distance of each region in the three-dimensional space from the vehicle. For example, if a region in the three-dimensional space is classified as a second group, the device may determine a class for that region as a first group based on the confidence level of the class.

[0099] FIG. 11 is a schematic diagram illustrating an example of the internal configuration of a device for predicting occupancy status using a neural network model according to one embodiment.

[0100] Referring to FIG. 11, the device (1100) may include a processor (1110), a memory (1120), an input / output interface (1130), and a communication module (1140). For convenience of explanation, only components related to the present invention are illustrated in FIG. 11. Therefore, in addition to the components illustrated in FIG. 11, other general components may be further included in the device (1100). In addition, it will be apparent to a person skilled in the art related to the present invention that the processor (1110), memory (1120), input / output interface (1130), and communication module (1140) illustrated in FIG. 11 may be implemented as independent devices.

[0101] The processor (1110) can process computer program instructions by performing basic arithmetic, logic, and input / output operations. Here, the instructions may be provided from memory (1120) or an external device. In addition, the processor (1110) can generally control the operations of other components included in the device (1100).

[0102] The processor (1110) may be implemented as an array of a plurality of logic gates, or may be implemented as a combination of a general-purpose microprocessor and a memory storing a program that can be executed on the microprocessor. For example, the processor (1110) may include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the processor (1110) may include an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. For example, the processor (1110) may refer to a combination of processing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors coupled with a digital signal processor (DSP) core, or any other such combination of configurations.

[0103] The memory (1120) may include any non-transitory computer-readable recording medium. For example, the memory (1120) may include a non-permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. As another example, the non-permanent mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be a separate permanent storage device distinct from the memory. In addition, the memory (1120) may store an operating system (OS) and at least one program code (e.g., code for the processor (1110) to perform the operations described with reference to FIGS. 4 to 9).

[0104] These software components may be loaded from a computer-readable recording medium separate from the memory (1120). This separate computer-readable recording medium may be a recording medium directly connectable to the device (1100), and may include, for example, a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. Alternatively, the software components may be loaded into the memory (1120) through a communication module (1140) other than a computer-readable recording medium. For example, at least one program may be loaded into the memory (1120) based on a computer program (e.g., a computer program for the processor (1110) to perform the operations described with reference to FIGS. 4 to 9) that is installed by files provided by developers or a file distribution system that distributes installation files of applications through the communication module (1140).

[0105] The input / output interface (1130) may be a means for interfacing with a device (e.g., a keyboard, a mouse, etc.) for input or output that may be connected to or included in the device (1100). In FIG. 11, the input / output interface (1130) is illustrated as an element configured separately from the processor (1110), but is not limited thereto, and the input / output interface (1130) may be configured to be included in the processor (1110).

[0106] The communication module (1140) may provide a configuration or function for the device (1100) to communicate with other external devices via a network. For example, control signals, commands, data, etc. provided under the control of the processor (1110) may be transmitted to a server and / or external device via the communication module (1140) and the network.

[0107] In one embodiment, the device (1100) may be a mobile electronic device. For example, the device (1100) may be implemented as a smartphone, tablet PC, PC, smart TV, personal digital assistant (PDA), laptop, media player, navigation device, camera-equipped device, or other mobile electronic device. Furthermore, the device (1100) may be implemented as a wearable device, such as a watch, glasses, hair band, or ring, equipped with communication and data processing capabilities.

[0108] In another embodiment, the device (1100) may be an electronic device embedded within a vehicle. For example, the device (1100) may be an electronic device inserted into a vehicle through tuning after the production process.

[0109] In another embodiment, the device (1100) may be a server located outside the vehicle. The server may be implemented as a computer device or multiple computer devices that communicate over a network to provide commands, codes, files, content, services, etc. The server receives data necessary to predict occupancy from devices mounted on the vehicle and can predict three-dimensional occupancy based on the received data.

[0110] In another embodiment, the process performed in the device (1100) may be performed by at least some of a mobile electronic device, an electronic device embedded in a vehicle, and a server located outside the vehicle.

[0111] Embodiments of the present invention may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. In this case, the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memories.

[0112] Meanwhile, the computer program may be specifically designed and constructed for the present invention, or may be one known and available to those skilled in the computer software field. Examples of computer programs may include not only machine language code, such as that generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

[0113] According to one embodiment, the method according to various embodiments of the present disclosure may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0114] Unless the steps constituting the method according to the present invention are explicitly described in a specific order or are not described to the contrary, the steps may be performed in any appropriate order. The present invention is not necessarily limited to the order in which the steps are described. The use of all examples or exemplary terms (e.g., "for example," etc.) in the present invention is merely intended to illustrate the present invention in detail, and the scope of the present invention is not limited by the examples or exemplary terms unless otherwise defined by the claims. Furthermore, those skilled in the art will appreciate that various modifications, combinations, and variations can be configured according to design conditions and factors within the scope of the appended claims or their equivalents.

[0115] Therefore, the idea of ​​the present invention should not be limited to the embodiments described above, and not only the scope of the patent claims described below but also all scopes equivalent to or equivalently modified from the scope of the patent claims are considered to fall within the scope of the idea of ​​the present invention.

Claims

1. A method for predicting occupancy status using a neural network model, A step of acquiring multi-view images acquired from a camera; and A step of using the multi-view images as input data for the neural network model, and training the neural network model to output a three-dimensional prediction result regarding the occupancy state using the multi-view images; including; The step of training the above neural network model is: A method comprising the steps of training the neural network model to perform a first task of extracting a two-dimensional image feature for each of the multi-view images, a second task of transforming the two-dimensional image feature into a three-dimensional space, and a third task of predicting the occupancy state for the three-dimensional space.

2. In paragraph 1, The first task above is, A method comprising: additionally receiving a Lidar point cloud acquired from a Lidar sensor in addition to the above multi-view images, and further outputting a two-dimensional prediction result regarding the occupancy state.

3. In paragraph 1, The second task above is, A method further comprising measuring depth information for said multi-view images.

4. In paragraph 1, The third task above is, A method comprising extracting a three-dimensional feature for the three-dimensional space, predicting whether each region of the three-dimensional space is occupied by a predetermined object based on the three-dimensional feature, and classifying a class for each region based on a label of the predetermined object.

5. In paragraph 1, The step of training the above neural network model is: A method comprising the step of training the neural network model based on the first loss, the second loss and the third loss calculated for each of the first task, the second task and the third task.

6. In paragraph 5, The above first loss includes a loss calculated based on a two-dimensional prediction result regarding the occupancy state, The second loss includes a loss calculated based on depth measurements for the multi-view images, A method wherein the third loss comprises a loss calculated based on the classification of classes for the three-dimensional space and the frequency of the classes.

7. In paragraph 1, The above method, A step of inputting the multi-view images as input data for the neural network model learned by the method according to the first clause; and A method further comprising: a step of obtaining the three-dimensional prediction result regarding the occupancy state as output data of the neural network model; 8. In paragraph 7, The above method, A method further comprising the step of determining a class based on the reliability of the class classified for each region of the three-dimensional space.

9. In paragraph 8, The above reliability is, A method wherein each area of ​​the three-dimensional space is calculated based on the distance from the ego car.

10. In paragraph 8, The above decision-making steps are: A method comprising the step of determining a class as a first group for a certain area of ​​the three-dimensional space based on the reliability of the class when the class is classified into a second group for the certain area.

11. In a device that predicts occupancy status using a neural network model, memory in which at least one program is stored; and Comprising a processor for executing at least one program, The above processor, Acquire multi-view images acquired from the camera, A device that trains a neural network model to output a three-dimensional prediction result regarding an occupancy state by using the multi-view images as input data for the neural network model.

12. A computer-readable recording medium having recorded thereon a program for executing the method of clause 1 on a computer.

Images