Data processing method and apparatus using a neural network

Abstract

To provide a data processing method and device using a neural network.SOLUTION: A method executed by a processor includes: a step (first video and a second video have a different curvature) of determining a first deformed video by deforming the first video on the basis of the second video such that curvature of the first video corresponds to curvature of the second video; a step of determining a first re-deformed video by deforming the first deformed video such that curvature of the first deformed video corresponds to curvature of the first video; and a step of training a first curvature field generator for determining a first relative curvature field for expressing relative curvature from the first video to the second video and a second curvature field generator for determining a second relative curvature field for expressing relative curvature from the second video to the first video on the basis of loss between the first re-deformed video and the first video.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The following embodiments relate to a data processing method and apparatus that use a neural network. [Background technology]

[0002] Machine running improves the performance of image recognition in environments where video and annotations are supervised. Because securing video and annotations is crucial for such image recognition, it is based on rectilinear images, which are easier to access. However, rectilinear images have limitations in their field of view, and to overcome this, cameras utilizing fisheye lenses are beginning to emerge. While fisheye lenses are less constrained by field-of-view limitations than conventional lenses for rectilinear images, images captured through a fisheye lens may exhibit distortion.

[0003] There is a lack of annotations necessary to train inference models on images with such distortion, and distortion caused by lenses and cameras varies, making it difficult and inefficient to generate annotations directly. [Overview of the project] [Problems that the invention aims to solve]

[0004] The object of the present invention is to provide a data processing method and apparatus that uses a neural network. [Means for solving the problem]

[0005] A data processing method using a neural network, performed by a processor according to one embodiment, includes the steps of: deforming the first image based on the second image so that the distortion of the first image corresponds to the distortion of the second image, thereby determining a first deformed image (the first image and the second image have different distortions); deforming the first deformed image so that the distortion of the first deformed image corresponds to the distortion of the first image, thereby determining a first re-deformed image; and training a first distortion field generator that determines a first relative distortion field representing the relative distortion from the first image to the second image, and a second distortion field generator that determines a second relative distortion field representing the relative distortion from the second image to the first image, based on the loss between the first re-deformed image and the first image.

[0006] A data processing method according to one embodiment may further include the steps of: determining a second deformed image by deforming the second image so that the distortion of the second image corresponds to the distortion of the first image; determining a second re-deformed image by deforming the second deformed image so that the distortion of the second image corresponds to the distortion of the second image; and training the first distortion field generator and the second distortion field generator based on the loss between the second re-deformed image and the second image.

[0007] In a data processing method according to one embodiment, the initial parameters of the first distortion field generator can be determined through training based on the loss between a third deformed image, which is deformed from a first image based on a fisheye simulation, and the first deformed image.

[0008] In a data processing method according to one embodiment, the first relative distortion field and the second relative distortion field may have the characteristics of an inverse transformation relationship.

[0009] In a data processing method according to one embodiment, the step of determining the first deformed image can be to apply the first relative distortion field to the first image and determine the first deformed image having the distortion of the second image.

[0010] In a data processing method according to one embodiment, the step of determining the first re-deformed image can be performed by applying the second relative distortion field to the first deformed image to determine the first re-deformed image having the distortion of the first image.

[0011] In a data processing method according to one embodiment, the first video and the second video may be unpaired images containing different content and / or different scenes.

[0012] In a data processing method according to one embodiment, the step of training the first distortion field generator and the second distortion field generator can be performed using unsupervised learning without any information regarding the distortion of the first video and the second video, respectively.

[0013] A data processing method according to one embodiment may further include the step of training an inference model for the second video based on a deformed label video obtained by deforming the label video so that the distortion of the label video corresponding to the first video corresponds to the distortion of the second video, and the first deformed video.

[0014] A data processing method according to one embodiment may further include the step of training an inference model for the second video based on an unsupervised domain adaptation scheme using a fourth deformed video obtained by deforming the first video so that the distortion of the first video corresponds to the distortion and texture of the second video, a deformed labeled video, and the second video.

[0015] In a data processing method according to one embodiment, the second video may not have a corresponding label video.

[0016] A data processing method using a neural network, executed by a processor according to one embodiment, includes the steps of: determining a relative distortion field that represents the relative distortion from the source image to the target image based on a source image and a target image having different distortions; and applying the relative distortion field to the source image to determine a distorted source image having the distortion of the target image.

[0017] A data processing device according to one embodiment includes one or more processors, which deform the first image based on the second image so that the distortion of the first image corresponds to the distortion of the second image to determine a first deformed image (the first image and the second image have different distortions), deform the first deformed image so that the distortion of the first deformed image corresponds to the distortion of the first image to determine a first re-deformed image, and train a first distortion field generator that determines a first relative distortion field that represents the relative distortion from the first image to the second image based on the loss between the first re-deformed image and the first image, and a second distortion field generator that determines a second relative distortion field that represents the relative distortion from the second image to the first image. [Effects of the Invention]

[0018] Even when there is no distortion-related information between two images (e.g., camera parameters) and the two images contain different scenes (e.g., unpaired image contents), it is possible to easily generate an image with the same or similar distortion characteristics as an unlabeled target image from a labeled source image. This can then be used to obtain various inference models that are robust to distortion (e.g., segmentation models, object detection models, etc.). Furthermore, the cost of generating labels for images from cameras with unknown distortion characteristics can be efficiently reduced. [Brief explanation of the drawing]

[0019] [Figure 1] This figure illustrates the operation of generating a source video converted using a distortion field generator according to one embodiment. [Figure 2] This figure illustrates the operation of training a distortion field generator according to one embodiment. [Figure 3] This figure illustrates the operation of training a distortion field generator according to one embodiment. [Figure 4] This is a diagram illustrating the operation of training an inference model according to one embodiment. [Figure 5] This is a diagram illustrating the operation of training an inference model according to one embodiment. [Figure 6] This is a diagram illustrating the operation of training an inference model according to one embodiment. [Figure 7] This figure shows a data processing method using a neural network according to one embodiment. [Figure 8] This is a diagram illustrating a data processing device using a neural network according to one embodiment. [Figure 9] This is a diagram illustrating a data processing device using a neural network according to one embodiment. [Modes for carrying out the invention]

[0020] The specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and can be modified in various ways. Therefore, the embodiments are not limited to the specific disclosure, and the scope of this specification includes modifications, equivalents, or substitutions that are part of the technical idea.

[0021] Terms such as "first" or "second" may be used to describe multiple components, but such terms should be interpreted solely for the purpose of distinguishing one component from others. For example, the first component may be named the second component, and similarly, the second component may also be named the first component.

[0022] When it is mentioned that one component is “linked” or “connected” to another component, it should be understood that it is directly linked to or connected to the other component, but that other components may be present in between.

[0023] A singular expression includes plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “includes” or “has” indicate the presence of features, figures, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood not to presuppose the existence or addition of one or more other features, figures, steps, actions, components, parts, or combinations thereof.

[0024] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by a person of ordinary skill in the art to which this embodiment belongs. Commonly used, predefined terms should be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as ideal or overly formal unless expressly defined herein.

[0025] Furthermore, when explaining with reference to the drawings, the same components will be assigned the same reference numerals regardless of the reference numerals used in the drawings, and redundant explanations will be omitted. In the description of embodiments, if it is determined that a specific explanation of related prior art would unnecessarily obscure the gist of the present invention, such detailed explanation will be omitted.

[0026] Figure 1 is a diagram illustrating the operation of generating a source image converted using a distortion field generator according to one embodiment.

[0027] Referring to Figure 1, source image I has different distortions. B 110 and target image I A 130 is given.

[0028] Source video I B 110 is a rectilinear image captured by a general camera, and may include, for example, a planar image without distortion. Source image I B For 110, there is a corresponding source label image 120. Source label image 120 is source image I B This shows the results of classifying or detecting objects contained in 110, and can include various types of labels depending on the inference operation being performed.

[0029] Target image I A130 is a fisheye video taken by a camera equipped with a fisheye lens, and may include, for example, a video with distortion. Since a video taken through a fisheye lens with an angle of view exceeding 180° is represented as a two-dimensional video, distortion may inevitably occur. Such distortion is affected by various parameters for the lens and the camera, and thus the target video I having the corresponding distortion A There may be no target label video corresponding to 130. The target video I A To train an inference model for 130, the target video I A is required not only for 130 but also for a target label video corresponding to the relevant video. The target video I A The operation of generating a video in which 130 has distortion and a corresponding label video exists will be described below.

[0030] The distortion field generator 140 may include a neural network or a neural network that estimates a relative distortion field indicating the relative distortion between two input videos. The neural network is represented as a mathematical model using nodes and edges, and may include such a mathematical model. The neural network may be an architecture of a deep neural network (DNN) and / or an n-layer neural network. The DNN or the n-layer neural network may be, but is not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network, a restricted Boltzmann machine, etc.

[0031] In FIG. 1, the distortion field generator 140 is the source video I B 110 to the target video IA Relative distortion field φ that represents the relative distortion to 130 B→A Estimate 150. Relative distortion field φ B→A 150 is source video I B Target image I A In order to deform the source image I to have a distortion of 130, B Each pixel within 110 corresponds to the target image I A This indicates which of the 130 pixels must be moved to.

[0032] Source video I B 110 and relative distortion field φ B→A Based on 150, spatial warping 160 is performed. For example, spatial warping 160 is performed on the relative distortion field φ B→A Using 150, source video I B It may be performed at 110. Spatial warping 160 may perform grid-based sampling. As a result, target image I A Converted source video with 130 distortions I ^ B→A 170 is source video I B It may be generated from 110.

[0033] Similarly, the source label image 120 has a relative distortion field φ B→A Applying 150 and performing spatial warping 160 results in target image I A A converted source label video 180 having a distortion of 130 may be generated. The converted source label video 180 is the converted source video I ^ B→A Along with 170, Target Image I A It can be used to train inference models on 130.

[0034] Source video I BFor linear images without distortion, such as 110, annotation (e.g., labeled images) is sufficient, thus reducing the relative distortion field φ B→A Target image I via spatial warping 160 based on 150 A Sufficient annotation can be easily obtained even for 130 types of distortion. Therefore, the considerable costs required to generate annotations for various types of distortion can be efficiently avoided.

[0035] The above operation is performed on source video I B 110 and Target Image I A Even without information about the distortion that each of the 130s possesses (for example, intrinsic parameters, extrinsic parameters, calibration parameters, etc. of the camera that captured the image), the distortion field generator 140 estimates the relative distortion field φ B→A It can be executed based on 150.

[0036] Source video I B 110 and Target Image I A 130 may be non-paired videos containing different content and / or scenes. Also, in Figure 1, for the sake of explanation, source video I B 110 is an undistorted image, target image I A 130 was described as a distorted image, but there are other examples as well, such as Target Image I A Not only 130, but also source video I B The source video I may also be distorted. B Target video I A If a different type of distortion exists and a corresponding labeled image exists, the description herein may be applied without limitation. Thus, the description herein may be applied to a variety of applications using cameras, from pinhole cameras to fisheye cameras.

[0037] The operation for training the distortion field generator 140 will be described in detail with reference to Figures 2 and 3.

[0038] Figures 2 and 3 illustrate the operation of training a distortion field generator according to one embodiment.

[0039] Referring to Figure 2, distortion field generators G1203 and G2213 are trained based on the inverse transform relationship without distortion information for the two images. For training distortion field generators G1203 and G2213, source and target images with different distortions, where one of the two images contains a labeled image, may be used. For example, source image I input to distortion field generator G1203. B 201 has a label image, is a flat image without distortion, and is target image I A 202 is a distorted image without a corresponding labeled image. However, source image I B 201 and Target Image I A 202 is not limited to those exemplified, and if other distortions exist, the description herein may apply if one of the two images has a label.

[0040] The distortion field generator G1203 generates source video I B Target image I from 201 A Relative distortion field φ that represents the relative distortion to 202 B→A Estimate 204. Source video I B 201 and relative distortion field φ B→A The transformed source video I is obtained by performing spatial warping 205 based on 204. ^ B→A 206 may be determined. Converted source video I ^ B→A 206 is source video I B 201 may be a distorted image, similar to the distortion present in the target image IA202.

[0041] Converted source video I ^ B→A 206 and relative distortion field φ A→BSpatial warping 207 is performed based on 214. Here, the relative distortion field φ A→B 214 is estimated by distortion field generator G2213, and unlike distortion field generator G1203, distortion field generator G2213 estimates the source image I with distortion. A Undistorted target image from 211 I B Relative distortion field φ that represents the relative distortion to 212 A→B 214 can be estimated. In other words, distortion field generator G1203 and distortion field generator G2213 are in an inverse transformation relationship. Transformed source video I ^ B→A 206 and relative distortion field φ A→B The re-transformed source video (not shown) is determined via spatial warping 207 based on 214. The re-transformed source video is the transformed source video I ^ B→A Source video I B This is an image that has been re-transformed to resemble a flat image with zero distortion, in other words, an image that has had the distortion inherent in 201.

[0042] Re-converted source video and source video I B A loss (or error) 209 between 208 and source video I is calculated, and distortion field generators G1203 and G2213 may be trained to minimize such loss 209. Loss 209 is calculated between the re-transformed source video and source video I B This represents the difference between 208 and, for example, L1-loss may be included, but the embodiment is not limited thereto.

[0043] The distortion field generator G2213 may receive the source video and target video input to the distortion field generator G1203 in reverse. The distortion field generator G2213 receives the source video I A Target image I from 211 B Relative distortion field φ that represents the relative distortion to 212 A→B 214 can be estimated. Relative distortion field φA→B 214 and source video I A Based on 211, spatial warping 215 is performed, and the converted source video I ^ A→B 216 can be determined. The converted source video I ^ A→B 216 is the source video I A 211 into target video I B is a video deformed like the distortion that target video I B 212 has. In other words, like a planar video with zero distortion like target video I A the video with the distortion of 211 removed is the converted source video I ^ A→B may be 216.

[0044] The converted source video I ^ A→B 216 and relative distortion field φ B→A Based on 204, spatial warping 217 is performed, and a re-converted source video (not shown) may be determined. The re-converted source video is the converted source video I ^ A→B 216 into source video I A is a video re-converted like the distortion of 211, in other words, like a fisheye video with distortion.

[0045] The difference between the re-converted source video and source video I A 218, loss 219 is calculated, and distortion field generator G1203 and distortion field generator G2213 may be trained so that such loss 219 is minimized. Loss 219 indicates the difference between the re-converted source video and source video I A 218, and may include, for example, L₁ loss, but the embodiments are not limited thereto.

[0046] Thus, distortion field generator G1203 and distortion field generator G2213 can be trained simultaneously so that the loss of the following mathematical formula (1) is minimized using the characteristics of the inverse conversion relationship.

[0047]

number

[0048] Therefore, the distortion field generators G1203 and G2213 can be trained to estimate a relative distortion field that represents relative distortion even when distortion information is not provided.

[0049] Considering that the learning operation is based on the properties of the inverse transform relation, the initial parameters of the distortion field generators G1203 and G2213 can be set to ensure learning stability rather than being set randomly, as will be explained in detail with reference to Figure 3.

[0050] Referring to Figure 3, a block diagram is shown illustrating the operation for determining the initial parameters of the distortion field generator.

[0051] Based on a fisheye simulation, source image I is a flat image without distortion. B From 301, converted source video I B→A 307 may be determined. Target image I is a fisheye image with distortion. A 302 is based on fisheye simulation, source image I B 301 may be determined from other planar images. Converted source image I B→A 307 and Target Image I A302 has the same distortion, determined based on a fisheye simulation with the same parameters applied. Or, target image I A 302 is a video captured with a fisheye camera, containing parameters related to the camera and target image I. A Source image I converted based on a fisheye simulation with 302 distortion-related parameters applied. B→A 307 is source video I B It is generated from 301. In this case as well, the converted source video I B→A 307 and Target Image I A 302 has the same distortion.

[0052] The distortion field generator G303 generates source video I B Target image I from 301 A Relative distortion field φ that represents the relative distortion to 302 B→A Determine 304. Relative distortion field φ B→A 304 and Source Video I B Spatial warping 305 based on 301 is performed, resulting in the converted source video I ^ B→A 306 may be generated. Converted source video I ^ B→A 306 is source video I B 301 is target image I A The image may be distorted, like the distortion in 302. Converted source image I ^ B→A 306 and converted source video I B→A A loss 308 is calculated between 307 and the distortion field generator G303 may be trained to minimize the loss 308. The loss 308 is the converted source video I ^ B→A 306 and converted source video I B→A This represents the difference between 307, and may include, for example, L1-loss, but the embodiment is not limited thereto.

[0053] The initial parameters of the distortion field generator G303 are determined through pre-training of the distortion field generator G303, and training of the distortion field generator G303 may be performed based on these initial parameters, as described with reference to Figure 2.

[0054] In Figure 3, for the sake of explanation, source video I B Target image I from 301 A Relative distortion field φ that represents the relative distortion to 302 B→A The explanation was based on the distortion field generator G303, which estimates 304, but see the video I in Figure 2. A Target image I from 211 B Relative distortion field φ that represents the relative distortion to 212 A→B The same description herein can also be applied to determining the initial parameters of the distortion field generator G2213 that estimates 214.

[0055] Figures 4 to 6 illustrate the operation of training an inference model according to one embodiment.

[0056] By using the distortion field generator described earlier, it is possible to generate training data containing captured video and corresponding labeled video for any type of distortion, and to train an inference model that performs recognition of the distorted video using this training data. The inference model may include, for example, a segmentation model or an object detection model as a neural network that performs inference on the input video. The training operation of the inference model will be described in detail below with reference to the diagram.

[0057] Referring to Figure 4, the inference model 430 can be trained based on the input video 410 and the labeled video 420.

[0058] The input video 410 is a video that has been transformed to have the same distortion as the target video to be inferred via the inference model 430. The target video may include distortion from the fisheye lens and / or camera, as it is a video taken through a fisheye lens. Using the distortion field generator described above, various videos with corresponding label videos may be transformed to have the same distortion as the target video. The label video 420 corresponds to the input video 410 as a video that has been transformed to have the same distortion as the target video. The inference model 430 may be trained to minimize the loss between the result of inference on the input video 410 and the label video 420. Therefore, the inference model 430 that infers the target video can be efficiently trained even without any information about the distortion of the target video to be inferred or the camera that took the video.

[0059] Referring to Figure 5, a segmentation model 521 (e.g., an inference model) can be trained using a distortion field generator and UDA (unsupervised domain adaptation). The training operation of the segmentation model 521 is divided into a disentangling distortion and texture step and a segmentation adaptation model learning step.

[0060] The distortion field generator 513 can estimate a relative distortion field 514 that represents the relative distortion from the source video 511 to the target video 512. The source video 511 is a video for which a corresponding label video exists, and the target video 512 is a video for which a corresponding label video does not exist, but which has the same distortion as the video that the segmentation model 521 is trying to infer. The texture-aware translator 515 may determine the texture conversion data 516 by recognizing the texture difference between the source video 511 and the target video 512 and converting the texture (e.g., color, brightness) of the source video 511 to match that of the target video 512. Spatial warping 517 is performed based on the texture conversion data 516 and the relative distortion field 514, and the converted source video 518 may be determined. In other words, the converted source video 518 is a video in which the source video 511 has been converted to have the distortion and texture of the target video 512.

[0061] The converted source video 518 and target video 519 may be used to train the segmentation model 521. The segmentation model 521 is a model that classifies objects contained in the input video, for example, people, roads, vehicles, signs, etc. The segmentation model 521 performs inference on the converted source video 518 and determines the source probabilities 522. The segmentation model 521 determines the source probability for each class to be classified. For example, the first source probability indicates the probability that each pixel in the converted source video 518 is a person, and the second source probability indicates the probability that each pixel in the converted source video 518 is a road. The number of source probabilities 522 may be the same as the number of classes to be classified in the segmentation model 521. Similarly, the segmentation model 521 may perform inference on the target video 519 and determine the target probability 523.

[0062] Adversarial learning may be performed on the split model 521 based on the source probability 522 and the target probability 523. A discriminator (not shown) determines whether the probability output from the split model 521 is based on inferences from the transformed source video 518 or from inferences from the target video 519, and the split model 521 may be trained to deceive the discriminator.

[0063] Source prediction 524 indicates which class each pixel in the converted source video 518 is most likely to belong to, and is a video showing the class with the highest probability among the source probabilities 522. Similarly, target prediction 525 is a video showing which class each pixel in the target video 519 is most likely to belong to.

[0064] Source label 526 is a label video corresponding to source video 511 that has been transformed to have the same distortion as target video 512. A split model 521 may be trained based on the difference between source prediction 524 and source label 526. Source prediction 524, generated by the inference result of split model 521, may be trained in the same way as source label 526.

[0065] The target pseudo label 527 may be determined by selecting labels that are above a pre-set threshold in the target prediction 525. Therefore, uncertain information with relatively low probability can be removed in the target prediction 525. The split model 521 can be trained based on the target pseudo label 527.

[0066] The distortion and texture disentanglement steps and the segmented adaptive model training steps can be performed alternately and repeatedly, or simultaneously, to train a segmented model 521 that can perform robust inferences for images with the same distortion as the target image 512, even without labels, with high accuracy.

[0067] Operation 530 can be similarly described in Figures 1 to 3. Operation 540 may be part of the UDA-based inference model training.

[0068] Referring to Figure 6, the object detection model 610 may be trained using a distortion field generator and UDA. Since the explanation in Figure 5 above can also be applied to training the object detection model 610, a detailed explanation of training the object detection model 610 will be omitted.

[0069] As explained earlier, by training an inference model, it is possible to efficiently train an inference model that can infer the target image even when there is no label for the target image to be inferred, a label exists for an image that is different in distortion and / or texture from the target image, and there is no distortion-related information between the two images, and the images include different scenes.

[0070] Figure 7 shows a data processing method using a neural network according to one embodiment.

[0071] In the embodiments described below, each step may be performed sequentially, but is not necessarily required. For example, the order of each step may be changed, and at least two steps may be performed in parallel. Operations 710 to 730 may be performed by at least one component (e.g., a processor) of an electronic device (e.g., a data processing device).

[0072] In step S710, the data processing device determines a first distorted image by distorting the first image to have the same distortion as the second image, from among the first and second images having different distortions. The data processing device may also determine a first distorted image having the distortion of the second image by applying a first relative distortion field to the first image. The first and second images may not be pairs of images containing different content and / or scenes. The second image may not have a corresponding label image.

[0073] In step S720, the data processing device deforms the first deformed image in the same way as the distortion of the first image to determine the first re-deformed image. The data processing device may also apply a second relative distortion field to the first deformed image to determine the first re-deformed image having the distortion of the first image.

[0074] In step S730, the data processing device trains a first distortion field generator that determines a first relative distortion field representing the relative distortion from the first image to the second image, and a second distortion field generator that determines a second relative distortion field representing the relative distortion from the second image to the first image, based on the loss between the first re-deformed image and the first image. The first relative distortion field and the second relative distortion field have the properties of an inverse transformation relationship. The data processing device can train the first distortion field generator and the second distortion field generator by unsupervised learning without information about the distortions of the first image and the second image, respectively.

[0075] Furthermore, the data processing device may deform the second image in the same way as the distortion of the first image to determine a second deformed image, deform the second deformed image in the same way as the distortion of the second image to determine a second re-deformed image, and train the first distortion field generator and the second distortion field generator based on the loss between the second re-deformed image and the second image.

[0076] The initial parameters of the first distortion field generator may be determined through training based on the loss between the first distorted image and a third distorted image, which is distorted from the first image based on a fisheye simulation.

[0077] The data processing device trains an inference model for the second video based on a distorted label video, which is created by transforming the label video corresponding to the first video in a way that mimics the distortion of the second video, and the first distorted video. The data processing device can also train an inference model for the second video based on an unsupervised domain adaptation scheme using a fourth distorted video, which is created by transforming the first video in a way that mimics the distortion and texture of the second video, a distorted label video, and the second video.

[0078] Even if the distortion characteristics of the image to be inferred are not provided and there is no label for that image, a distortion-robust inference model can be obtained by utilizing other images with labels and distortion characteristics.

[0079] Figures 8 and 9 illustrate a data processing device using a neural network according to one embodiment.

[0080] Referring to Figure 8, the data processing device 800 includes a processor 810 (e.g., one or more processors), memory 820 (e.g., one or more memories), a camera 830, a storage device 840, an input device 850, an output device 860, and a network interface 870, which can communicate via a communication bus or PCIe (Peripheral Component Interconnect Express), NoC (Network on a Chip), etc. 880. For example, the data processing device 800 can be used with mobile devices such as mobile phones, smartphones, PDAs, netbooks, tablet computers, and laptop computers, wearable devices such as smartwatches, smart bands, and smart glasses, and desktop computers. computerComputing devices such as servers, home appliances such as televisions, smart TVs, and refrigerators, doors B This can be implemented as at least part of a vehicle, such as a security device like a lock, an autonomous vehicle, or a smart vehicle.

[0081] The processor 810 executes functions and instructions for execution within the data processing unit 800. For example, the processor 810 may process instructions stored in the memory 820 or the storage device 840. The processor 810 may perform one or more operations as described with reference to Figures 1 to 7. The memory 820 may include a computer-readable storage medium or computer-readable storage device. The memory 820 stores instructions for execution by the processor 810 and may also store related information while the software and / or application is executed by the data processing unit 800.

[0082] The camera 830 can take photographs and / or videos. The storage device 840 may include a computer-readable storage medium or a computer-readable storage device. The storage device 840 can store a larger amount of information than the memory 820 and can store information for a longer period of time. For example, the storage device 840 may include a magnetic hard disk, an optical disk, flash memory, a floppy disk, or other forms of non-volatile memory known in the art.

[0083] The input device 850 can receive input from the user via traditional input methods such as a keyboard and mouse, and newer input methods such as touch input, voice input, and image input. For example, the input device 850 may include a keyboard, mouse, touchscreen, microphone, or any other device that can detect input from the user and transmit the detected input to the data processing device 800. The output device 860 can provide the user with the output of the data processing device 800 via a visual, auditory, or tactile channel. The output device 860 may include, for example, a display, touchscreen, speaker, vibration generator, or any other device that can provide output to the user. The network interface 870 can communicate with external devices via a wired or wireless network.

[0084] Using the methods described herein, a data processing device 800 according to one or more embodiments can acquire an inference model that performs robust inference even when the video captured by the camera 830 has any type of distortion and / or texture, and can perform inference with high accuracy on the video captured by the camera 830.

[0085] Referring to Figure 9, the vehicle 900 may include all forms of transportation that run on roads or railway tracks. The vehicle 900 may include, for example, automobiles and motorized bicycles, and automobiles may include various forms such as passenger cars, cargo vans and motorcycles. The vehicle 900 may also include autonomous vehicles, intelligent vehicles, and vehicles equipped with driving assistance systems. In this specification, the vehicle 900 refers to a vehicle equipped with a data processing device 910.

[0086] The data processing device 910 includes memory 911 (for example, one or more memory units), a processor 913 (for example, one or more processors), and a camera 915.

[0087] Memory 911 may contain instruction words that can be read from the computer. The processor 913 performs the operations described later by executing the instruction words stored in memory 911. Memory 911 may be volatile memory or non-volatile memory.

[0088] The processor 913 may include, for example, a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit) as a device that executes instructions or programs and controls the data processing device 910.

[0089] The processor 913 can perform inference on the video acquired from the camera 915. The processor 913 may perform one or more of the operations and methods described with reference to Figures 1 to 8. The camera 915 is a camera that captures images in one direction, such as the front, rear, or side of the vehicle 900, and the vehicle 900 may be equipped with one or more cameras. For example, four fisheye cameras can be placed on the vehicle 900 at 90-degree intervals to capture 360-degree images of the area around the vehicle 900. The cameras 915 installed on the vehicle 900 vary depending on the size, shape, and type of the vehicle 900, and the distortion and / or texture shown in the captured images will also vary accordingly. By using the method described above, it is possible to obtain an inference model that can perform robust inference regardless of the distortion and / or texture of the video captured through the camera 915, thereby enabling inference with high accuracy on the video captured by the camera 915.

[0090] Thus, the information described herein can be applied without limitation to SVM (Surround View Monitor) systems, RVC (Rear View Camera) systems, ADAS (advanced driver assistance systems), or IVI (in-vehicle infotainment) chips.

[0091] The embodiments described above are embodied in hardware components, software components, or combinations of hardware and software components. For example, the devices and components described in these embodiments are embodied using one or more general-purpose or special-purpose computers, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPA (field programmable array), PLU (programmable logic unit), microprocessor, or different devices that execute and respond to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the OS. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For convenience of understanding, the processing device may sometimes be described as being used as a single unit, but a person with ordinary skill in the art will understand that the processing device includes multiple processing elements and / or multiple types of processing elements. For example, the processing device includes multiple processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

[0092] Software includes computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired and can instruct the processing unit independently or in combination. Software and / or data can be permanently or temporarily embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, in order to be interpreted by the processing unit and to provide instructions or data to the processing unit. Software can be distributed across a network of computer systems, stored and executed in a distributed manner. Software and data can be stored on a recording medium readable by one or more computers.

[0093] The method according to this embodiment is embodied in the form of program instructions that are implemented via various computer means and recorded on a computer-readable recording medium. The recording medium includes program instructions, data files, data structures, etc., individually or in combination. The recording medium and program instructions may be specifically designed and configured for the purposes of the present invention, or they may be known and usable by those skilled in the art who have technology in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code executed by a computer using an interpreter or the like.

[0094] The hardware device described above may be configured to operate as one or more software modules to perform the operations shown in the present invention, and vice versa.

[0095] As described above, although embodiments have been illustrated with limited drawings, a person with ordinary skill in the art can apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or assembled in a different manner than described, or replaced or substituted with other components or equivalents, and still achieve suitable results.

[0096] Therefore, other embodiments, other embodiments, and claims equivalent to those described below also fall within the scope of the claims.

Claims

1. A data processing method that uses a neural network and is executed by a processor, A step of determining a first deformed image by deforming the first image based on the second image such that the distortion of the first image corresponds to the distortion of the second image, wherein the first image and the second image have different distortions. The steps include: deforming the first deformed image so that the distortion of the first deformed image corresponds to the distortion of the first image, and determining the first re-deformed image; A step of training a first distortion field generator that determines a first relative distortion field that represents the relative distortion from the first image to the second image based on the loss between the first re-deformed image and the first image, and a second distortion field generator that determines a second relative distortion field that represents the relative distortion from the second image to the first image, A data processing method that includes this.

2. The steps include: determining a second deformed image by deforming the second image so that the distortion of the second image corresponds to the distortion of the first image; The steps include: determining a second re-deformed image by deforming the distortion of the second deformed image so that it corresponds to the distortion of the second image; A step of training the first distortion field generator and the second distortion field generator based on the loss between the second re-deformed image and the second image, The data processing method according to claim 1, further comprising:

3. The data processing method according to claim 1, wherein the initial parameters of the first distortion field generator are determined through training based on the loss between a third deformed image, which is deformed from a first image based on a fisheye simulation, and the first deformed image.

4. The data processing method according to claim 1, wherein the first relative distortion field and the second relative distortion field have the characteristics of an inverse transformation relationship.

5. The data processing method according to claim 1, wherein the step of determining the first deformed image is to apply the first relative distortion field to the first image to determine the first deformed image having the distortion of the second image.

6. The data processing method according to claim 1, wherein the step of determining the first re-deformed image is to apply the second relative distortion field to the first deformed image to determine the first re-deformed image having the distortion of the first image.

7. The data processing method according to claim 1, wherein the first video and the second video are not paired videos containing different content and / or different scenes.

8. The data processing method according to claim 1, wherein the step of training the first distortion field generator and the second distortion field generator is to train the first distortion field generator and the second distortion field generator by unsupervised learning without any information about the distortions of the first video and the second video, respectively.

9. The data processing method according to claim 1, further comprising the step of training an inference model for the second video based on a deformed label video obtained by deforming the label video such that the distortion of the label video corresponding to the first video corresponds to the distortion of the second video, and the first deformed video.

10. The data processing method according to claim 1, further comprising the step of training an inference model for the second image based on an unsupervised domain adaptation scheme using a fourth deformed image obtained by deforming the first image so that the distortion of the first image corresponds to the distortion and texture of the second image, a deformed labeled image, and the second image.

11. The data processing method according to claim 1, wherein the second video does not have a corresponding labeled video.

12. A data processing method using a neural network, which is executed by a processor, A step of determining a relative distortion field that represents the relative distortion from the source image to the target image, based on a source image and a target image having different distortions, The steps include applying the relative distortion field to the source image to determine a distorted source image having the distortion of the target image, A data processing method that includes this.

13. A computer program that causes a computer to execute the data processing method described in any one of claims 1 to 12.

14. A data processing device including one or more processors, The one or more processors deform the first image based on the second image so that the distortion of the first image corresponds to the distortion of the second image, thereby determining a first deformed image, and the first image and the second image have different distortions. The one or more processors deform the first deformed image so that the distortion of the first deformed image corresponds to the distortion of the first image, thereby determining the first re-deformed image. A data processing device comprising one or more processors, which trains a first distortion field generator that determines a first relative distortion field representing the relative distortion from the first image to the second image based on the loss between the first re-deformed image and the first image, and a second distortion field generator that determines a second relative distortion field representing the relative distortion from the second image to the first image.

15. The data processing apparatus according to claim 14, wherein one or more processors deform the second image to determine a second deformed image so that the distortion of the second image corresponds to the distortion of the first image, deform the second deformed image to determine a second re-deformed image so that the distortion of the second deformed image corresponds to the distortion of the second image, and train the first distortion field generator and the second distortion field generator based on the loss between the second re-deformed image and the second image.

16. The data processing apparatus according to claim 14, wherein the initial parameters of the first distortion field generator are determined through training based on the loss between a third deformed image, which is deformed from a first image based on a fisheye simulation, and the first deformed image.

17. The data processing apparatus according to claim 14, wherein the first relative distortion field and the second relative distortion field have the characteristics of an inverse transformation relationship.

18. The data processing apparatus according to claim 14, wherein one or more processors apply the first relative distortion field to the first image to determine the first deformed image having the distortion of the second image.

19. The data processing apparatus according to claim 14, wherein one or more processors apply the second relative distortion field to the first deformed image to determine the first re-deformed image having the distortion of the first image.

20. The data processing device according to any one of claims 14 to 19, wherein the data processing device is one of a mobile phone, smartphone, PDA, netbook, tablet computer, laptop computer, mobile device, smartwatch, smart band, smart glasses, wearable device, desktop computer, server, computing device, television, smart television, refrigerator, home appliance, door lock, security device, or vehicle.

Images