Image processing apparatus and method

Abstract

The invention relates to an image processing apparatus and method. In an embodiment, an image processing apparatus may include a memory storing computer executable code and at least one processor configured to access the memory and execute the code. The code includes instructions to cause the at least one processor to generate an input image by pre-processing the image based on detecting a part from an image around the part, apply the input image to an image analysis model, and analyze the input image based on the detected part. The image is processed to obtain a defect type and a defect size of the part detected from the image, and the defect type and the defect size are applied to the input image to generate an output image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Patent Application No. 10-2023-0090538, filed on Jul. 12, 2023, which is hereby incorporated by reference. Technical Field

[0003] The present invention relates to an image processing device and a method thereof. Background Art

[0004] The process of identifying defects in used automobile parts is a process in which workers directly visually inspect used parts and identify defects in used parts. In particular, it is estimated that tens of thousands of vehicles are scrapped in Korea every year.

[0005] Even though dozens of parts from abandoned vehicles can be salvaged, they are still being abandoned.

[0006] Since workers directly identify defects of used auto parts with their own eyes in the above identification process, it takes a long time. For example, due to the nature of used parts, one item requires one identification process. In addition, defect information based on workers' subjective judgment may not be objective.

[0007] In order to solve these problems, it is necessary to develop a technology that automatically identifies defects of used automobile parts. In addition, it is necessary to develop a technology that automatically records the identified defect information to eliminate errors in handwritten records. Summary of the invention

[0008] The present invention relates to an image processing device and a method thereof, and more particularly, to a technology for obtaining a defect type and a defect size of a component included in an image.

[0009] In the embodiments of the present invention, the above-mentioned problems occurring in the prior art can be solved while completely retaining the advantages achieved by the prior art.

[0010] In an embodiment of the present invention, an image processing device and method for obtaining the defect type and defect size of a detected component based on detecting the component from an image can reduce the time used to identify defects in used automotive parts, eliminate the user's subjective judgment to improve the reliability of the status of used parts, and maintain a certain inspection quality.

[0011] In an embodiment of the present invention, an image processing device and method thereof for calculating the availability of parts and the extent of damage to parts based on the defect size obtained from an image analysis model can provide stakeholders related to the transaction of used automotive parts with accurate and reliable information about the status of used parts, and eliminate the user's subjective judgment to overcome the limitations of inspection results based on user experience.

[0012] In an embodiment of the present invention, an image processing device and method thereof for improving the accuracy of information input for selling used parts through an interface can receive or provide images, develop a healthy used auto parts trading ecosystem to develop the parts industry, and reduce carbon emissions due to the active used parts industry.

[0013] The technical problems solved by the embodiments of the present invention are not necessarily limited to the above-mentioned problems, and those skilled in the art to which the present invention belongs can clearly understand any other technical problems not mentioned herein from the following description.

[0014] According to an embodiment of the present invention, an image processing device may include a memory storing computer executable instructions and at least one processor accessing the memory and executing the instructions. The at least one processor may generate an input image by preprocessing the image based on detecting a component from an image around the component, may apply the input image to an image analysis model to obtain a defect type and defect size of the component detected from the image, and may apply the defect type and defect size to the input image to generate an output image.

[0015] In an embodiment, the at least one processor may generate an input image by using a cropped image including a predetermined area, and may apply normalization to pixel values ​​included in the cropped image according to a predetermined pixel interval to change the pixel values, wherein the predetermined area includes all parts of the detected component.

[0016] In an embodiment, the at least one processor may change a channel of the cropped image whose pixel values ​​are changed to a first channel or a second channel, and may change a plurality of pixel values ​​included in the cropped image whose pixel values ​​are changed to a one-dimensional array in an order of channel features included in the first channel based on changing the channel of the cropped image whose pixel values ​​are changed to a first channel including a plurality of channels.

[0017] In an embodiment, the at least one processor may concatenate the image, the input image, and the output image, and may send the concatenated image to an image storage server.

[0018] In an embodiment, the at least one processor may adjust at least one of a color feature of an input image, an edge feature of an input image, a polygon feature of an input image, a saturation feature of an input image, a color temperature feature of an input image, a definition feature of an input image, a contrast feature of an input image, a blur feature of an input image, or a brightness feature of an input image, or any combination thereof, and apply the image having at least one of the adjusted features to an image analysis model to obtain a defect type and a defect size.

[0019] In an embodiment, the defect types may include a first defect, a second defect, a third defect, and a fourth defect, the first defect including surface scratches on a painted surface area of ​​a component, the second defect including deformation of a component, in which a portion or all of the shape of a component may be different from an existing shape of the component, and the fourth defect including a gap generated at a joint of a component.

[0020] In an embodiment, the at least one processor may calculate the availability of the component and the damage range of the component based on the obtained defect size.

[0021] In an embodiment, the at least one processor may provide an interface to receive images from a parts supplier supplying used parts, and may provide the parts supplier with the output image, defect type, and defect size stored in the image storage server through the interface.

[0022] In an embodiment, the at least one processor can generate an image analysis model including multiple pooling layers and multiple non-pooling layers based on the image analysis model being a U-Net neural network model, and can train the image analysis model using a loss function defined as the sum of losses between an input image and an output image.

[0023] In an embodiment, the plurality of pooling layers may be connected with a nonlinearity of a rectified linear unit (ReLU) function included in the image analysis model. The image analysis model may include skip connections from the plurality of pooling layers to the plurality of non-pooling layers.

[0024] According to an embodiment of the present invention, an image processing method may include generating an input image by preprocessing the image based on detecting a component from an image around the component, applying the input image to an image analysis model to obtain a defect type and defect size of the component detected from the image, and applying the defect type and defect size to the input image to generate an output image.

[0025] In an embodiment, generating the input image may include generating the input image by using a cropped image including a predetermined area, applying normalization to pixel values ​​included in the cropped image according to a predetermined pixel interval to change the pixel values, wherein the predetermined area includes all parts of the detected component.

[0026] In an embodiment, the image processing method may further include changing a channel of the cropped image whose pixel values ​​are changed to a first channel or a second channel, based on changing the channel of the cropped image whose pixel values ​​are changed to a first channel including multiple channels, and changing the multiple pixel values ​​included in the cropped image whose pixel values ​​are changed to a one-dimensional array in an order of channel features included in the first channel.

[0027] In an embodiment, the image processing method may further include stitching the image, the input image and the output image, and sending the stitched image to an image storage server.

[0028] In an embodiment, obtaining the defect type and defect size may include adjusting at least one of a color feature of an input image, an edge feature of an input image, a polygon feature of an input image, a saturation feature of an input image, a color temperature feature of an input image, a clarity feature of an input image, a contrast feature of an input image, a blur feature of an input image, a brightness feature of an input image, or any combination thereof, and applying the image with at least one of the features adjusted to an image analysis model to obtain the defect type and defect size.

[0029] In an embodiment, the defect types may include a first defect, a second defect, a third defect, and a fourth defect, the first defect including surface scratches on a painted surface area of ​​a component, the second defect including deformation of a component, in which a portion or all of the shape of the component is different from the existing shape of the component, and the fourth defect including a gap generated at a joint of the component.

[0030] In an embodiment, the image processing method may further include calculating the availability of the component and the damage range of the component based on the obtained defect size.

[0031] In an embodiment, the image processing method may further include providing an interface to receive the image from a parts supplier supplying used parts, and providing the output image, defect type, and defect size stored in the image storage server to the parts supplier through the interface.

[0032] In an embodiment, the image processing method may further include generating an image analysis model including multiple pooling layers and multiple non-pooling layers based on that the image analysis model is a U-Net neural network model, and training the image analysis model using a loss function defined as the sum of the losses between the input image and the output image. The multiple pooling layers may be connected to the nonlinearity of the rectified linear unit (ReLU) function included in the image analysis model. The image analysis model may include jump connections from multiple pooling layers to multiple non-pooling layers.

[0033] According to an embodiment of the present invention, an image storage server may include: receiving at least one of a defect type and a defect size of a vehicle part, an image of the part, or any combination thereof from an image processing device, wherein the defect type and the defect size are obtained by applying the image to an image analysis model; providing an interface to receive inquiries about information about used parts from a parts supplier that supplies used parts; and providing the parts supplier with at least one of the defect type, the defect size, the image, or any combination thereof based on the inquiries about information about used parts received through the interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and other features and advantages of the embodiments of the present invention will become more apparent through the following detailed description taken in conjunction with the accompanying drawings, in which:

[0035] Figure 1 is a schematic diagram showing an image processing apparatus according to an embodiment of the present invention;

[0036] Figure 2 is a flowchart for describing an image processing method according to an embodiment of the present invention;

[0037] Figure 3 is a schematic diagram showing a method for training an image analysis model in an image processing device according to an embodiment of the present invention;

[0038] Figure 4 is a schematic diagram showing a process of processing an image in an image processing apparatus according to an embodiment of the present invention;

[0039] Figure 5 is a schematic diagram illustrating a method for applying an input image to an image analysis module in an image processing device according to an embodiment of the present invention;

[0040] Figure 6 is a schematic diagram showing a process of querying an output image in an image processing apparatus according to an embodiment of the present invention;

[0041] Figure 7is a schematic diagram showing an output image generated in an image processing apparatus according to an embodiment of the present invention;

[0042] Figure 8 is a schematic diagram showing defect information paired with an output image generated in an image processing apparatus according to an embodiment of the present invention;

[0043] Fig. 9 is a schematic diagram showing an output image generated in an image processing apparatus according to an embodiment of the present invention;

[0044] Fig.10 is a schematic diagram showing defect information paired with an output image generated in an image processing apparatus according to an embodiment of the present invention; and

[0045] Fig.11 A computing system related to an image processing apparatus and method thereof according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0046] Hereinafter, some embodiments of the present invention will be described in detail with reference to the example drawings. In the accompanying drawings, the same reference numerals will be used throughout to represent the same or equivalent elements. In addition, in order to avoid unnecessarily blurring the main points of the present invention, a detailed description of known features or functions will be excluded. Hereinafter, various embodiments of the present invention may be described with reference to the accompanying drawings. However, it should be understood that it is not intended to limit the present invention to a specific implementation form, and various modified forms, equivalent forms and / or alternative forms of the embodiments of the present invention are included. With regard to the description of the accompanying drawings, the same or similar components may be referenced with the same or similar reference numerals.

[0047] When describing the elements of the exemplary embodiments of the present invention, the terms "first", "second", "A", "B", "(a)", "(b)", etc. may be used herein. These terms may only be used to distinguish one component from another, and regardless of the order or priority of the corresponding components, these terms do not necessarily limit the corresponding components. In addition, unless otherwise defined, all terms (including technical terms and scientific terms) used herein may be interpreted as the usual meaning of the field to which the present invention belongs, for example. It is understood that the terms used herein may be interpreted as having the same meaning as in the context of the relevant field and the present invention, and should not be interpreted as having an idealized or overly formalized meaning, unless clearly defined herein. For example, terms such as "first", "second", etc. used in the present invention may be used to refer to various components, regardless of the order and / or priority, and are used to distinguish one component from another, but do not necessarily limit these components. For example, a first user device and a second user device indicate different user devices, regardless of the order and / or priority. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

[0048] In the present invention, the expressions “have”, “may have”, “include” and “comprise” or “may include” and “may include” indicate the existence of corresponding features (for example, components such as values, functions, operations or parts), but do not exclude the existence of additional features.

[0049] It is understood that when a component (e.g., a first component) is referred to as being "(operably or communicatively) coupled to / coupled to" another component (e.g., a second component) or "connected to" another component, the component may be directly coupled to / coupled to or connected to another component, or an intermediate component (e.g., a third component) may exist. In contrast, when a component (e.g., a first component) is referred to as being "directly coupled to / coupled to" another component (e.g., a second component) or "directly connected to" another component, it is understood that there is no intermediate component (e.g., a third component).

[0050] As used herein, the expression “configured to” may be used interchangeably with, for example, expressions “suitable to,” “capable of,” “designed to,” “appropriate to,” “formed to,” or “able to,” depending on circumstances.

[0051] The term "configured to" cannot mean only "specifically designed to" in hardware. On the contrary, the statement "device is configured to" can mean that the device "can" run with another device or other components. For example, "a processor configured to perform A, B and C" can represent a general-purpose processor (e.g., a central processing unit (CPU) or an application processor) that can perform the corresponding operations by executing one or more software programs or codes that can be stored in a storage device. The terms used in the present invention can be used only to describe the specified embodiment and are not necessarily intended to limit the scope of another embodiment. Unless the context clearly indicates otherwise, a term in the singular form may include a plural form. All terms used herein (including technical terms or scientific terms) may have the same meaning as the meanings generally understood by those skilled in the art described in the present invention. It will be further understood that the terms defined in the dictionary and commonly used may also be interpreted as the usual meaning in the corresponding related art, rather than idealized or overly formalized, unless explicitly defined in various embodiments of the present invention herein. In some cases, even if a term is a term defined in the specification, the term is not necessarily interpreted as excluding embodiments of the present invention.

[0052] In the present invention, the expressions "A or B", "at least one of A or / and B", or "one or more of A or / and B", etc. may include any and all combinations of the relevant listed items. For example, the term "A or B", "at least one of A and B", or "at least one of A or B" may refer to all cases including the case of at least one A (1), the case of at least one B (2), or the case of both at least one A and at least one B (3). In addition, when describing the embodiments of the present invention, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", "at least one of A, B or C", and "at least one of A, B or C, or any combination thereof" may include any one or all possible combinations of the items listed together in the corresponding one of these phrases. In particular, a phrase such as "at least one of A, B or C, or any combination thereof" may include "A", "B" or "C" as various combinations, or "AB", "BC", "AC", or "ABC".

[0053] In the following, reference will be made to Figures 1 to 11 Embodiments of the present invention will be described in detail.

[0054] Figure 1 is a schematic diagram showing an image processing apparatus according to an embodiment of the present invention.

[0055] According to an embodiment, the image processing device 100 may include a processor 110 , a memory 120 including code or instructions 122 , and a communication device 130 .

[0056] The image processing device 100 may indicate a device for obtaining a defect type and defect size of an automobile part detected from an image based on detecting a part from an image including automobile parts. For example, based on detecting a part from an image around the part, the image processing device 100 may preprocess the image to generate an input image. The image processing device 100 may detect parts included in an image based on a histogram of gradient (HOG) for representing the distribution direction of brightness as a histogram and using the distribution direction of brightness as a feature vector. The HOG algorithm may be an algorithm for extracting features of a part object, which may indicate an algorithm for dividing an image into squares in the form of a chessboard and calculating a local histogram in the brightness distribution direction in each square grid.

[0057] The image processing device 100 may apply the input image to the image analysis model to obtain the degree to which the state of the component when the component is manufactured (e.g., the original new working condition component) is different from the state of the component detected from the image. For example, the image analysis model may indicate a model that is trained by a machine learning method and processes the image to obtain information (e.g., the defect type and defect size of the component) from the image. Figure 5 A detailed description is given of the image analysis model. The image processing apparatus 100 may obtain the state of the component detected from the image by using the defect type and defect size of the component detected from the image.

[0058] The defect types of the parts may include: a first defect including surface scratches on the painted surface area of ​​the parts, a second defect including deformation of the parts, a third defect in which a part or all of the shape of the parts is different from the existing shape of the parts, and a fourth defect including a gap generated at the joint of the parts.

[0059] The first defect may be a surface scratch on the painted surface area of ​​the component, which may include, for example, a defect in which the coating on the component has peeled off, a defect in which a layer sprayed on the component has peeled off, a defect in which scratches exist on the iron plate, or a defect in which a color different from the existing color of the component exists, or at least one of any combination thereof.

[0060] The second defect may be about deformation of the component, which may include, for example, at least one of a defect in a digging area in the component, a defect in a recessed area in the component, a defect in an extruded deformation area in the component, or a defect in a bent area in the component, or any combination thereof.

[0061] The third defect may be that the shape of the component is different from the new original shape, which may include at least one of a defect in which the component is broken, a defect in which a damaged area exists on the component, a defect in which a torn area exists on the component, or a defect in which a perforated area exists in the component, or any combination thereof.

[0062] The fourth defect may be a gap generated at a joint of the components, which may include, for example, at least one of a state in which each of the plurality of components is misaligned or a state in which a joint between the plurality of components is disconnected, or any combination thereof.

[0063] The image processing device 100 can calculate the availability of the parts according to the defect size of the defect type. For example, the defect size can indicate the number of pixels of the defect degree in the image. The image processing device 100 can calculate the availability according to the number of pixels of the defect degree in the image based on the defect type and defect size that can be obtained by applying the input image to the image analysis model. Specifically, for example, when the defect size is less than 400 pixels, the image processing device 100 can calculate the conclusion that the parts with the defect size are similar to new products and are available parts. In addition, for example, when the defect size is greater than or equal to 400 pixels and less than 1500 pixels, the image processing device 100 can calculate the conclusion that the parts with the defect size have slight damage but are available parts. For example, when the defect size is greater than or equal to 1500 pixels and less than 4000 pixels, the image processing device 100 can calculate the conclusion that the parts with the defect size have a lot of damage and are unavailable parts. Specifically, for example, when the defect size is greater than or equal to 4000 pixels, the image processing device 100 can calculate the conclusion that the parts with the defect size are damaged and are unavailable parts. For reference, for the convenience of description in the specification, the defect size is described as the number of pixels of an area indicating a defect in an image, but is not necessarily limited thereto.

[0064] The image processing device 100 may apply the defect type and defect size to the input image to generate an output image. For example, the input image may indicate an image before being applied to an image analysis model. The image processing device 100 may generate an input image by using preprocessing of the image and may apply the generated input image to the image analysis model to obtain the defect type and defect size. The image processing device 100 may apply the obtained defect type and the obtained defect size to the input image. The image processing device 100 may apply the defect type and defect size to the input image to generate an output image. However, the method for generating an output image is not necessarily limited thereto. For example, the image processing device 100 may apply the input image to the image analysis model to obtain an output image that reflects the defect type and defect size. The following will refer to Figures 7 to 10A detailed description describing an example output image.

[0065] The processor 110 may execute software or code, and may control at least one other component (e.g., hardware or software component) connected to the processor 110. In addition, the processor 110 may perform various data processing or calculations. For example, the processor 110 may store an image, an input image, an output image, a defect type, and a defect size in the memory 120.

[0066] For reference, the processor 110 may perform all operations performed by the image processing apparatus 100. Therefore, for the convenience of description in the specification, the operations performed by the image processing apparatus 100 are mainly described as operations performed by the processor 110.

[0067] In addition, for the convenience of description in the specification, the processor 110 is mainly described as a processor. For example, the image processing device 100 may include at least one processor. Each of the at least one processor may perform part of the operations or all the operations related to the image processing operation.

[0068] The memory 120 may temporarily and / or permanently store various data and / or information required to perform image processing. For example, the memory 120 may store images, input images, output images, defect types, and defect sizes.

[0069] The communication device 130 may assist in performing communication between the image processing device 100 and the image storage server 140. For example, the communication device 130 may include one or more components for communicating between the image processing device 100 and the image storage server 140. For example, the communication device 130 may include a short-range wireless communication unit, a microphone, and the like. At this time, for example, the short-range communication technology may be, but is not necessarily limited to, wireless LAN (Wi-Fi), Bluetooth, ZigBee, Wi-Fi Direct (WFD), Ultra Wideband (UWB), Infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), and the like.

[0070] The image storage server 140 may provide the parts supplier supplying the used parts with the information received from the image processing apparatus 100. For example, the image storage server 140 may receive at least one of a defect type and a defect size of a part, an image, or any combination thereof, which may be obtained by applying an image including an automobile part to an image analysis model, from the image processing apparatus 100. The image received by the image storage server 140 from the image processing apparatus 100 may include an input image.

[0071] The image storage server 140 may provide an interface to receive inquiries about information about used parts from a parts supplier that supplies used parts. The parts supplier that supplies used parts may inquire about information about the defect type, defect size, and image of the parts through the interface provided by the image storage server 140.

[0072] The image storage server 140 may provide at least one of a defect type, a defect size, or an image, or any combination thereof to a parts supplier inquiring for information based on a query for information about a used part received through an interface.

[0073] Figure 2 is a flowchart for describing an image processing method according to an embodiment of the present invention.

[0074] In operation S210, an image processing device (eg, Figure 1 The image processing device 100 may detect the component from the image around the component and pre-process the image to generate an input image. For example, the image may indicate a storage device stored in an image storage server (e.g., Figure 1 The image storage server 140 may store images of used automobile parts and may include used automobile parts. Specifically, the images may mainly include parts with defects in used automobile parts. A third party (e.g., a supplier of used automobile parts) may store images including parts in the image storage server. The image processing device may obtain images provided by the supplier from the image storage server. The image processing device may detect parts from the obtained images. Based on detecting parts from images around parts, the image processing device may pre-process the images to generate input images.

[0075] In operation S220, the image processing device may apply the input image to the image analysis model to obtain the defect type and defect size of the parts detected from the image. For example, the image processing device may arrange the pixels included in the input image in the order of channel features. The image processing device may obtain an input vector including the above pixels by using the arrangement of the pixels included in the input image in the order of channel features. However, the input vector is not necessarily limited to this. For example, the image processing device may extract a plurality of feature vectors from the input image. The plurality of feature vectors may include elements corresponding to the number of nodes of the input layer included in the image analysis model. The image processing device may apply the input vector and / or feature vector to the nodes of the input layer included in the image analysis model to forward propagate the input vector and / or feature vector to the output layer through a link having a connection weight between a node of any layer and a node of another layer. Based on the input vector and / or feature vector forward propagated in the image analysis model, the defect type and defect size of the parts can be obtained from the output layer included in the image analysis model.

[0076] In operation S230, the image processing device may apply the defect type and defect size to the input image to generate an output image. For example, the image may indicate an original image including automobile parts. The output image may indicate an image reflecting the defect type and defect size extracted by the image analysis model in the input image. The image processing device may display the area where the parts related to the defect type and defect size are located on the input image by a predetermined method.

[0077] Figure 3 is a schematic diagram showing a method for training an image analysis model in an image processing device according to an embodiment of the present invention.

[0078] In operation S310, a third party (eg, a supplier of used automobile parts) may upload an image to an image storage server (eg, Figure 1 The third party may upload an image file that conforms to at least one of JPEG (jpg, jpeg), PNG (png), BMP (bmp) or any combination thereof. In addition, the third party may upload an image file of a predetermined size or larger. However, the image file upload requirement is not necessarily limited thereto. For example, the third party may upload an image file according to the conditions required in the image storage server. The above conditions may indicate a value initially set in the image storage server. This may be changed later by the image processing device.

[0079] In operation S320, an image processing device (eg, Figure 1The image processing device 100) can pre-process the image uploaded to the image storage server. Figure 5 Give a detailed description of the preprocessing of images.

[0080] In operation S330, the image processing device may detect damaged parts among the second-hand parts from the preprocessed image. Thus, in operation S340, the image processing device may generate an image for training the image analysis model (e.g., an image for training). In addition, in operation S350, the image processing device may store the defect type and defect size regarding the damaged parts as text. The image processing device may generate an image for training the image analysis model, and train the image analysis model using the defect type and defect size.

[0081] In operation S360, the image processing device may train an image analysis model. Exemplarily, the image analysis model may include a neural network. The neural network may include multiple layers. Each layer may include multiple nodes. The nodes may have node values ​​determined based on an activation function. A node of any layer may be connected to a node of another layer (e.g., another node) via a link (e.g., a connection edge) having a connection weight. The node value of the node may be propagated to other nodes via the link. In the reasoning operation of the neural network, the node value may be propagated forward from the previous layer to the next layer.

[0082] Exemplarily, a forward propagation calculation in an image analysis model may indicate a calculation that propagates a node value based on input data in a direction from an input layer to an output layer of the image analysis model. In other words, the node value of a node may be propagated (e.g., forward propagated) to a node (e.g., a next node) of a next layer connected to the node by a connecting edge. For example, a node may receive a value weighted by a connection weight from a previous node (e.g., multiple nodes) connected by a connecting edge.

[0083] The node value of the node can be determined based on applying the activation function to the sum (e.g., weighted sum) of the weighted values ​​received from the previous nodes. The parameters of the neural network can illustratively include the above-mentioned connection weights. The parameters of the neural network can be updated to change in a direction (e.g., a direction of minimizing loss) directed by an objective function (to be described below).

[0084] The trained image analysis model may indicate a model trained by using machine learning and may be a trained machine learning model that outputs a training output (e.g., defect type and defect size) from a training input (e.g., an input image). Specifically, the image processing device may be a trained machine learning model that outputs pixels of components (e.g., the position of components in the image) and states of components (e.g., the degree of defects of components in the image) included in the training input. However, the trained image analysis model is not necessarily limited to this. For example, the trained image analysis model may be a trained machine learning model that outputs a training output (e.g., an output image to which defect type and defect size are applied) from a training input (e.g., an input image).

[0085] A machine learning model (e.g., a trained image analysis model) can be generated using machine learning. For example, a learning algorithm can include, but is not necessarily limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0086] The machine learning model may include multiple artificial neural network layers. Specifically, the trained image analysis model may include a shared layer and multiple classifier layers (e.g., task-specific layers) connected to the shared layer, wherein the shared layer includes at least one convolution operation. For example, the artificial neural network may be, but is not necessarily limited to, a deep neural network (DNN), a convolutional neural network (CNN), a U-net for image segmentation (U-net), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q network, or a combination of at least one thereof.

[0087] For reference, based on the image analysis model being a U-Net neural network model, the image analysis model may include multiple pooling layers and multiple non-pooling layers. The image processing device may train the image analysis model using a loss function defined as the sum of the losses between the input image and the output image. Regarding the loss function, a cross entropy loss function may be used. Multiple pooling layers may be connected to the nonlinearity of a rectified linear unit (ReLU) included in the image analysis model. The image analysis model may include jump connections from multiple pooling layers to multiple non-pooling layers.

[0088] The image analysis model used by the image processing device may have a basic U-Net backbone network. U-Net may consist of an encoder, a decoder, and their connections. The encoder may typically extract a hierarchical structure of an image feature map from low complexity to high complexity, while the decoder may transform features and reconstruct outputs from low resolution to high resolution.

[0089] The image analysis model may include a convolution layer for performing a linear transformation operation, a batch normalization (BN) layer for performing a normalization operation, a rectified linear unit (ReLU) layer for performing a nonlinear function operation, and a channel concatenation layer or a channel sum layer for concatenating the outputs of multiple layers.

[0090] For supervised learning, the above-mentioned machine learning model can be trained based on training data including a pair of training inputs and training outputs mapped to the training inputs. For example, the machine learning model can be trained to output a training output from the training input. The machine learning model can output a temporary output in response to the training input when being trained, and can be trained to minimize the loss of the temporary output and the training output (e.g., the training target). The parameters of the machine learning model in the learning process (e.g., the connection weights between nodes / layers in a neural network) can be updated according to the loss. This learning can be performed in the image processing device itself that executes the machine learning model or can be performed by using a separate server. The machine learning model that performs training (e.g., the trained image analysis model) can be stored in a memory (e.g., Figure 1 in the memory 120).

[0091] For reference, the image processing device can adjust at least one of the color features of the training input, the edge features of the training input, the polygon features of the training input, the saturation features of the training input, the color temperature features of the training input, the definition features of the training input, the contrast features of the training input, the blur features of the training input, the brightness features, or any combination thereof, and can train an image analysis model for images including various features.

[0092] Figure 4 is a schematic diagram illustrating a process of processing an image in an image processing apparatus according to an embodiment of the present invention.

[0093] In operation S410, at least one third party (eg, a supplier of used automobile parts) may upload an image including used automobile parts to an image storage server (eg, Figure 1 image storage server 140). Image processing device (e.g., Figure 1 The image processing device 100) may provide an interface to receive images from a parts supplier that supplies used parts.

[0094] In operation S420, the image processing apparatus may store the image uploaded by the third party in the internal storage space of the image storage server. Thus, the image processing apparatus may manage the image including defect information by using the server, thereby minimizing errors generated by the user's handwritten record.

[0095] In operation S430, the image processing apparatus may respond to a user's request for image preprocessing and a request for image analysis. For example, in response to the user's request, in operation S440, the image processing apparatus may preprocess the image to generate an input image, and may perform image analysis. For example, the image processing apparatus may apply the input image to an image analysis model to obtain a defect type and defect size of a component detected from the image, thereby performing image analysis.

[0096] In operation S450, the image processing apparatus may stitch the image, the input image, and the output image, and may send the stitched image to the image storage server. The output image may include an image reflecting the defect type and defect size obtained by the image processing apparatus applying the input image to the image analysis model.

[0097] Figure 5 is a schematic diagram illustrating a method for applying an input image to an image analysis module in an image processing apparatus according to an embodiment of the present invention.

[0098] In operation S510, based on detecting a component from an image around the component, an image processing device (eg, Figure 1 The image processing apparatus 100 may pre-process the image. For example, the image processing apparatus may pre-process the image through operations S520 to S550 to be described below.

[0099] In operation S520, the image processing apparatus may adjust the size of the image. The image processing apparatus may generate an input image by using a cropped image including a selected (eg, preset or predetermined) area including all portions of the detected part.

[0100] In operation S530, the image processing device may apply normalization to the pixel values ​​included in the cropped image according to the selected (e.g., preset or predetermined) pixel interval to change the pixel value. However, the normalization of the cropped image is not necessarily limited to this. For example, the image processing device may apply normalization to the pixel values ​​included in the cropped image according to the selected (e.g., preset or predetermined) binary pixel. The pixel of the cropped image to which the normalization is applied may indicate, for example, at least one of a pixel value indicating "0" or a pixel value indicating "1".

[0101] In operation S540, the image processing apparatus may adjust the number of channels of the cropped image. For example, the image processing apparatus may change the channel whose pixel value of the cropped image is changed by normalization to the first channel or the second channel. For example, the first channel may indicate an RGB channel, and the second channel may indicate a grayscale channel.

[0102] In operation S550, the image processing apparatus may change the multiple pixel values ​​included in the cropped image whose pixel values ​​are changed into a one-dimensional array in the order of the channel features included in the first channel based on the channel of the cropped image whose pixel values ​​are changed being changed into a first channel including a plurality of channels. In other words, the image processing apparatus may obtain an input vector including the above-mentioned pixels by using the arrangement of the pixels included in the input image in the order of the channel features. In addition, the image processing apparatus may change the multiple pixel values ​​included in the cropped image whose pixel values ​​are changed into a one-dimensional array in the order in which the multiple pixels are stored based on the channel of the cropped image whose pixel values ​​are changed being changed into a second channel including a plurality of channels.

[0103] Finally, in operations S560 to S580, the image processing apparatus may apply the pre-processed input image to the image analysis model to obtain the defect type and defect size of the component detected from the image.

[0104] Figure 6 is a schematic diagram illustrating a process of querying an output image in an image processing apparatus according to an embodiment of the present invention.

[0105] In operation S610, at least one third party (eg, a supplier of used automobile parts or a buyer of used automobile parts) may query information of used automobile parts. Figure 1 The image processing apparatus 100 may provide the at least one third party with the information stored in the image storage server (eg, Figure 1 The output image, defect type and defect size in the image storage server 140).

[0106] In operation S620, the image processing apparatus may obtain information of the second-hand parts from the image storage server. For example, the image processing apparatus may obtain information of the second-hand parts based on the output image, the defect type, and the defect size.

[0107] In operation S630, the image processing apparatus may send the information of the second-hand parts queried by at least one third party as an output image. For example, the image processing apparatus may provide the second-hand parts information related to the obtained output image, the obtained defect type, and the obtained defect size to at least one third party.

[0108] Figure 7 is a schematic diagram showing an output image generated in the image processing apparatus according to the embodiment of the present invention.

[0109] Image processing device (e.g. Figure 1 The image processing device 100 may apply the input image to the image analysis model to obtain the defect type and defect size of the component detected from the image, and may apply the defect type and defect size to the input image to generate the output image 700. For example, the output image 700 may include at least one second-hand automobile component. Figure 7 , the output image 700 may include a first component 710 and a second component 720. For reference, for the convenience of description in the specification, the used automobile components are mainly described as damaged and defective components.

[0110] The first component 710 may indicate a hood (engine cover) of a vehicle, and the second component 720 may indicate a front bumper of the vehicle. The image processing device may obtain defect information (e.g., defect type and defect size) of the component included in the original image (e.g., the image before the input image is preprocessed) of the output image 700. The image processing device may apply the defect information to the input image to generate the output image 700.

[0111] In conjunction with the first component 710 , the image processing apparatus may apply the defect type and defect size of the hood to the input image based on the position of the hood of the vehicle. Thus, the image processing apparatus may generate an output image 700 to which the defect information of the first component 710 is applied.

[0112] In combination with the second component 720, the image processing device can apply the defect type and defect size of the front bumper to the input image based on the position of the front bumper of the vehicle. Thus, the image processing device can generate an output image 700 to which the defect information of the second component 720 is applied.

[0113] Figure 8 is a schematic diagram showing defect information paired with an output image generated in an image processing apparatus according to an embodiment of the present invention.

[0114] Image processing device (e.g. Figure 1 The image processing apparatus 100 may provide information stored in an image storage server (eg, Figure 1 The output image 800 and defect information 810 in the image storage server 140 of the output image 800 may include a defect type of a component included in the output image 800 and a defect size corresponding to the defect type.

[0115] In conjunction with the front bumper included in the output image 800, the defect information 810 may include the defect size of the front bumper (e.g., its Figure 8 ) and the defect type of the front bumper (e.g., due to the shape of the part being different from the new or original shape, which is Figure 8 In other words, the defect information 810 may include the second defect, the third defect, and the defect size of the front bumper. In addition, for example, because the defect size of the front bumper is greater than or equal to 1500 pixels and less than 4000 pixels, the image processing device may calculate that the front bumper has a lot of damage and is an unusable component.

[0116] In conjunction with the hood included in the output image 800, the defect information 810 may include the defect size of the hood (e.g., its Figure 8 4905 pixels) and the type of defect in the hood (e.g., due to the shape of the part being different from the new or original shape, which is Figure 8 In other words, the defect information 810 may include the second defect, the third defect, and the defect size of the hood. In addition, for example, because the defect size of the hood is greater than or equal to 4000 pixels, the image processing device can calculate that the hood is damaged and is an unusable component.

[0117] Fig. 9 is a schematic diagram showing an output image generated in the image processing apparatus according to the embodiment of the present invention.

[0118] Image processing device (e.g. Figure 1 The image processing device 100 may apply the input image to the image analysis model to obtain the defect type and defect size of the component detected from the image, and may apply the defect type and defect size to the input image to generate the output image 900. For example, the output image 900 may include at least one second-hand automobile component. Fig. 9 , the output image 900 may include a first component 910 and a second component 920. For example, as a reference, and for the convenience of description in this specification, used automobile components are mainly described as damaged and defective components.

[0119] The first component 910 may indicate a side fender of the vehicle, and the second component 920 may indicate a rear bumper of the vehicle. The image processing device may obtain defect information (e.g., defect type and defect size) of the component included in the original image (e.g., the image before the input image is preprocessed) of the output image 900. The image processing device may apply the defect information to the input image to generate the output image 900.

[0120] In conjunction with the first component 910 , the image processing apparatus may apply the defect type and defect size of the side fender to the input image based on the position of the side fender of the vehicle. The image processing apparatus may generate an output image 900 to which the defect information of the first component 910 is applied.

[0121] In conjunction with the second component 920, the image processing apparatus may apply the defect type and defect size of the rear bumper to the input image based on the position of the rear bumper of the vehicle. The image processing apparatus may generate an output image 900 to which the defect information of the second component 920 is applied.

[0122] Fig.10 is a schematic diagram showing defect information paired with an output image generated in an image processing apparatus according to an embodiment of the present invention.

[0123] Image processing device (e.g. Figure 1 The image processing apparatus 100 may provide a method for storing images on an image storage server (eg, Figure 1 The output image 1000 and defect information 1010 in the image storage server 140 of the output image 1000 may include a defect type of a component included in the output image 1000 and a defect size corresponding to the defect type.

[0124] In conjunction with the side guard included in the output image 1000, the defect information 1010 may include the defect size of the side guard (e.g., its Fig.10 10 pixels in the figure) and the defect type of the side guard (e.g., Fig.10 In other words, the defect information 1010 may include a state and defect size that do not correspond to any of the first to fourth defects of the side guard. In addition, for example, because the defect size of the side guard is less than 1500 pixels, the image processing device can calculate that the side guard has slight damage and is a usable component.

[0125] In conjunction with the rear bumper included in the output image 1000, the defect information 1010 may include the defect size of the rear bumper (e.g., its Fig.10 ) and the defect type of the rear bumper (e.g., Fig.10In other words, the defect information 1010 may include the first defect and defect size of the rear bumper. In addition, for example, because the defect size of the rear bumper is less than 1500 pixels, the image processing device may calculate that the rear bumper has slight damage and is a usable component.

[0126] The above method is part of a vehicle assembly process. During this process, the vehicle is assembled and inspected. When a particular component is determined to be damaged and unusable, the component can be replaced and assembly of the vehicle can continue using the replacement component. The damaged component can then be scrapped or recycled.

[0127] Fig.11 : is a schematic diagram showing a computing system related to an image processing apparatus and method thereof according to an embodiment of the present invention.

[0128] Reference Fig.11 For example, in some embodiments, a computing system 1000 associated with an image processing apparatus and method thereof may include at least one processor 1100, a memory 1300, a user interface input device 1400, a user interface output device 1500, a storage device 1600, and a network interface 1700 interconnected via a bus 1200.

[0129] The processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes codes or instructions stored in the memory 1300 and / or the storage device 1600. The memory 1300 and the storage device 1600 may include various types of volatile or non-volatile storage media. For example, the memory 1300 may include a ROM (Read Only Memory) 1310 and a RAM (Random Access Memory) 1320.

[0130] Therefore, the operations of the methods or algorithms described in conjunction with the embodiments disclosed in the specification can be implemented directly using hardware modules, software modules, or a combination of hardware modules and software modules executed by the processor 1100. The software modules or codes may exist in a storage medium (i.e., the memory 1300 and / or the storage device 1600), such as RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disks, removable disks, and CD-ROMs.

[0131] An example storage medium may be coupled to the processor 1100. The processor 1100 may read information from the storage medium and may write information to the storage medium. Alternatively, the storage medium may be integrated with the processor 1100. The processor and the storage medium may be present in an application specific integrated circuit (ASIC). The ASIC may be present in a user terminal. In another case, the processor and the storage medium may be present in a user terminal as separate components.

[0132] Although the embodiments of the present invention have been described above with reference to the exemplary embodiments and the accompanying drawings, the present invention is not necessarily limited thereto and various modifications and changes may be made by those skilled in the art without departing from the spirit and scope of the present invention as claimed in the appended claims.

[0133] The above-mentioned embodiments can be realized by hardware elements, software elements and / or a combination of hardware elements and software elements.For example, the device, method and assembly described in the embodiment can be realized using a general-purpose computer or a special-purpose computer, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor or any device that can execute instructions and respond. The processing unit can execute an operating system (OS) or a software application running on the OS. In addition, the processing unit can access, store, operate, process and generate data in response to the execution of software or code. It will be understood by those skilled in the art that, although a single processing unit can be shown for ease of understanding, the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or a processor and a controller. In addition, the processing unit may have different processing configurations, such as parallel processors.

[0134] Software may include computer programs, codes, instructions, or one or more combinations thereof, and may configure a processing unit to operate in a desired manner or may command a processing unit independently or collectively. Software and / or data may be permanently or temporarily contained in any type of machine, component, physical device, virtual device, computer storage medium or unit, or transmission signal wave to be interpreted by a processing unit or to provide instructions or data to a processing unit. Software may be dispersed throughout a computer system connected by a network and may be stored or executed in a dispersed manner. Software and data may be recorded in one or more computer-readable storage media.

[0135] The method according to the embodiment can be implemented in the form of a code or program instruction that can be executed by various computer devices, and can be recorded in a computer-readable medium. Computer-readable media can include codes, program instructions, data files, data structures, etc., alone or in combination, and the code or program instructions recorded on the medium can be, for example, specially designed and configured, or can be known and available to technicians in the field of computer software. For example, examples of computer-readable media include magnetic media such as hard disks, floppy disks, and tapes; optical media such as compact disk read-only memory (CD-ROM) disks and digital versatile disks (DVDs); magneto-optical media such as floppy disks; and hardware devices specially configured to store and execute program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, etc. For example, program codes and instructions can include machine code (such as machine code generated by a compiler), high-level code that can be executed by a computer using an interpreter, or a combination thereof.

[0136] The above-described hardware devices may be configured to function as one or more software modules for performing the operations of the embodiments, or vice versa.

[0137] Although some embodiments are described with reference to limited drawings, it is obvious to those skilled in the art that various changes or modifications can be made to the embodiments based on the above description and afterwards. For example, even if the above processes and methods are performed in an order different from the above order, and / or the above elements (e.g., systems, structures, devices or circuits) are combined or connected in a form and mode different from the above, or can be replaced or transformed with other components or equivalents, sufficient effects can be achieved.

[0138] According to at least one embodiment of the present invention, an image processing device can obtain the defect type and defect size of a detected component based on detecting the component from an image to reduce the time for identifying defects in used automotive parts, and can eliminate the user's subjective judgment to improve the reliability of the status of used parts, thereby maintaining a certain inspection quality.

[0139] Furthermore, according to at least one embodiment of the present invention, the image processing device can calculate the availability of parts and the extent of damage to parts based on the defect size obtained from the image analysis model to provide stakeholders involved in the transaction of used automotive parts with accurate and reliable information about the status of used parts, and can eliminate the user's subjective judgment to overcome the limitations of inspection results based on user experience.

[0140] In addition, according to at least one embodiment of the present invention, the image processing device can improve the accuracy of information input for selling used parts through an interface for receiving or providing images, a healthy used auto parts trading ecosystem can be developed to develop the parts industry, and carbon emissions can be reduced due to the active used parts industry.

[0141] Furthermore, various effects directly or indirectly determined by the present invention can be provided.

[0142] Accordingly, other implementations, other embodiments, and equivalents of the claims are within the scope of the appended claims.

[0143] Therefore, the embodiments of the present invention are not necessarily intended to limit the technical spirit of the present invention, but may be provided for illustrative purposes only. The scope of the present invention should be interpreted based on the appended claims, and all technical concepts within the scope equivalent to the claims may be included within the scope of the present invention.

Claims

1. An image processing device, comprising: a memory storing computer executable code; as well as At least one processor is configured to access the memory and execute the code, wherein the code includes instructions for causing the at least one processor to perform the following operations: based on detecting the component from an image around the component, generating an input image by preprocessing the image, applying the input image to an image analysis model to obtain a defect type and a defect size of the component detected from the image, and applying the defect type and the defect size to the input image to generate an output image.

2. The image processing device according to claim 1, wherein: The code includes instructions for causing the at least one processor to generate an input image by using a cropped image including a selected area, and applying normalization to pixel values ​​included in the cropped image according to a selected pixel interval to change the pixel values, wherein the selected area includes all portions of the detected component.

3. The image processing device according to claim 2, wherein: The code includes instructions for causing the at least one processor to: Change the channel of the cropped image whose pixel values ​​are changed to the first channel or the second channel, Based on the channel of the cropped image whose pixel values ​​are changed being changed to a first channel including a plurality of channels, a plurality of pixel values ​​included in the cropped image whose pixel values ​​are changed are changed into a one-dimensional array in an order of channel features included in the first channel.

4. The image processing device according to claim 3, wherein: The code includes instructions for causing the at least one processor to stitch the image, the input image, and the output image, and send the stitched image to an image storage server.

5. The image processing device according to claim 1, wherein: The code includes instructions for causing the at least one processor to perform the following operations: adjusting one or any combination of color features of an input image, edge features of an input image, polygon features of an input image, saturation features of an input image, color temperature features of an input image, sharpness features of an input image, contrast features of an input image, blur features of an input image, and brightness features of an input image, and applying the adjusted image to an image analysis model to obtain a defect type and a defect size.

6. The image processing device according to claim 1, wherein: The defect types include: A first defect, which includes a surface scratch on the painted surface area of ​​the component; Second defects, which include deformation of parts; A third defect, wherein at least a portion of the shape of the component differs from an existing shape of the component; and The fourth defect includes gaps generated at the joints of parts.

7. The image processing device according to claim 1, wherein: The code includes instructions for causing the at least one processor to calculate the availability of a component and the damage range of the component based on the obtained defect size.

8. The image processing device according to claim 1, wherein: The code includes instructions for causing the at least one processor to perform the following operations: providing an interface to receive the image from a parts supplier supplying used parts, and providing the output image, defect type, and defect size stored in an image storage server to the parts supplier through the interface.

9. The image processing apparatus according to claim 1, wherein: The code includes instructions for causing the at least one processor to perform the following operations: based on the image analysis model being a U-Net neural network model, generate an image analysis model including multiple pooling layers and multiple non-pooling layers, and train the image analysis model using a loss function defined as the sum of losses between an input image and an output image.

10. The image processing device according to claim 9, wherein: The plurality of pooling layers are connected to the nonlinearity of a rectified linear unit function included in the image analysis model, and the image analysis model includes skip connections from the plurality of pooling layers to the plurality of non-pooling layers.

11. An image processing method, comprising: Based on detecting the component from the image surrounding the component, generating an input image by preprocessing the image; Applying the input image to an image analysis model to obtain defect types and defect sizes of components detected from the image; Applies a defect type and defect size to an input image to generate an output image.

12. The image processing method according to claim 11, wherein: Generating the input image involves: generating an input image by using a cropped image including a selected region including all portions of the detected part; Normalization is applied to pixel values ​​included in the cropped image according to the selected pixel interval to change the pixel values.

13. The image processing method according to claim 12, further comprising: Changing the channel of the cropped image whose pixel value is changed to the first channel or the second channel; Based on the channel of the cropped image whose pixel values ​​are changed being changed to a first channel including a plurality of channels, a plurality of pixel values ​​included in the cropped image whose pixel values ​​are changed are changed into a one-dimensional array in an order of channel features included in the first channel.

14. The image processing method according to claim 13, further comprising: splicing the image, the input image and the output image; The stitched images are sent to the image storage server.

15. The image processing method according to claim 11, wherein: Obtaining the defect type and defect size of the component detected from the image includes: Adjust one or any combination of a color feature of an input image, an edge feature of an input image, a polygon feature of an input image, a saturation feature of an input image, a color temperature feature of an input image, a clarity feature of an input image, a contrast feature of an input image, a blur feature of an input image, and a brightness feature of an input image; Apply the adjusted image to the image analysis model to obtain the defect type and defect size.

16. The image processing method according to claim 11, wherein: The defect types include: A first defect, which includes a surface scratch on the painted surface area of ​​the component; Second defects, which include deformation of parts; A third defect, wherein at least a portion of the shape of the component differs from an existing shape of the component; and The fourth defect includes gaps generated at the joints of parts. 17 . The image processing method according to claim 11 , further comprising calculating the availability of the component and the damage range of the component based on the obtained defect size.

18. The image processing method according to claim 11, further comprising: providing an interface to receive the image from a parts supplier supplying used parts; The output image, defect type and defect size stored in the image storage server are provided to the parts supplier through the interface.

19. The image processing method according to claim 11, further comprising: Based on the image analysis model being a U-Net neural network model, generating an image analysis model including a plurality of pooling layers and a plurality of non-pooling layers; training the image analysis model using a loss function defined as the sum of the losses between the input image and the output image, wherein the plurality of pooling layers are connected to the nonlinearity of the rectified linear unit function included in the image analysis model, The image analysis model includes skip connections from the plurality of pooling layers to the plurality of non-pooling layers.

20. An image storage server, comprising: receiving, from an image processing device, one or any combination of an image including a component of a vehicle, a defect type of the component of the vehicle, and a defect size of the component of the vehicle, wherein one or both of the defect type and the defect size are obtained by applying the image to an image analysis model; providing an interface for receiving a query for information about used parts from a parts supplier supplying used parts; Based on a query for information about used parts received through the interface, one or any combination of the defect type, the defect size, and the image is provided to the parts supplier.

Images