Method and system for performing defect classification of automobile assembly images
Through the residual neural network and feature correlation learning model, the defect area in the automotive assembly image is automatically paid attention to, and the feature discrimination is enhanced by using the comparative learning model, which solves the problem that defect areas are easily ignored and have high similarity in categories, and achieves efficient defect classification.
Patent Information
- Application Number
- CN202311390751.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-25
- Publication Date
- 2025-04-25
AI Technical Summary
In automotive assembly images, defect areas usually occupy a small proportion, resulting in easy neglect when classification, and the similarity between different defect categories is high and difficult to distinguish.
Residual neural network is used to extract image features, and the model based on feature correlation learning automatically focuses on defect areas. By comparing the learning model, the distance between similar features and the distance between different types of features is increased, thereby improving the discriminability of defect areas.
Effectively and automatically pay attention to and classify defect areas in car assembly images, improving the accuracy and efficiency of classification and avoiding the need for manual labeling.
Smart Images
Figure CN119888296A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of machine learning, and more particularly to a method, system, device, and medium for performing defect classification of automobile assembly images. Background Art
[0002] For automobile manufacturers, when conducting quality inspection on automobile assembly, for example, a camera can be used to take photos of parts or components (e.g., engines) of assembled automobiles. There may be some defects in the photos taken. By classifying these defects, the scale and cause of the defects can be obtained, which is conducive to targeted prevention and process improvement. By collecting images with defects and then training classification models, defect categories can be distinguished. By automatically classifying assembly defects, the cause of the defect can be found in time, which helps to improve maintenance efficiency and thus improve work efficiency.
[0003] However, the defective areas in the car assembly images taken by the camera often only occupy a small part of the entire image, and the normal areas in the image that are not related to the defects may cause great interference to the classification. In addition, since the defective areas in the image are very small relative to the entire image, these areas are easily ignored during classification, resulting in the defects in these areas being missed.
[0004] There are some methods in the prior art that manually mark the regions containing defects by adding bounding boxes to these regions in the image, and then focus on the marked regions to perform feature extraction.
[0005] In addition, for automobile assembly images, the similarity between different defect categories is high, which poses great difficulties for the classification model. For example, the image features of different engine defects are very similar, and existing work does not consider the discriminability between such features. Summary of the invention
[0006] The present disclosure aims to provide a method, system, device and medium for implementing a method for performing defect classification of automobile assembly images.
[0007] According to some embodiments of the present disclosure, a method for performing defect classification of an automobile assembly image is provided, comprising: receiving image data of an automobile assembly image containing defects; extracting a first feature from the image data using a residual neural network; processing the first feature using a model based on feature association learning to generate a second feature, wherein the second feature causes an area containing defects in the automobile assembly image to be automatically focused on; inputting a corresponding plurality of second features generated for a plurality of the image data into a model based on contrastive learning for processing to generate a corresponding plurality of third features; and inputting the plurality of third features into a fully connected network respectively to classify the defects contained in the automobile assembly image.
[0008] According to some embodiments of the present disclosure, the region containing defects in the automobile assembly image occupies a small portion of the entire automobile assembly image.
[0009] According to some embodiments of the present disclosure, the residual neural network includes one of the following: Resnet-101, Resnet-18, Resnet-50, and VGGNet.
[0010] According to some embodiments of the present disclosure, the automobile assembly image includes local details of the automobile but does not include any marked area.
[0011] According to some embodiments of the present disclosure, the model based on feature association learning and the model based on contrastive learning are trained using a set of labeled automobile assembly images containing defects.
[0012] According to some embodiments of the present disclosure, processing the first feature using a model based on feature association learning includes: respectively comparing the matrix representing the first feature with the query matrix W q , key matrix W k Sum matrix W v Multiply to obtain expressions Q, K, and V of the input information in the query space, key space, and value space, respectively; obtain a feature correlation matrix based on Q and K; and obtain the second feature based on V and the feature correlation matrix.
[0013] According to some embodiments of the present disclosure, the query matrix W q , key matrix W k Sum matrix W v It is learned by training the model based on feature association learning.
[0014] According to some embodiments of the present disclosure, the focusing includes increasing the weight of a region containing defects in the automobile assembly image.
[0015] According to some embodiments of the present disclosure, compared with the plurality of second features, the plurality of third features output by the contrastive learning-based model reduce the distance between features belonging to the same class and increase the distance between features belonging to different classes.
[0016] According to some embodiments of the present disclosure, the following contrastive loss function is defined in the training phase of the contrastive learning-based model:
[0017]
[0018] In the training phase, image data as training data are input in batches, I is a set of second features of a batch of image data, and F′ i is a second feature in the set, P i is the same as F′ i The feature set of samples belonging to the same class, F′ j is the set P i A sample feature of
[0019]
[0020] Where e is a natural constant, exp represents the exponent of e, T i is the same as F′ i The feature set of samples that do not belong to the same class, F′ k is the set T i is a sample characteristic in , and τ is the temperature coefficient.
[0021] According to some embodiments of the present disclosure, in the training phase, the learning of the contrastive learning based model is performed in a manner that minimizes the contrastive loss function.
[0022] According to some embodiments of the present disclosure, a computer system is provided, comprising: one or more processors, and a memory coupled to the one or more processors, the memory storing computer-readable program instructions, which when executed by the one or more processors perform the method described above.
[0023] According to some embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer-readable program instructions are stored. When the instructions are executed by the processor, the method described above is performed.
[0024] According to some embodiments of the present disclosure, there is provided an apparatus for performing defect classification of automobile assembly images, comprising means for implementing the steps of the method described above. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.
[0026] Figure 1 A block diagram of an exemplary computer system / server 12 suitable for use in implementing embodiments of the present invention is shown.
[0027] Figure 2 A set of example images of an assembled automobile engine taken by a camera are shown.
[0028] Figure 3 Another set of example images of an assembled automobile engine taken by a camera is shown.
[0029] Figure 4 A flow chart of a method for performing defect classification of automobile assembly images according to an embodiment of the present invention is shown.
[0030] Figure 5 is a schematic diagram showing a process of processing a first feature using a model based on feature association learning according to an embodiment of the present disclosure.
[0031] Figure 6 is a schematic diagram showing the processing effect of a model based on contrastive learning according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0032] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.
[0033] Figure 1 A block diagram of an exemplary computer system / server 12 suitable for use in implementing embodiments of the present invention is shown. Figure 1 The computer system / server 12 shown is merely an example and should not bring any limitation to the functionality and scope of use of embodiments of the present invention.
[0034] like Figure 1 As shown, computer system / server 12 is in the form of a general-purpose computing device. Components of computer system / server 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).
[0035] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media. Although Figure 1 Not shown, a magnetic disk drive and an optical disk drive may also be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the various embodiments of the present invention.
[0036] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.
[0037] The computer system / server 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the computer system / server 12, and / or any device that enables the computer system / server 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. In addition, the computer system / server 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer system / server 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer system / server 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0038] As mentioned above, in the case where the region containing the defect occupies a very small portion of the entire image, the region is easily overlooked during classification. Figure 2 A set of example images of an assembled automobile engine taken by a camera is shown. Figure 2 In the picture on the left side of , the area surrounded by the box indicated by reference numeral 21 is the area containing defects. Figure 2 The lower right picture is an enlarged display of the area 21. The cause of the defect is that the white plug cap was not removed in time after the test engine was completed. Figure 2 The upper right picture shows the situation when the plugging cap is removed, that is, a normal image without defects. Figure 2 As can be seen from the figure, region 21 occupies a very small part of the entire image and is easily overlooked during classification. However, if borders and annotation information are added manually, additional supervision information and manual annotation costs will be introduced, and the amount of calculation will be increased.
[0039] In addition, the inventors of the present invention have found that for automobile assembly images (especially engine assembly images), the similarity between different defect categories is relatively high, which creates great difficulties for the classification model. The prior art does not consider the discriminability between such features. Figure 3 Another set of example images of an assembled automobile engine taken by a camera is shown. Figure 3 Image A in shows an example of an oil cap that does not contain defects. That is, Figure 3 Image A in the figure shows a normal image with the assembly in place. Figure 3 Image B in the figure shows an example image of an engine with the oil cap installed upside down. Please note that the small oil pot symbol shown in the circle in Image B is upside down. Figure 3 Image C in the figure shows an example of an engine with its oil cap not tightened. Note that the oil bottle mark in Image C is facing forward. Figure 3 It can be seen that although image B and image C contain defects belonging to different categories (i.e., the oil cap is installed upside down and the oil cap is not tightened), their images are extremely similar and very difficult to distinguish.
[0040] Based on the above findings, the inventors of the present invention propose a method and system for performing defect classification of automobile assembly images, which is intended to overcome the above defects.
[0041] Now refer to Figure 4 , Figure 4 FIG. 4 is a flow chart of a method 400 for performing defect classification of an automobile assembly image according to an embodiment of the present invention. The method 400 is, for example, Figure 1 The computer system / server 12 executes the method 400 , which includes an image receiving step 410 , a feature extraction step 420 , a defect area location step 430 , a discriminability enhancement step 440 , and a classification step 450 .
[0042] The method 400 begins with an image receiving step 410. In the image receiving step 410, image data of an automobile assembly image containing defects is received. The method 400 is intended to classify the image data containing defects using a trained machine learning model. The received automobile assembly image is, for example, taken using a camera at an assembly station or an inspection station in an automobile manufacturing plant. According to one embodiment of the present disclosure, for example, Figure 2 As shown, the area containing defects in the automobile assembly image occupies a small part of the entire automobile assembly image. In one example according to the present invention, the small part occupies less than 50% of the entire image. In another example, the small part occupies less than 30% of the entire image. In another example, the small part occupies less than 10% or 20% or 5% of the entire image. In addition, the automobile assembly image includes local details of the automobile, but does not include any marked area. It should be noted that Figure 2 The box 21 in the figure is an enlarged image shown only for illustration purposes ( Figure 2 The lower right corner of the image) in the original image ( Figure 2 When used as input image data of method 400, the actual input image data does not contain any box or annotation information, but is only the original image taken by the camera.
[0043] According to an embodiment of the present disclosure, the automobile assembly image is a plurality of automobile assembly images. In one example, each of the plurality of automobile assembly images is an image obtained by photographing an engine on an assembly line.
[0044] Now return to reference Figure 4 , in the feature extraction step 420, a first feature is extracted from the image data using a residual neural network. In one example, the first feature may be a feature matrix; and in another example, the first feature may be an assembled tensor. According to one embodiment of the present disclosure, the residual neural network may include one of the following: Resnet-101, Resnet-18, Resnet-50, and VGGNet. In addition to the residual neural networks listed above, other residual neural networks that can be thought of by those skilled in the art are included in the scope of the present invention. In step 420, a first feature will be generated for each of the multiple automobile assembly images input in step 410.
[0045] In the defect area location step 430, the first feature is processed using a model based on feature association learning to generate a second feature. The second feature causes the area containing defects in the automobile assembly image to be automatically focused on. According to one embodiment of the present disclosure, gaining attention means increasing the weight of the area containing defects in the automobile assembly image. This attention causes the defect area in the automobile assembly image to be given more importance than other areas during classification, and reduces the interference of other areas (i.e., normal areas) in the automobile assembly image on classification.
[0046] Figure 5 FIG. 1 is a schematic diagram showing a process of processing a first feature using a model based on feature association learning according to an embodiment of the present disclosure. Figure 5 In , the first feature is shown as feature F. Figure 5 In the example of , there are multiple features F corresponding to multiple automobile assembly images respectively. Figure 5 The process in can be performed for each feature F to obtain the corresponding feature F', ie, the second feature.
[0047] According to an embodiment of the present disclosure, processing the first feature using a model based on feature association learning may include respectively comparing the matrix representing the first feature F with the query matrix W q , key matrix W k Sum matrix W v Multiply to obtain the expressions Q, K, and V of the input information in the query space, key space, and value space, respectively. Here, Q, K, and V are all matrices or tensors. Then, the feature correlation matrix can be obtained based on Q and K. In one example, the feature correlation matrix is obtained by the dot product calculation of Q and K. The feature correlation matrix can be used to reflect the different weights of different areas in the image. Finally, feature F' (i.e., the second feature) is obtained based on V and the feature correlation matrix. In one example, the second feature is obtained by the cross product calculation of V and the feature correlation matrix.
[0048] Figure 5 This is just an example of a model based on feature association learning for illustration purposes. Those skilled in the art may also conceive of other models, as long as the model can process the first feature so that the region containing defects in the automobile assembly image receives more attention.
[0049] According to one embodiment of the present disclosure, the model based on feature association learning is trained using a set of labeled automobile assembly images containing defects. According to one embodiment of the present disclosure, the query matrix W q , key matrix W k Sum matrix W v It is learned by training the model based on feature association learning.
[0050] Through the above processing of the model based on feature association learning, the obtained second feature can focus more on the area containing defects in the automobile assembly image. This ensures that the area containing defects will not be ignored even if it is small, and avoids the need for manual labeling of the input image data.
[0051] Now return to reference Figure 4 In the discriminability enhancement step 440, the corresponding plurality of second features generated for the plurality of image data are input into the contrastive learning-based model for processing to generate corresponding plurality of third features. According to one embodiment of the present disclosure, compared with the plurality of second features, the plurality of third features output by the contrastive learning-based model reduce the distance between features belonging to the same class and increase the distance between features belonging to different classes.
[0052] Figure 6 is a schematic diagram showing the processing effect of a model based on contrastive learning according to an embodiment of the present disclosure. Figure 6 In , different shapes represent samples of different categories, samples of the same shape belong to the same category, and arrows indicate the directions in which samples will move after being processed by the contrastive learning-based model. Figure 6 It can be seen that through the processing of the model based on contrastive learning, samples of the same type will be closer together and further away from other samples.
[0053] Various methods can be thought of to achieve the purpose of reducing the distance between features belonging to the same class and increasing the distance between features belonging to different classes. The present invention, for example, adopts contrastive learning. According to one embodiment of the present disclosure, the contrastive learning-based model can be obtained by training using a set of labeled automobile assembly images containing defects.
[0054] According to one embodiment of the present disclosure, the following contrastive loss function is defined in the training phase of the contrastive learning-based model:
[0055]
[0056] In the training phase, image data as training data are input in batches. In equation (1), I is a set of second features of a batch of image data, and F′ is i is a second feature in the set, P i is the same as F′ i The feature set of samples belonging to the same class, F′ j is the set P i In the above equation (1),
[0057]
[0058] In equation (2), e is a natural constant, exp represents the exponent of e, and T i is the same as F′ i The feature set of samples that do not belong to the same class, F′ k is the set T i is a sample characteristic in , and τ is the temperature coefficient.
[0059] The temperature coefficient τ is a preset constant, for example, a number between 0 and 1. The smaller the temperature coefficient τ is, the stronger the effect of contrastive learning is, that is, contrastive learning makes the distance between samples belonging to the same class closer and the distance between samples not belonging to the same class farther. If you want the sample features to be distributed, the temperature coefficient τ should be selected to be moderate.
[0060] According to an embodiment of the present disclosure, during the training phase of the contrastive learning model, the contrastive learning model is learned in a manner that minimizes the contrastive loss function. That is, the training goal of the contrastive learning model is to minimize the contrastive loss function.
[0061] Through the above processing of the contrastive learning-based model, the obtained third feature has higher discriminability, making it easier to distinguish similar features (e.g., Figure 3 ing the case of images B and C in the figure).
[0062] Now return to reference Figure 4 In the classification step 450, the plurality of third features are respectively input into a fully connected network to classify the defects contained in the automobile assembly image. For example, Figure 3 Image B and Image C in the figure are very similar, and the classification method in the prior art usually classifies them both as "the oil cap is not tightened". However, by adopting the method 400 of the present invention, Image B will be classified as "the oil cap is installed upside down" and Image C will be classified as "the oil cap is not tightened".
[0063] By performing classification based on the third feature finally obtained after processing in step 430 and step 440 instead of based on the first feature, the technical solution of the present invention can automatically focus on the smaller area containing defects in the automobile assembly image and avoid the interference of the non-defective area in the image. In addition, since the positioning of the defective area is automatically performed, the need to introduce additional supervision information and manual annotation is avoided. In addition, by adopting the technical solution of the present invention, through contrast learning, the inter-class distance is increased and the intra-class distance is reduced, thereby obtaining a feature with higher discriminability, so that the classification result is more accurate.
[0064] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.
[0065] According to one embodiment of the present invention, a computer system is provided, the computer system comprising one or more processors and a memory coupled to the one or more processors. The memory stores computer-readable program instructions, and the instructions, when executed by the one or more processors, perform: receiving image data of an automobile assembly image containing defects; extracting a first feature from the image data using a residual neural network; processing the first feature using a model based on feature association learning to generate a second feature, wherein the second feature automatically focuses on the area containing defects in the automobile assembly image; inputting the corresponding plurality of second features generated for the plurality of image data into a model based on contrastive learning for processing to generate corresponding plurality of third features; and inputting the plurality of third features into a fully connected network respectively to classify the defects contained in the automobile assembly image.
[0066] According to another embodiment of the present invention, a computer-readable storage medium is provided, on which computer-readable program instructions are stored, and when executed by a processor, the instructions perform: receiving image data of an automobile assembly image containing defects; extracting a first feature from the image data using a residual neural network; processing the first feature using a model based on feature association learning to generate a second feature, wherein the second feature causes an area containing defects in the automobile assembly image to be automatically focused on; inputting a corresponding plurality of second features generated for a plurality of the image data into a model based on contrastive learning for processing to generate a corresponding plurality of third features; and inputting the plurality of third features into a fully connected network respectively to classify the defects contained in the automobile assembly image.
[0067] According to another embodiment of the present invention, there is provided an apparatus for performing defect classification of an automobile assembly image, comprising: a device for receiving image data of an automobile assembly image containing defects; a device for extracting a first feature from the image data using a residual neural network; a device for processing the first feature using a model based on feature association learning to generate a second feature, wherein the second feature causes an area containing defects in the automobile assembly image to be automatically focused on; a device for inputting a corresponding plurality of second features generated for a plurality of the image data into a model based on contrastive learning for processing to generate a corresponding plurality of third features; and a device for inputting the plurality of third features respectively into a fully connected network to classify the defects contained in the automobile assembly image.
[0068] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.
[0069] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0070] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.
[0071] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.
[0072] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0073] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0074] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.
[0075] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A method for performing defect classification of an automobile assembly image, comprising: receiving image data of an automobile assembly image including a defect; Extracting a first feature from the image data using a residual neural network; Processing the first feature using a model based on feature association learning to generate a second feature, wherein the second feature causes an area containing defects in the automobile assembly image to be automatically focused on; Inputting the corresponding plurality of second features generated for the plurality of image data into a model based on contrastive learning for processing to generate corresponding plurality of third features; as well as The plurality of third features are respectively input into a fully connected network to classify defects contained in the automobile assembly image. 2 . The method according to claim 1 , wherein the region containing defects in the automobile assembly image occupies a small portion of the entire automobile assembly image.
3. The method according to claim 1, wherein the residual neural network comprises one of the following: Resnet-101, Resnet-18, Resnet-50, and VGGNet. 4 . The method of claim 1 , wherein the vehicle assembly image includes local details of the vehicle but does not include any marked areas. 5 . The method according to claim 1 , wherein the model based on feature association learning and the model based on contrastive learning are trained using a collection of labeled automobile assembly images containing defects.
6. The method according to claim 1, wherein processing the first feature using a model based on feature association learning comprises: The matrix representing the first feature is respectively combined with the query matrix W q , key matrix W k Sum matrix W v Multiply to obtain the expressions Q, K, and V of the input information in the query space, key space, and value space respectively; Obtain feature correlation matrix based on Q and K; as well as The second feature is obtained based on V and the feature correlation matrix.
7. The method according to claim 6, wherein the query matrix W q , key matrix W k Sum matrix W v It is learned by training the model based on feature association learning. 8 . The method of claim 2 , wherein the focusing includes increasing the weight of regions of the automotive assembly image that contain defects.
9. The method according to claim 1, wherein: Compared to the plurality of second features, the plurality of third features output by the contrastive learning-based model reduce the distance between features belonging to the same class and increase the distance between features belonging to different classes.
10. The method according to claim 1, wherein: The following contrastive loss function is defined during the training phase of the contrastive learning based model: In the training phase, image data as training data are input in batches, I is a set of second features of a batch of image data, and F′ i is a second feature in the set, P i is the same as F′ i The feature set of samples belonging to the same class, F′ j is the set P i A sample feature of Where e is a natural constant, exp represents the exponent of e, T i is the same as F′ i The feature set of samples that do not belong to the same class, F′ k is the set T i is a sample characteristic in , and τ is the temperature coefficient.
11. The method according to claim 10, wherein: During the training phase, the contrastive learning-based model is learned in a manner that minimizes the contrastive loss function.
12. A computer system comprising: one or more processors; and A memory coupled to the one or more processors, the memory storing computer-readable program instructions, which, when executed by the one or more processors, perform the method of any one of claims 1-11.
13. A computer-readable storage medium storing computer-readable program instructions, wherein the instructions, when executed by a processor, perform the method according to any one of claims 1 to 11.
14. An apparatus for performing defect classification of automotive assembly images, comprising means for implementing the steps of the method as claimed in any one of claims 1 to 11.