Automobile appearance defect detection method, device and storage medium

By combining multi-angle photography with defect description text, global and local features are extracted and spatiotemporally aligned, solving the accuracy problem of new energy vehicle appearance defect detection and improving detection precision and efficiency.

CN118657787BActive Publication Date: 2024-11-12CHENGDU CELIS TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411154985.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-12
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

Existing technologies are insufficient to comprehensively and accurately detect exterior defects in new energy vehicles, especially when shooting from multiple angles, it is difficult to capture defects on the vehicle's surface.

Method used

By capturing multiple images from multiple angles and combining them with defect description text, features are extracted using a multi-view image encoder and a text encoder to achieve spatiotemporal alignment of global and local features and predict defect types.

Benefits of technology

It improves the accuracy and efficiency of defect detection, enhances the characteristic description of the same defect, and improves the accuracy of defect classification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118657787B_ABST
    Figure CN118657787B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of appearance inspection technology, and in particular, to a method, device and storage medium for automobile appearance defect detection. The method comprises photographing a target vehicle surface area to be inspected to obtain multiple images at different shooting times and different shooting angles; obtaining a defect description text of the target vehicle surface to be inspected; extracting features of the multiple images by a multi-view image encoder to obtain global image features and local image features, extracting features of the defect description text by a text encoder to obtain global text features and local text features, and predicting the defect type according to the global image features and the global text features; the present invention effectively mines and aligns the spatiotemporal features of the defect, so that the local text features are aligned with the local image features in time and space, and the global text features are aligned with the global image features in time and space, so as to accurately detect the defect category.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of appearance inspection, and in particular to a method, device and storage medium for automobile appearance defect inspection. Background Art

[0002] The production and processing of new energy vehicles involves the positioning of fixtures, welding of parts, manual assembly and debugging, etc. These processes all involve contact with the exterior surface of parts, which can easily cause surface defects of the vehicle body. Common surface defects include pits, lumps, scratches, deformations, glue residues and welding residues, etc. Therefore, it is necessary to detect vehicle appearance defects in a timely manner and take corresponding control measures.

[0003] Currently, it is necessary to shoot the vehicle's exterior surface at a certain angle and identify appearance defects from the captured image. This method is limited to the shooting angle and it is difficult to accurately and comprehensively capture appearance defects.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The purpose of the present invention is to provide a method, device and storage medium for detecting automobile appearance defects, which can effectively mine and align the temporal and spatial characteristics of defects and accurately detect the defect category.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for detecting automobile appearance defects, comprising:

[0008] By shooting the area to be detected on the surface of the target vehicle at multiple angles, multiple images at different shooting times and different shooting angles are obtained;

[0009] Obtaining a defect description text of the area to be inspected; the defect description text includes the time when the defect occurred and the shooting angle;

[0010] Extracting features from the multiple images using a multi-view image encoder to obtain global image features and local image features, and extracting features from the defect description text using a text encoder to obtain global text features and local text features;

[0011] Predicting the defect type of the area to be detected according to the global image features and the global text features;

[0012] The local text features are aligned with the local image features in time and space, and the global text features are aligned with the global image features in time and space.

[0013] In a second aspect, the present invention provides an electronic device, comprising:

[0014] at least one processor, and a memory communicatively coupled to at least one of the processors;

[0015] Wherein, the memory stores instructions that can be executed by at least one of the processors, and the instructions are executed by at least one of the processors so that at least one of the processors can execute any of the methods for detecting automobile appearance defects.

[0016] In a third aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to enable the computer to execute any of the above-mentioned methods for detecting vehicle appearance defects.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] (1) The present invention uses multiple images taken from different shooting angles to comprehensively judge defects. Since defect images taken from different angles can effectively reflect the spatial characteristics of defects, integrating defect features from different angles can effectively improve the accuracy of defect detection and the efficiency of control.

[0019] (2) In addition to using multi-view images, the present invention also combines defect description text, thereby combining text with images to effectively mine the temporal and spatial characteristics of defects and accurately detect defect categories.

[0020] (3) In the present application, the local text features are aligned with the local image features in time and space, and the global text features are aligned with the global image features in time and space, thereby bringing the visual and text features of the same defect closer and moving away from the visual and text features of different defects, thereby enhancing the feature description of the same defect and facilitating improving the accuracy of defect classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a flow chart of a method for detecting automobile appearance defects provided by an embodiment of the present invention;

[0023] Figure 2 is a schematic diagram of the structure of the prediction model provided by an embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of feature alignment provided by an embodiment of the present invention;

[0025] Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0027] Example 1

[0028] Figure 1 1 is a flow chart of a method for detecting defects in vehicle appearance provided by this embodiment. This embodiment is applicable to the scenario of detecting defects on the vehicle surface. The method can be executed by an electronic device.

[0029] like Figure 1 As shown, this embodiment provides a method for detecting automobile appearance defects, comprising the following steps:

[0030] S110 , obtaining multiple images at different shooting times and different shooting angles by shooting the to-be-detected area on the surface of the target vehicle at multiple angles.

[0031] The target vehicle may be a vehicle in the process of production and manufacturing, and the vehicle surface may have defects such as dents, scratches, and deformations due to external factors. The present invention divides the vehicle surface into different areas to be detected, for example, a size of 20 cm × 20 cm is divided into an area to be detected. Generally, there is one defect in one area to be detected. The present invention aims to detect the defect type in the area to be detected.

[0032] Cameras are set up around the target vehicle to capture images of the area to be detected. The same area to be detected can be captured by cameras at different locations, or one camera can capture the same area to be detected at different angles, thereby obtaining multiple images from different shooting angles.

[0033] In each link of vehicle production and manufacturing, images of the area to be inspected can be tracked and captured, thereby forming multiple images at different shooting times and different shooting angles.

[0034] S120, obtaining a defect description text of the area to be inspected; the defect description text includes the time when the defect occurs and the shooting angle.

[0035] During the vehicle manufacturing process, R&D personnel will also check the defects of the vehicle appearance, such as watching the images taken by the camera, confirming the time and shooting angle of the defect in a certain area to be inspected, and recording it in the quality report. However, due to the large number of images, it takes a lot of labor costs to identify defects by naked eyes. Therefore, it is only necessary to record the time of defect occurrence (i.e. the camera shooting time) and shooting angle (i.e. the camera shooting angle) according to the camera parameters, and there is no need to identify the specific defect type.

[0036] The present invention reads the defect description text from the quality report, that is, the time when the defect occurs and the shooting angle.

[0037] After S110 and S120, we can get:

[0038] ; (1)

[0039] in, It means from t=1 to t=t max The input sequence corresponding to the i-th defect type in the time period (i.e., the inventor assumes that there is one main defect type in a region to be inspected), It represents the image of the defect taken from angles 1, 2, 3, and 4 at time t, and the maximum value of t is t max . Note that the 4 here are just an enumeration, and there may actually be 5, 6, or other shooting angles. Represents the defect description text of defect i at time t.

[0040] S130, extracting features from the multiple images using a multi-view image encoder to obtain global image features and local image features, and extracting features from the defect description text using a text encoder to obtain global text features and local text features.

[0041] For each input sequence , Multi-view Image Encoder It is used to extract features from each image obtained in S110 and is expressed by formula (2).

[0042] ; (2)

[0043] in, For the input sequence The global image features at time t, For the input sequence The local image features at time t, g is a superscript, representing the global features, is the length of the local image feature, that is, the number of local image feature elements, and d is the dimension of the feature. Represents the input sequence The first image in Represents the input sequence The last image in Represents the input sequence The local image features are obtained by splitting the global image features, that is, the local image features are spliced ​​to form the global image features. The multi-view image encoder in this embodiment The input sequence The multi-view images at the same time t are fused, so the global image features and local image features no longer reflect the characteristics of a single view.

[0044] Text Encoder It is used to extract features from the defect description text obtained in S120 and is expressed by formula (3).

[0045] ; (3)

[0046] in, Represents the input sequence The global text features at time t, Represents the input sequence Where W is the length of the local text feature, that is, the number of local text feature elements, and d is the dimension of the feature, which is the same as the dimension of the global image feature and the local image feature.

[0047] Referring to formula (2) and formula (3), this embodiment performs one-time calculation on an input sequence To predict the defect type, the variable i can be omitted and the feature obtained by S130 can be simplified as ;in, is the global image feature at time t, is the local image feature at time t, is the global text feature at time t, is the local text feature at time t.

[0048] This embodiment pre-trains the multi-view image encoder and text encoder so that the local text features and local image features output by them are aligned in time and space, and the global text features and global image features are aligned in time and space. The global features aligned in time and space (the local features within the global features are also aligned) can enhance the features of the defects, which is conducive to improving the accuracy of defect type prediction.

[0049] S140 , predicting the defect type of the area to be detected according to the global image features and the global text features.

[0050] The global image features and global text features are input into the classification layer ( softmax Layers, including normalized exponential functions) and fully connected layers, predict the defect type represented by one-hot encoding.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] (1) The present invention uses multiple images taken from different shooting angles to comprehensively judge defects. Since defect images taken from different angles can effectively reflect the spatial characteristics of defects, integrating defect features from different angles can effectively improve the accuracy of defect detection and the efficiency of control.

[0053] (2) In addition to using multi-view images, the present invention also combines defect description text, thereby combining text with images to effectively mine the temporal and spatial characteristics of defects and accurately detect defect categories.

[0054] (3) In the present application, the local text features are aligned with the local image features in time and space, and the global text features are aligned with the global image features in time and space, thereby bringing the visual and text features of the same defect closer and moving away from the visual and text features of different defects, thereby enhancing the feature description of the same defect and facilitating improving the accuracy of defect classification.

[0055] Example 2

[0056] Based on the above embodiments, this embodiment refines the structures of the multi-view image encoder and the text encoder.

[0057] Figure 2 1 is a schematic diagram of a prediction model provided by an embodiment of the present invention, wherein the prediction model includes a multi-view image encoder, a text encoder, a classification layer, and a fully connected layer. Each of the multiple images is input into the multi-view image encoder to obtain global image features and local image features.

[0058] The present invention designs a novel hybrid expert model as a multi-view image encoder Mining defect features. Specifically, the multi-view image encoder includes a fully connected layer, a multi-head attention layer, and multiple serially connected mixed expert layers.

[0059] The fully connected layer is used to divide each image into multiple image blocks I, and flatten and connect the feature vectors of multiple image blocks to obtain the fully connected feature Among them, the feature vector of each image block is flattened to obtain a feature vector set with dimension Y , u is the number of eigenvectors, is the number of feature vectors of an image block. Then, add the learnable embedding vector to the flattened feature vector , introduce the position encoding vector , thereby retaining the position information of each eigenvector in the original image. The set of eigenvectors obtained after flattening Connect to form fully connected features , the process can be expressed as:

[0060] ; (4)

[0061] in, Indicates a connection operation. represents the flattened feature vector of the first image block, represents the feature vector of the last image block after flattening. Formula (4) indicates that the feature vectors of all image blocks after flattening need to be connected. , we can get the fully connected features of each image.

[0062] In order to enable the prediction model to capture defect features from different perspectives, the present invention adds a novel hybrid expert layer to the VIT (Vision Transformer) structure. The multi-head attention layer is used to use a multi-head self-attention mechanism for the fully connected features of multiple images at different shooting perspectives to obtain attention features. The multi-head attention layer can handle problems with multiple points of interest and can better handle the complex relationship between the semantics of multi-perspective images. Each of the hybrid expert layers is used to input the feature groups of the previous layer into the corresponding experts, and fuse the output features of multiple experts to obtain global image features. Specifically, the number of hybrid expert layers is multiple and they are connected in series. Figure 2 Nn hybrid expert layers are shown as an example. The structure of each hybrid expert layer is marked with a dotted box, including: two forward feedback layers as two experts that capture complementary features, processing the forward The eigenvectors and feature vectors, a total of feature vectors. The outputs of the two experts are connected to get the output of the entire hybrid expert layer. The attention feature is input to the first hybrid expert layer to capture the spatial features, and then the output of the first hybrid expert layer is provided to the second hybrid expert layer, and so on. The output of the last hybrid expert layer is the global image feature. . Global image features After row / column splitting, local image features are obtained .

[0063] Input the defect description text into the EncoderLayer of the bidirectional Transformer model for feature extraction to obtain global text features , the global text feature After row / column splitting, local text features are obtained The BERT (Bidirectional Encoder Representations from Transformers) model uses the EncoderLayer of the bidirectional Transformer model for feature extraction (there is no Decoder part in the BERT model), which has better parallel performance and shorter training time, and is conducive to improving the efficiency of text feature extraction.

[0064] The global image features and global text features are input into the classification layer to obtain the fused features, and the fused features are input into the fully connected layer for linear mapping to obtain the defect type represented by the one-hot encoding of the area to be detected.

[0065] This embodiment provides a novel hybrid expert-based multi-view image encoder that can accurately capture image features from different viewpoints. By fusing image and text features through the entire prediction model, multi-modal feature fusion is achieved, thereby improving the accuracy of defect type prediction.

[0066] Example 3

[0067] Based on the above embodiment, this embodiment limits the training process of the multi-view image encoder and the text encoder so that the global image features and local features output by the two can be aligned in time and space. Figure 3 The parameters of the classification layer and the fully connected layer can be fixed parameters, or can be trained together with the multi-view image encoder and the text encoder, which is not limited in this embodiment. The training process is as follows:

[0068] Step 1: Obtain training samples for the prediction model.

[0069] The training samples include multiple image samples obtained at different shooting times and different shooting angles by shooting various defect types on the surface of the sample vehicle at multiple angles, and defect description text samples of various defect types. The sample vehicle is an area with known defect types and a vehicle with defect types. The defect type area here is divided in the same way as the area to be detected in S110.

[0070] Assume that the training sample set is represented as , Indicates the number of defect types, .

[0071] Step 2: Input the training samples into the prediction model to obtain the global image features and local image features output by the multi-view image encoder, the global text features and local text features output by the text encoder, and the defect type output by the fully connected layer.

[0072] It should be noted that local image features and local text features are only used during model training and will not be input into the classification layer.

[0073] Step 3: Optimize the parameters in the multi-view image encoder and the text encoder by minimizing the spatiotemporal distance between the global image features and the global text features, the spatiotemporal distance between the local image features and the local text features, and the distance between the defect type and the true type.

[0074] When using training samples to train the prediction model, this embodiment simultaneously constrains the temporal and spatial distance between the text and the image, as well as the distance between the defect type and the true type, to optimize the parameters in the multi-view image encoder and the text encoder. Optionally, the parameters in the classification layer and the fully connected layer can also be optimized at the same time. Specifically, the third step includes S310~S350.

[0075] S310: construct a global spatial loss function according to the spatial distance between the global image feature and the global text feature.

[0076] Specifically, a contrastive learning algorithm is used to calculate the first spatial distance between the global image features and the global text features in the region of each defect type and at each moment; and a global spatial loss function is constructed according to the first spatial distance.

[0077] Contrastive learning is used to bring the visual and textual features of the same defect closer together and move away from the visual and textual features of different defects. The process can be expressed as:

[0078] ; (5)

[0079] ; (6)

[0080] in, represents the cosine similarity of two feature vectors, represents the scaling factor, represents the first spatial distance of the global text feature relative to all global image features at each moment, It represents the first spatial distance between the global image features at each moment and all global text features, and k is the number of the moment.

[0081] When the input samples include multiple defect types, the global space loss function is obtained by combining the number of defect types D of all input samples: :

[0082] ; (7)

[0083] S320: construct a local spatial loss function according to the spatial distance between the local image feature and the local text feature.

[0084] Considering that most defects exist in specific locations in images and text descriptions, the present invention targets local text features. and local image features , design a novel multi-modal alignment method to achieve a spatial simulation effect. For the convenience of description, The jth local text feature element in is expressed as , The jth local image feature element in is expressed as .

[0085] First, for the local image features and local text features of the same defect type area at the same time, each local text feature element in the local text feature Are mapped to local image features On the top, get each visual text feature element Specifically, in this embodiment, softmax The function is applied to all cosine similarities On, get , thereby obtaining the corresponding visual text representation , is the local image feature Similarly, according to Each local image feature element in the local image feature Are mapped to local text features On the top, get each text visual feature element , is the local text feature The transpose of .

[0086] Then, referring to formula (8) and formula (9), the contrastive learning algorithm is used to calculate each local text feature element Corresponding visual text feature elements The second spatial distance.

[0087] ; (8)

[0088] ; (9)

[0089] Wherein, c is a serial number, and W is the number of local text feature elements or the number of visual text feature elements. represents the scaling factor, Represents the second spatial distance of the visual text feature element at each moment relative to all local text feature elements; Represents the second spatial distance of the local text feature element relative to all visual text feature elements at each moment, represents the local text feature element when j=c, Represents the visual text feature element when j=c.

[0090] Similarly, referring to formula (10) and formula (11), the contrastive learning algorithm is used to calculate each local image feature element Corresponding text visual feature elements The third space distance.

[0091] ; (10)

[0092] ; (11)

[0093] in, Indicates the third space distance of the text visual feature elements relative to all local image feature elements at each moment; Indicates the third space distance between the local image feature elements at each moment and all text visual feature elements, represents the local image feature element when j=c, represents the text visual feature element when j=c, and M is the number of local image feature elements or the number of text visual feature elements.

[0094] Next, each local text feature element Corresponding visual text feature elements Fusion is performed by multiplying the corresponding elements to achieve multimodal feature fusion. The process can be expressed as formula (12). Generate weights for each second spatial distance , see formula (13).

[0095] ; (12)

[0096] ; (13)

[0097] in, Capture the relationships and importance of different dimensions and form a matrix ,in, , W is a natural number greater than 1. , Represents the fused features The average value of and represents the learnable matrix, and d represents the dimensions of the matrix Q and the matrix K.

[0098] Similarly, each local image feature element Corresponding text visual feature elements The fusion process can be expressed as formula (14). Generate weights for each third space distance , see formula (15).

[0099] ; (14)

[0100] ; (15)

[0101] in, ,in, , M is a natural number greater than 1. , Represents the fused features The average value of and represents a learnable matrix, and d represents a matrix With the matrix Dimension.

[0102] At the end of step S320, a local space loss function is constructed according to each second space distance and the corresponding weight, and each third space distance and the corresponding weight, see formulas (16~18).

[0103] ; (16)

[0104] ; (17)

[0105] ; (18)

[0106] in, L TLA is the local alignment loss of image features, L VTA is the local alignment loss of text features, L local is the local space loss function.

[0107] S330 , constructing a global time loss function according to the time distance between the global image feature and the global text feature, and the distance between the defect type and the true type.

[0108] Taking into account the existence of temporal features in the original data, this model uses temporal features through a novel bidirectional cyclic alignment constraint to further shorten the distance between the global feature vectors of the same defective text image in the high-dimensional space and increase the distance between the global feature vectors of different defective text images.

[0109] For an input sequence over a period of time , Multi-view Image Encoder and text encoder Mining features from it, we get two feature sequences: global image feature sequence ,Global text feature sequence , then and A bidirectional recurrent alignment constraint is implemented to achieve temporal alignment of global image features with global text features.

[0110] In the bidirectional loop alignment constraint, the two-way includes forward propagation and backward propagation. In forward propagation and backward propagation, the text vector space is mapped to the image vector space for feature alignment, and the image vector space is reversely mapped to the text vector space for feature alignment, thus achieving bidirectional alignment. Through this mapping, two feature sequences can be perceived and The temporal feature differences of the same defect type can be used to better achieve cross-modal matching. and global text feature sequence , according to the global image feature sequence Each global image feature With global text feature sequence The first distance , calculated theoretically with each global image feature The first target vector closest to , see formula (19).

[0111] ; (19)

[0112] in, is the global text feature at t=z, is the global text feature when t=k. Note that t, k and z are used only as variables.

[0113] Similarly, according to the global text feature sequence Each global text feature in With the global image feature sequence The second distance , calculated with each global text feature The closest second target vector .

[0114] ; (20)

[0115] in, is the global image feature when t=z. Note that t, k and z are only used as variables.

[0116] Then, let the first target vector With the global image feature sequence The similarity distribution of tends to Gaussian distribution, and the first back propagation loss function is constructed. Specifically, in the back propagation process, the calculation and The similarity distribution of all feature vectors in :

[0117] ;(twenty one)

[0118] in, It is the global image feature when t=k, and k is only used as a variable.

[0119] Since the distance from time t is greater, and The smaller the similarity of all feature vectors in the Approximate Gaussian distribution. In order to make the similarity distribution Approaching Gaussian distribution, so in the back propagation, Apply a Gaussian prior probability and then maximize , thus ensuring that the peak of the similarity distribution appears at time t, that is, The value is close to 0. Then introduce the regularization term To reduce the impact of abnormal variables on the optimization results, the first back propagation loss function in the bidirectional loop alignment constraint is obtained :

[0120] ;(twenty two)

[0121] in, , , z is a variable, represents the regularization weight, and All represent hyperparameters.

[0122] Similarly, let the second target vector With global text feature sequence The similarity distribution of tends to Gaussian distribution. Specifically, in the back propagation process, the calculation and The similarity distribution of all feature vectors in , thereby constructing the second back propagation loss function .

[0123] ;(twenty three)

[0124] ;(twenty four)

[0125] in, It is the global text feature when t=k, and k is only used as a variable. , , z is a variable, represents the regularization weight, and All represent hyperparameters.

[0126] Next, in order to improve the prediction model’s ability to capture temporal features, it is necessary to introduce a classification loss function in the forward propagation and apply one-hot encoding to the prediction results, so that the global image features can be captured. With global text features Specifically, according to the distance between the defect type and the true type in the area of ​​the same defect type, a forward classification loss function (including the first forward classification loss function and the second forward classification loss function) is constructed. Specifically, according to the aforementioned first distance Get the first forward classification loss function .

[0127] ; (25)

[0128] The fully connected layers in this embodiment are two parallel fully connected layers, so that the prediction model can output two parallel vectors. The length of the vector is the total number of defect types, and the elements in the vector are the probability values ​​of the defect types. In formula (25), , is the probability value of the defect type output by the prediction model, Q l Represents the operation of the first fully connected layer. It means that the prediction result is marked with a one-hot encoding vector, which is the label of the training sample, indicating the defect type, and z is a variable. is the middle value. Therefore, for the sequence S, the first forward classification loss function in the bidirectional cyclic alignment constraint is It can be expressed as:

[0129] ; (26)

[0130] Similarly, according to the aforementioned second distance Get the second forward classification loss function .

[0131] ; (27)

[0132] in, , which is also the probability value of the defect type output by the prediction model, Represents the operation of the second fully connected layer. It means that the prediction result is marked with a one-hot encoding vector, which is the label of the training sample, indicating the defect type, and z is a variable. is the middle value. Therefore, the second forward classification loss function in the bidirectional cycle alignment constraint is It can be expressed as:

[0133] ; (28)

[0134] Finally, a global time loss function is constructed based on the first back propagation loss function, the second back propagation loss function, the first forward classification loss function and the second forward classification loss function, see formula (29).

[0135] ; (29)

[0136] S340. Construct a total loss function L according to the global spatial loss function, the local spatial loss function and the global temporal loss function, see formula (30).

[0137] ; (30)

[0138] in, and is a hyperparameter that controls the weight of the loss function.

[0139] S350, optimizing the parameters in the multi-view image encoder and the text encoder by minimizing the loss function.

[0140] The total loss function in this embodiment is composed of a global spatial loss function, a local spatial loss function, and a global temporal loss function. By continuously iterating the parameters in the multi-view image encoder and the text encoder, the total loss function L is minimized, that is, the distance between local text features and local image features in time is minimized, the distance between local text features and local image features in space is minimized, the distance between global text features and global image features in time is minimized, and the distance between global text features and global image features in space is minimized, thereby obtaining the optimal parameters in the multi-view image encoder and the text encoder. When the multi-view image encoder and the text encoder are used for feature extraction, the multi-view image encoder and the text encoder with the aforementioned optimal parameters can extract spatiotemporally aligned local text features and local image features, as well as spatiotemporally aligned global text features and global image features from the input image and text.

[0141] Furthermore, when the fully connected layer and the classification layer have parameters to be learned, they can also be iterated synchronously with the parameters in the multi-view image encoder and the text encoder.

[0142] This embodiment provides a novel forward-backward loss function calculation method based on the bidirectional cyclic alignment constraint, which realizes the spatiotemporal alignment of global image features and global text features, and the spatiotemporal alignment of local image features and local text features.

[0143] Example 4

[0144] This embodiment provides an electronic device, see Figure 4 , comprising at least one processor 301, and a memory 302 communicatively connected to at least one of the processors 301;

[0145] The memory 302 stores instructions that can be executed by at least one of the processors 301. The instructions are executed by at least one of the processors 301 so that at least one of the processors 301 can execute the above-mentioned automobile appearance defect detection method, thereby having at least the same advantages as the above-mentioned method.

[0146] Optionally, the electronic device also includes interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on a memory to display graphical information of a GUI (Graphical User Interface) on an external input / output device (such as a display device coupled to an interface). In other embodiments, if necessary, multiple processors can be used together with multiple memories, and / or multiple buses can be used together with multiple memories. Similarly, multiple electronic devices can be connected (for example, as a server array, a group of blade servers, or a multi-processor system), and each device provides some necessary operations.

[0147] The memory 302 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the automobile appearance defect detection method in the embodiment of the present invention. The processor 301 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 301, that is, realizing the above-mentioned automobile appearance defect detection method.

[0148] The memory 301 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 302 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0149] The electronic device may further include: an input device 303 and an output device 304. The processor 301, the memory 301, the input device 303 and the output device 304 may be connected via a bus or other means.

[0150] The input device 303 can receive input digital or character information, and the output device 304 can include a display device, an auxiliary lighting device (e.g., LED), and a tactile feedback device (e.g., a vibration motor), etc. The display device can include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device can be a touch screen.

[0151] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0152] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting automobile appearance defects, characterized in that: include: By shooting the area to be detected on the surface of the target vehicle at multiple angles, multiple images at different shooting times and different shooting angles are obtained; Obtaining a defect description text of the area to be inspected; the defect description text includes the time when the defect occurred and the shooting angle; Extracting features from the multiple images using a multi-view image encoder to obtain global image features and local image features, and extracting features from the defect description text using a text encoder to obtain global text features and local text features; Predicting the defect type of the area to be detected according to the global image features and the global text features; Wherein, the local text feature is aligned with the local image feature in time and space, and the global text feature is aligned with the global image feature in time and space; When training the multi-view image encoder and the text encoder, a global temporal loss function is constructed according to the temporal distance between the global image features and the global text features, and the distance between the defect type and the true type; a bidirectional circular alignment constraint is performed on the global image features and the global text features to achieve temporal alignment of the global image features and the global text features.

2. The method according to claim 1, characterized in that Extracting features from the multiple images using a multi-view image encoder to obtain global image features and local image features includes: Inputting the multiple images into a multi-view image encoder to obtain global image features and local image features; Among them, the multi-view image encoder includes a fully connected layer, a multi-head attention layer and multiple serially connected mixed expert layers; the fully connected layer is used to divide each image into multiple image blocks, and flatten and connect the feature vectors of the multiple image blocks to obtain fully connected features; the multi-head attention layer is used to use a multi-head self-attention mechanism for the fully connected features of multiple images under different shooting perspectives to obtain attention features; each of the mixed expert layers is used to input the feature groups of the previous layer into the corresponding expert, and fuse the output features of multiple experts to obtain global image features.

3. The method according to claim 1, characterized in that: The defect description text is subjected to feature extraction by a text encoder to obtain global text features and local text features, including: The defect description text is input into the EncoderLayer of the bidirectional Transformer model for feature extraction to obtain global text features and local text features.

4. The method according to claim 1, characterized in that: The prediction model includes the multi-view image encoder, the text encoder, the classification layer and the fully connected layer; Predicting the defect type of the area to be detected according to the global image features and the global text features includes: Inputting the global image features and the global text features into the classification layer and the fully connected layer to obtain the defect type of the area to be detected; Before extracting features from the multiple images by a multi-view image encoder to obtain global image features and local image features, the method further includes: Acquire training samples of the prediction model; the training samples include multiple image samples obtained by shooting multiple defect types on the surface of the sample vehicle at different shooting times and different shooting angles, and defect description text samples of the defect types; Inputting the training samples into the prediction model, obtaining global image features and local image features output by the multi-view image encoder, global text features and local text features output by the text encoder, and defect types output by the fully connected layer; The parameters in the multi-view image encoder and the text encoder are optimized by minimizing the spatiotemporal distance between the global image features and the global text features, the spatiotemporal distance between the local image features and the local text features, and the distance between the defect type and the true type.

5. The method according to claim 4, characterized in that The method optimizes the parameters in the multi-view image encoder and the text encoder by minimizing the spatiotemporal distance between the global image features and the global text features, the spatiotemporal distance between the local image features and the local text features, and the distance between the defect type and the true type, including: According to the spatial distance between the global image features and the global text features, a global spatial loss function is constructed; According to the spatial distance between local image features and local text features, a local spatial loss function is constructed; Constructing a total loss function according to the global spatial loss function, the local spatial loss function and the global temporal loss function; By minimizing the loss function, the parameters in the multi-view image encoder and the text encoder are optimized.

6. The method according to claim 5, characterized in that The global spatial loss function is constructed according to the spatial distance between the global image feature and the global text feature, including: A contrastive learning algorithm is used to calculate the first spatial distance between the global image features and the global text features in the region of each defect type and at each moment; A global spatial loss function is constructed according to the first spatial distance.

7. The method according to claim 5, characterized in that The local spatial loss function is constructed according to the spatial distance between the local image feature and the local text feature, including: For the local image features and local text features of the same defect type area and at the same time, each local text feature element in the local text feature is mapped to the local image feature to obtain each visual text feature element; each local image feature element in the local image feature is mapped to the local text feature to obtain each text visual feature element; A contrastive learning algorithm is used to calculate the second spatial distance between each local text feature element and the corresponding visual text feature element; and a contrastive learning algorithm is used to calculate the third spatial distance between each local image feature element and the corresponding text visual feature element; Fusing each local text feature element with the corresponding visual text feature element, and generating the weight of each second spatial distance according to the fused features; and fusing each local image feature element with the corresponding text visual feature element, and generating the weight of each third spatial distance according to the fused features; A local spatial loss function is constructed based on each second spatial distance and the corresponding weight, and each third spatial distance and the corresponding weight.

8. The method according to claim 5, characterized in that The global time loss function is constructed according to the time distance between the global image feature and the global text feature, and the distance between the defect type and the true type, including: For a global image feature sequence and a global text feature sequence of an area of ​​the same defect type, a first target vector closest to each global image feature is calculated according to the distance between each global image feature in the global image feature sequence and the global text feature sequence; and a second target vector closest to each global text feature is calculated according to the distance between each global text feature in the global text feature sequence and the global image feature sequence; The similarity distribution between the first target vector and the global image feature sequence is made to tend to Gaussian distribution, and a first back propagation loss function is constructed; and the similarity distribution between the second target vector and the global text feature sequence is made to tend to Gaussian distribution, and a second back propagation loss function is constructed; Constructing a forward classification loss function according to the distance between the defect type and the true type in the area of ​​the same defect type; A global time loss function is constructed according to the first back propagation loss function, the second back propagation loss function and the forward classification loss function.

9. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to at least one of the processors; Wherein, the memory stores instructions that can be executed by at least one of the processors, and the instructions are executed by at least one of the processors so that at least one of the processors can execute the automobile appearance defect detection method described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The medium stores computer instructions, and the computer instructions are used to enable the computer to execute the automobile appearance defect detection method according to any one of claims 1 to 8.