Product defect detection method, apparatus, medium, and program product

Abstract

The application provides a product defect detection method, device, medium and program product, which can be applied to the technical field of computer vision. The method comprises: performing attention feature processing on an initial image of a product to be detected to obtain an attention feature of the product to be detected; encoding the initial image to obtain a plurality of encoding features with different levels; performing multiple feature reconstructions on the plurality of encoding features to obtain a target fusion feature meeting a target level; determining a defect detection result of the product to be detected according to the attention feature and the target fusion feature; and performing a feature reconstruction, comprising: decoding a fusion feature of a previous level to obtain a decoded fusion feature of a current level; and fusing a decoded feature of the current level, an encoding feature of the current level and the decoded fusion feature of the current level to obtain a fusion feature of the current level, wherein the decoded feature of the current level is obtained by decoding an encoding feature of a previous level or a decoded feature of the previous level.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a product defect detection method, device, medium, and program product. Background Art

[0002] In industrial production, product defect detection is crucial for quality control. For high-precision products like servers, even more rigorous and detailed monitoring is required to ensure the absence of defects. Currently, commonly used detection methods are primarily based on traditional image processing techniques, such as edge detection, threshold segmentation, and template matching. These methods are simple to implement and can be automated, significantly improving the efficiency of product defect detection.

[0003] In the process of implementing this application, it was found that there are at least the following problems in the relevant technology: traditional image processing technology usually only focuses on some image features used for single image processing, and therefore does not fully utilize image features for defect detection. It can only make rough judgments on the edges of the product, and the detection accuracy is not high, which makes it difficult to meet the high-precision detection requirements of server products. Summary of the Invention

[0004] In view of the above problems, the present application provides a product defect detection method, device, medium and program product.

[0005] According to the first aspect of the present application, a product defect detection method is provided, comprising: performing attention feature processing on an initial image of a product to be tested to obtain an attention feature of the product to be tested; encoding the initial image to obtain a plurality of coding features of different levels, with the feature scales of the same level being the same; performing multiple feature reconstructions on the plurality of coding features to obtain a target fusion feature that meets the target level; determining a defect detection result of the product to be tested based on the attention feature and the target fusion feature; wherein, performing a feature reconstruction once, comprising: decoding the fusion feature of the previous level to obtain a decoded fusion feature of the current level; fusing the decoded feature of the current level, the coding feature of the current level and the decoded fusion feature of the current level to obtain a fusion feature of the current level, wherein the decoding feature of the current level is obtained by decoding using the coding feature of the previous level or the decoding feature of the previous level.

[0006] The second aspect of the present application provides a product defect detection device, including: an attention feature extraction module, which is used to perform attention feature processing on the initial image of the product to be tested to obtain the attention feature of the product to be tested; an image encoding module, which is used to encode the initial image to obtain multiple encoding features of different levels, and the feature scales of the same level are the same; a feature reconstruction module, which is used to perform multiple feature reconstructions on multiple encoding features to obtain target fusion features that meet the target level; a defect detection module, which is used to determine the defect detection result of the product to be tested based on the attention feature and the target fusion feature; wherein the feature reconstruction module includes: a feature decoding submodule, which is used to decode the fusion feature of the previous level to obtain the decoding fusion feature of the current level; a feature fusion submodule, which is used to fuse the decoding feature of the current level, the encoding feature of the current level and the decoding fusion feature of the current level to obtain the fusion feature of the current level, and the decoding feature of the current level is obtained by decoding using the encoding feature of the previous level or the decoding feature of the previous level.

[0007] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0008] The fourth aspect of the present application further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0009] The fifth aspect of the present application further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above contents and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings.

[0011] Figure 1 A diagram illustrating an application scenario of a product defect detection method, device, medium, and program product according to an embodiment of the present application is shown.

[0012] Figure 2 A flow chart of a product defect detection method according to an embodiment of the present application is shown.

[0013] Figure 3 A data flow diagram for determining combined features according to a product defect detection method according to an embodiment of the present application is shown.

[0014] Figure 4A data flow diagram for determining attention features according to a product defect detection method according to an embodiment of the present application is shown.

[0015] Figure 5 A schematic diagram showing decoding according to the product defect detection method according to an embodiment of the present application is shown.

[0016] Figure 6 A model diagram of encoding and decoding an initial image according to a product defect detection method according to an embodiment of the present application is shown.

[0017] Figure 7 The figure shows a structural block diagram of a product defect detection device according to an embodiment of the present application.

[0018] Figure 8 A block diagram of an electronic device suitable for implementing a product defect detection method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0020] The terms used herein are only for describing specific embodiments and are not intended to limit the present application. The terms "comprise," "include," etc. used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0021] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0022] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0023] An embodiment of the present application provides a product defect detection method, including: performing attention feature processing on an initial image of a product to be tested to obtain an attention feature of the product to be tested; encoding the initial image to obtain multiple coding features of different levels, and the feature scales of the same level are the same; performing multiple feature reconstructions on the multiple coding features to obtain target fusion features that meet the target level; determining the defect detection results of the product to be tested based on the attention features and the target fusion features; wherein, performing a feature reconstruction, including: decoding the fusion features of the previous level to obtain the decoded fusion features of the current level; fusing the decoded features of the current level, the coding features of the current level and the decoded fusion features of the current level to obtain the fusion features of the current level, and the decoding features of the current level are obtained by decoding using the coding features of the previous level or the decoding features of the previous level.

[0024] Figure 1 A diagram illustrating an application scenario of a product defect detection method, device, medium, and program product according to an embodiment of the present application is shown.

[0025] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first image acquisition device 101, a second image acquisition device 102, a third image acquisition device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first image acquisition device 101, the second image acquisition device 102, the third image acquisition device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0026] The user can use the first image acquisition device 101, the second image acquisition device 102, and the third image acquisition device 103 to capture appearance images of the product to be tested, and interact with the server 105 through the network 104 to upload the captured appearance images to the server 105 for defect detection.

[0027] The first image acquisition device 101 , the second image acquisition device 102 , and the third image acquisition device 103 may be various electronic devices capable of performing image acquisition, including but not limited to smart phones, cameras, video cameras, and the like.

[0028] The server 105 may be a server that provides various services, such as a server that performs defect detection on appearance images of the product to be tested uploaded by the user using the first image acquisition device 101 , the second image acquisition device 102 , and the third image acquisition device 103 .

[0029] It should be noted that the product defect detection method provided in the embodiment of the present application can generally be executed by the server 105. Accordingly, the product defect detection device provided in the embodiment of the present application can generally be set in the server 105. The product defect detection method provided in the embodiment of the present application can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first image acquisition device 101, the second image acquisition device 102, the third image acquisition device 103 and / or the server 105. Accordingly, the product defect detection device provided in the embodiment of the present application can also be set in a server or server cluster that is different from the server 105 and can communicate with the first image acquisition device 101, the second image acquisition device 102, the third image acquisition device 103 and / or the server 105.

[0030] It should be understood that Figure 1 The number of the first image acquisition device, the second image acquisition device, the third image acquisition device, the network, and the server is merely illustrative. Any number of the first image acquisition device, the second image acquisition device, the third image acquisition device, the network, and the server may be provided as required.

[0031] The following will be based on Figure 1 The scene described by Figures 2 to 6 The product defect detection method of the application embodiment is described in detail.

[0032] Figure 2 A flow chart of a product defect detection method according to an embodiment of the present application is shown.

[0033] like Figure 2 As shown, the product defect detection method of this embodiment includes operations S210 to S240.

[0034] In operation S210 , attention feature processing is performed on the initial image of the product to be tested to obtain an attention feature of the product to be tested.

[0035] In operation S220 , the initial image is encoded to obtain a plurality of encoding features at different levels, and features at the same level have the same scale.

[0036] In operation S230 , multiple feature reconstructions are performed on the multiple coding features to obtain target fusion features that meet the target level.

[0037] In operation S240 , a defect detection result of the product to be tested is determined based on the attention feature and the target fusion feature.

[0038] According to an embodiment of the present application, the product to be tested may include various products that require appearance inspection, such as servers, switches, etc. The initial image is an appearance image of the product to be tested obtained by capturing an image of the product to be tested.

[0039] According to an embodiment of the present application, attention features are processed on the initial image to obtain attention features that are used to represent the distribution of key features in the initial image.

[0040] According to the embodiments of the present application, the initial image can be encoded multiple times using different encoding methods to obtain multiple encoding features at different levels. The initial image can also be continuously chain-encoded, that is, after encoding the initial image to obtain an encoding result, the encoding result is further encoded to obtain multiple encoding features at different levels. The feature scales of the same level are the same, and the feature scales of different levels are different.

[0041] According to an embodiment of the present application, multiple coding features may be respectively decoded at least once to determine decoding feature sets corresponding to the multiple coding features, wherein each decoding feature set includes multiple decoding features at different levels.

[0042] According to an embodiment of the present application, each encoding feature is decoded to obtain a decoding feature having the same image size as the original image. The decoding process of at least one level of decoding can be an inverse process of the encoding process of the multi-level encoding.

[0043] For example, when multiple coded features at different levels are obtained by encoding the initial image multiple times using different encoding methods, when decoding each coded feature, each decoded feature can be decoded using a decoding method opposite to its encoding method to obtain a decoded feature corresponding to the coded feature. In this case, each decoded feature set only includes one decoded feature corresponding to the coded feature.

[0044] For another example, when multiple coding features at different levels are obtained by continuously chain encoding the initial image, during the decoding process of each coding feature, the arrangement order of the multiple coding methods included in the chain encoding can be determined, and multiple decoding methods that are respectively opposite to the coding methods can be arranged in an order opposite to the arrangement order, and the coding features can be continuously chain decoded. In this case, each decoded feature set includes multiple decoded features obtained during the chain decoding process of the coding feature.

[0045] According to an embodiment of the present application, after obtaining a decoding feature set, feature reconstruction is performed on multiple encoding features and multiple decoding feature sets to obtain a target fusion feature.

[0046] Specifically, feature reconstruction can be achieved by fusing multiple encoding features and multiple decoding features included in multiple decoding feature sets. Feature fusion is to place the elements included in multiple smaller feature matrices into a larger feature matrix by reorganizing them.

[0047] Taking the decoding of multiple coding features of different levels obtained by continuous chain coding of the initial image as an example, specifically, performing one feature reconstruction among multiple feature reconstructions, including: decoding the fusion features of the previous level to obtain the decoding fusion features of the current level; fusing the decoding features of the current level, the coding features of the current level and the decoding fusion features of the current level to obtain the fusion features of the current level, and the decoding features of the current level are obtained by decoding using the coding features of the previous level or the decoding features of the previous level.

[0048] According to an embodiment of the present application, the fusion features of the previous level are decoded to obtain the decoding fusion features of the current level, wherein the fusion features of the previous level are at a different level from the decoding fusion features of the current level. According to the level of the decoding fusion features of the current level, encoding features and multiple decoding features of the same level are selected from the encoding features and decoding feature sets, and the encoding features of the current level, multiple decoding features of the current level, and decoding fusion features of the current level are subjected to feature fusion to obtain the fusion features of the current level. Among them, the fusion features of the previous level are determined on the previous level using the same method. In particular, the fusion features of the first level are determined based on the encoding features that meet the level requirements.

[0049] Specifically, after the initial image is multi-level encoded and the encoding features that meet the hierarchical requirements are obtained, the encoding features are determined as the fusion features of the first level. In the decoding and set reconstruction process, the fusion features of the first level are used as the fusion features of the previous level, and are decoded to obtain the decoded fusion features of the current level. When the decoded fusion features of the current level do not meet the predetermined level, the decoded features of the current level are used as the fusion features of the previous level, and the above operation is repeated until the level of the fusion features of the current level meets the predetermined level, and the fusion features of the current level are used as the target fusion features. The predetermined level can be determined based on the initial image, that is, when the fusion features of the current level are at the same level as the initial image, the fusion features of the current level are used as the target fusion features.

[0050] According to the embodiments of this application, by fusing the encoding features, decoding features, and decoding fusion features of the current level, it is possible to gradually integrate multi-level feature information to form a more comprehensive target fusion feature. This hierarchical fusion method reduces information loss caused by encoding, while ensuring the integrity of the fusion results through judgment at predetermined levels, making defect detection results more reliable.

[0051] According to the embodiments of the present application, based on the attention features and the target fusion features, it is possible to simultaneously focus on which positions or features in the initial image of the product to be tested are more correlated with defects, as well as the detailed information of the initial image in various areas and scales, so as to determine the defect detection results of the product to be tested through the above information.

[0052] According to the embodiments of the present application, by performing attention feature processing, encoding, feature reconstruction and other operations on the initial image of the product to be tested, the key features and hierarchical information in the initial image can be effectively extracted. Among them, the attention feature highlights the importance of the defect-related area, and the encoding and feature reconstruction process captures the features of the initial image at different scales. Therefore, the target fusion feature integrates the detailed information of each level. Combining the attention feature and the target fusion feature can integrate the multi-dimensional information of the initial image to accurately judge the defects of the product to be tested and improve the detection accuracy.

[0053] According to an embodiment of the present application, attention feature processing is performed on the initial image of the product to be tested to obtain the attention feature of the product to be tested, including: performing convolution processing of multiple receptive fields on the initial image to obtain multiple convolution features; performing channel dimension combination on the multiple convolution features to obtain combined features; and performing attention feature processing on the combined features to obtain attention features.

[0054] According to an embodiment of the present application, a plurality of convolution kernels with different receptive fields are used to perform convolution processing on the initial image to obtain a plurality of convolution features, wherein the convolution processing method may include dilated convolution.

[0055] According to an embodiment of the present application, multiple convolutional features are used as input features of different channels and combined in the channel dimension to obtain a combined feature, wherein the dimension of the combined feature is increased by the channel dimension compared with the convolutional feature, and the scale in this dimension is the same as the number of convolutional features.

[0056] According to an embodiment of the present application, the combined features are subjected to attention processing, so as to determine which information in the combined features is more relevant to defect detection of the product to be tested, that is, the attention features of the initial image.

[0057] According to the embodiments of this application, multi-receptive field convolution processing generates convolution features of different scales, which are then combined into channel-dimensional composite features to capture multi-scale features in the image. Attention processing further enhances the saliency of key areas, allowing the attention features to focus more on potential defect areas, improving the targetedness of detection.

[0058] Figure 3 A data flow diagram for determining combined features according to a product defect detection method according to an embodiment of the present application is shown.

[0059] like Figure 3 As shown in the figure, the initial image is convolved using a dilated convolution method, where different receptive fields of dilated convolution can be distinguished by the dilation rate. In this embodiment, the selected convolution kernels may include a 1*1 convolution kernel, a 3*3 convolution kernel with a dilation rate of 6, a 3*3 convolution kernel with a dilation rate of 12, a 3*3 convolution kernel with a dilation rate of 18, and average pooling.

[0060] The initial image is processed using the above convolution kernels to obtain multiple convolution features, where the multiple convolution features have the same size to facilitate subsequent feature combination. The multiple convolution features are used as features of different channels, and the multiple convolution features are spliced ​​together to obtain a combined feature. The scale of the combined feature in the channel dimension is the same as the number of convolution features, which is 5 in this embodiment. The scale of the combined feature in the spatial dimension is the same as the scale of each convolution feature.

[0061] According to an embodiment of the present application, the combined features are subjected to attention feature processing to obtain attention features, including: performing channel attention processing and spatial attention processing on the combined features respectively to obtain channel attention features and spatial attention features; performing feature fusion on the combined features, channel attention features and spatial attention features to obtain attention features.

[0062] According to an embodiment of the present application, channel attention processing is performed on the combined features to determine which information in the combined features is more relevant to defect detection of the product to be tested in the channel dimension.

[0063] Specifically, for each convolution feature in the channel dimension, a feature extraction operation is performed to obtain the eigenvalues ​​of multiple convolution features. The multiple eigenvalues ​​are spliced ​​according to the channel dimension to obtain the attention result used to represent the combined feature in the channel dimension, namely the channel attention feature.

[0064] Among them, the feature extraction operation can include at least one of multiple feature extraction methods such as maximum pooling and average pooling. When the feature extraction operation represents the use of multiple feature extraction methods to extract features from multiple convolution features separately, channel attention sub-features obtained by using multiple feature extraction methods will be obtained. Multiple channel attention sub-features can be fused to obtain channel attention features.

[0065] According to an embodiment of the present application, spatial attention processing is performed on the combined features, which can determine which information in the combined features is more relevant to the defect detection of the product to be tested in the spatial dimensions of the multiple convolution features that constitute the combined features.

[0066] Specifically, the combined features are divided according to the spatial dimension, and the element values ​​of multiple channels at each spatial position can be obtained after the division according to the spatial dimension. Feature extraction operations are performed on the multiple element values ​​of each channel to obtain the characteristic values ​​of the multiple channels. The multiple characteristic values ​​are spliced ​​according to the spatial dimension to obtain the attention result used to represent the combined features in the spatial dimension, that is, the spatial attention feature.

[0067] Among them, the feature extraction operation in the spatial dimension is similar to the feature extraction operation in the channel dimension, and can also include at least one of multiple feature extraction methods such as maximum pooling and average pooling. When the feature extraction operation represents the use of multiple feature extraction methods to extract multiple convolution features separately, spatial attention sub-features obtained by using multiple feature extraction methods will be obtained. Multiple spatial attention sub-features can be fused to obtain spatial attention features.

[0068] According to an embodiment of the present application, after obtaining the channel attention feature and the spatial attention feature, the channel attention feature, the spatial attention feature and the combined feature can be fused to obtain the attention feature.

[0069] According to the embodiments of the present application, by performing channel attention processing and spatial attention processing respectively, the expressive ability of attention features can be enhanced from different dimensions. Channel attention highlights important feature channels, and spatial attention focuses on key areas. The fusion of the two can more comprehensively reflect the distribution and intensity of features that are more relevant to defect judgment in the initial image, thereby improving detection accuracy. On this basis, the above two attention features are further fused with the combined features, which can more accurately reflect the combined features and the original features of the initial image, thereby further improving the description ability of the attention features for the initial image.

[0070] Figure 4 A data flow diagram for determining attention features according to a product defect detection method according to an embodiment of the present application is shown.

[0071] like Figure 4 As shown, spatial attention processing and channel attention processing are performed on the combined features of the input respectively.

[0072] In the channel attention processing part, the combined features are divided according to the channel dimension to obtain the convolution features of multiple channels. The maximum pooling and average pooling processing are performed on the multiple convolution features respectively, and the maximum eigenvalue and average eigenvalue of the multiple convolution features can be obtained respectively. The maximum eigenvalues ​​of the multiple convolution features are arranged in the order of the multiple convolution features on the channel dimension to obtain the first channel attention sub-feature. The average eigenvalues ​​of the multiple convolution features are arranged in the order of the multiple convolution features on the channel dimension to obtain the second channel attention sub-feature.

[0073] After obtaining the first channel attention sub-feature and the second channel attention sub-feature, the first channel attention sub-feature and the second channel attention sub-feature are processed separately using a multi-layer perceptron to obtain the deep features of the first channel attention sub-feature and the second channel attention sub-feature, namely the first deep feature and the second deep feature. The above deep features are fused using an activation function to obtain the channel attention feature Mc, as shown in formula (1):

[0074] (1)

[0075] Among them, MLP(·) represents multi-layer perceptron processing, MaxPool(·) represents maximum pooling, AvgPool(·) represents average pooling, F represents combined features, and σ(·) represents activation function.

[0076] In the spatial attention processing part, the combined features are divided according to the spatial dimension to obtain vectors composed of element values ​​of multiple channels at multiple spatial positions. The vectors of each of the multiple spatial positions are subjected to maximum pooling and average pooling respectively to obtain the maximum eigenvalue and average eigenvalue of each of the multiple vectors respectively. The maximum eigenvalues ​​of each of the multiple vectors are arranged according to the spatial positions corresponding to the multiple vectors to obtain the first spatial attention sub-feature. The average eigenvalues ​​of each of the multiple vectors are arranged according to the spatial positions corresponding to the multiple vectors to obtain the second spatial attention sub-feature.

[0077] After obtaining the first spatial attention sub-feature and the second spatial attention sub-feature, the first spatial attention sub-feature and the second spatial attention sub-feature are fused and reduced in dimension through convolution operation to obtain the spatial attention feature Ms, as shown in formula (2):

[0078] (2)

[0079] Where Conv(·) represents the convolution operation.

[0080] After obtaining the channel attention feature and the spatial attention feature, the channel attention feature, the spatial attention feature and the combined feature can be multiplied to obtain the attention feature F out1 , as shown in formula (3):

[0081] (3)

[0082] According to an embodiment of the present application, the fusion features of the previous level are decoded to obtain the decoded fusion features of the current level, including: determining a decoding ratio based on the feature scales of the fusion features of the previous level and the decoded fusion features of the current level; determining multiple target convolution kernels from a preset convolution kernel set based on the decoding ratio; convolving the fusion features of the previous level based on the multiple target convolution kernels to obtain multiple output feature matrices; and reorganizing the multiple output feature matrices to obtain the decoded fusion features of the current level.

[0083] According to an embodiment of the present application, the feature scale may represent the size of the feature matrix corresponding to the feature. The decoding magnification may represent the multiple by which the size of the feature matrix needs to be magnified in the process of decoding the fused features of the previous level to obtain the fused features of the current level. For example, if the size of the fused features of the previous level is 3*3 and the size of the fused features of the current level is 6*6, the decoding magnification may be determined to be 4.

[0084] According to an embodiment of the present application, the number of target convolution kernels determined is the same as the decoding magnification, wherein the convolution kernels stored in the preset convolution kernel set may be 1*1 convolution kernels. For example, when the decoding magnification is 4, 4 target convolution kernels are selected from the preset convolution kernel set.

[0085] According to an embodiment of the present application, multiple target convolution kernels are used to convolve the fusion features of the previous level respectively, to obtain output feature matrices corresponding to the multiple target convolution kernels respectively.

[0086] Since the target convolution kernels are all 1*1 convolution kernels, after performing a 1*1 convolution on the fused features of the previous level, the size of the output feature matrix obtained is the same as the fused features of the previous level. In addition, the number of output feature matrices, that is, the number of target convolution kernels, is determined by the decoding magnification, which represents the multiple of the difference in size between the fused features of the previous level and the fused features of the current level. Therefore, the sum of the number of elements in multiple decoding feature matrices is the same as the number of elements required to constitute the fused features of the current level. By reorganizing multiple output feature matrices, the fused features of the current level can be obtained.

[0087] According to an embodiment of the present application, by dynamically determining the decoding ratio and selecting the corresponding number of target convolution kernels according to the decoding ratio, it is possible to ensure that the number of multiple output feature matrices obtained meets the requirements, thereby ensuring that the sum of the number of elements of the multiple output feature matrices meets the number of elements required to constitute the fusion features of the current level, thereby ensuring the feasibility of decoding. In addition, since each output feature matrix is ​​obtained by performing a convolution operation on the fusion features of the previous level, each output feature matrix has the same feature information as the fusion features of the previous level, thereby ensuring that the fusion features of the current level are the same as the features of the fusion features of the previous level, thereby ensuring the accuracy of the decoding process.

[0088] According to an embodiment of the present application, multiple output feature matrices are reorganized to obtain decoding fusion features of the current level, including: combining multiple eigenvalues ​​located at the same matrix position in the multiple output feature matrices, determining the decoding sub-matrix at the matrix position, and obtaining multiple decoding sub-matrices; and combining the multiple decoding sub-matrices to obtain decoding fusion features of the current level.

[0089] According to an embodiment of the present application, the multiple output feature matrices have the same size, and therefore, the multiple output feature matrices all have the same matrix position, for example, the Nth row and the Mth column.

[0090] According to an embodiment of the present application, after combining the eigenvalues ​​of each output feature matrix at the same matrix position, a decoding submatrix at that matrix position can be obtained, wherein the rows and columns of the decoding submatrix can be determined according to the decoding magnification. Specifically, the arithmetic square root of the decoding magnification can be used as the rows and columns of the decoding submatrix. For example, when the decoding magnification is 4, the rows and columns of the decoding submatrix can both be 2.

[0091] According to an embodiment of the present application, for each matrix position of the output feature matrix, after determining the decoding sub-matrix corresponding to the matrix position, multiple decoding sub-matrices are combined according to the positional relationship between the matrix positions to obtain the decoding fusion features of the current level.

[0092] For example, the decoding submatrix at the Nth row and the Mth column is positioned to the right of the decoding submatrix at the Nth row and the M-1th column, and below the decoding submatrix at the N-1th row and the Mth column.

[0093] According to the embodiments of the present application, by combining the eigenvalues ​​at the same position of multiple output feature matrices to obtain a decoding submatrix, and then integrating it into a complete decoding fusion feature at the current level, it is possible to more finely restore image details. This position-by-position reorganization method reduces information loss, ensuring that the resulting decoding fusion feature is closer to the local structure of the original image, which is beneficial for identifying subtle defects in details.

[0094] Figure 5A schematic diagram of decoding according to a product defect detection method according to an embodiment of the present application is shown.

[0095] like Figure 5 As shown, the size of the decoding feature of the previous level is 3*3, and the size of the decoding feature of the current level to be obtained is 6*6. By comparing the sizes of the two, it can be determined that the decoding magnification is 4, that is, the number of elements of the decoding feature of the previous level or the size of the feature matrix is ​​magnified by 4 times, and the decoding feature of the current level can be obtained.

[0096] According to the decoding ratio, four 1*1 target convolution kernels are determined from the preset convolution kernel set, and four 1*1 convolution operations are performed on the decoding matrix of the previous layer to obtain four output feature matrices.

[0097] For the four output feature matrices, the eigenvalues ​​at the same matrix position are extracted and combined in a preset order to obtain the decoding submatrix for that matrix position. Following the above approach, multiple eigenvalues ​​at each matrix position are combined in the same preset order. After determining the decoding submatrices for each of the multiple matrix positions, the multiple decoding submatrices are combined according to their respective matrix positions to obtain the decoding matrix for the current level.

[0098] According to an embodiment of the present application, for encoding the initial image to obtain multiple coding features of different levels, performing one encoding includes: encoding the coding features of the current level to obtain the coding features of the previous level, and the coding features of the current level are obtained by encoding the initial image at least once.

[0099] Specifically, encoding the initial image can yield a coding feature, which is then used as the coding feature for the current level, and encoding is continued to yield the coding feature for the previous level. If the coding feature for the previous level does not meet the level requirements, the coding feature for the previous level is used as the new coding feature for the current level, and encoding is continued to yield the new coding feature for the previous level until the obtained coding feature for the previous level meets the level requirements, at which point the encoding process is terminated.

[0100] According to the embodiments of this application, by progressively encoding and generating coded features of different sizes and levels, image information can be gradually compressed and multi-scale features can be extracted. The hierarchical relationship between the coded features of the current level and the coded features of the previous level ensures the consistency of feature extraction, allowing the encoding process to retain global information while focusing on local details, providing more comprehensive feature support for subsequent decoding and defect detection.

[0101] Figure 6 A model diagram of encoding and decoding an initial image according to a product defect detection method according to an embodiment of the present application is shown.

[0102] like Figure 6 As shown in FIG, the model is used to process the initial image to obtain the target fusion features of the initial image.

[0103] The initial image is encoded to obtain encoding features at the previous level compared to the level at which the initial image was encoded. These features are used as encoding features at the current level. Before encoding the initial image, a dilated convolution may be performed on it to reduce information loss during subsequent encoding. Similarly, before subsequent encoding and decoding operations, either the encoding features of the current level or the decoding features of the current level may be subjected to dilated convolution.

[0104] The coding features of the current level can be decoded to obtain decoding features of the same level as the initial image. Since the decoding features and the initial image are at the same level at this time, the decoding of the coding features of the first level can be ended.

[0105] Since the coding features of the current level do not meet the level requirements, the coding features of the current level continue to be encoded to obtain the coding features of the previous level. The coding features of the previous level can be decoded to obtain the decoding features of the current level at the same level as the coding features of the current level. The current level is different from the level of the initial image, so the decoding features of the current level need to be used as the previous decoding features and then decoded to obtain the decoding features of the current level. At this time, the decoding features of the current level are at the same level as the level of the initial image. The two decoding features obtained by decoding the coding features of the current level can constitute the decoding feature set corresponding to the coding features.

[0106] Similarly, the above operation is repeated until the coding features that meet the hierarchical requirements are obtained, and each coding feature is decoded at least once until the decoding features of the current level obtained after decoding are the same as the level of the initial image.

[0107] For the coding features that meet the hierarchical requirements, after performing dilated convolution on them, they are used as the fusion features of the previous level. The fusion features of the previous level are decoded to obtain the decoded fusion features of the current level, and the decoding features of the current level and the coding features of the current level that are at the same level as the decoding fusion features of the current level are selected. The decoding features of the current level, the coding features of the current level and the decoding fusion features of the current level are fused to obtain the fusion features of the current level. If the fusion features of the current level do not meet the predetermined level, the above feature fusion and decoding processes are repeated until the fusion features of the current level that meet the predetermined level are obtained, and the fusion features of the current level are used as the target fusion features.

[0108] According to an embodiment of the present application, a defect detection result of the product to be tested is determined based on the attention feature and the target fusion feature, including: performing feature fusion on the attention feature and the target fusion feature to determine the fusion feature of the product to be tested; determining the feature similarity between the fusion feature of the product to be tested and the reference feature of the defective product, where the defective product and the product to be tested are of the same product type; and when the feature similarity is greater than the similarity threshold, determining that the defect detection result indicates that the product to be tested is defective.

[0109] According to the embodiment of the present application, the attention feature and the target fusion feature are fused to determine the fusion feature F of the product to be tested. out , where feature fusion can be achieved by performing matrix multiplication on the feature matrices corresponding to the features, as shown in formula (4):

[0110] (4)

[0111] Among them, F out2 Represents the target fusion feature.

[0112] According to an embodiment of the present application, based on the product type of the product under test, a defective product of that product type is selected, and the reference features of the defective product are compared with the fused features of the product under test to determine the similarity between the two. If the similarity is greater than a similarity threshold, it can be determined that the features of the product under test and the defective product are relatively similar, and therefore it can be determined that the product under test is defective.

[0113] According to the embodiments of the present application, by fusing the attention feature and the target fusion feature to obtain a fusion feature, and performing a similarity comparison with the reference feature of the defective product, it is possible to quantitatively determine the degree of similarity between the product under test and defective products of the same product type, and to perform a quantitative and definitive classification based on the similarity threshold. Setting a similarity threshold for judgment can avoid subjective errors, making defect detection results more objective and reliable, and suitable for automated quality control scenarios.

[0114] Before using the product defect detection method of the present application to detect the product to be tested, a feature extraction method can be used to determine the image features of the initial image. Based on the image features, a preliminary judgment is made on the possible defect categories of the product to be tested. According to the defect category, the weights of channel attention and spatial attention are adjusted during the attention feature processing of the initial image; after the weight adjustment is completed, the initial image is processed for attention features, and the attention features of the product to be tested are obtained.

[0115] According to an embodiment of the present application, the feature extraction method for extracting features from the initial image may be convolution, pooling, etc., and the target fusion features of the initial image may be used as the image features of the initial image.

[0116] According to an embodiment of the present application, a classifier can be used to process image features to determine whether the possible defect category of the product to be tested requires a focus on spatial attention or channel attention.

[0117] For example, if the surface of the product under test has scratches, defect detection requires higher spatial positioning accuracy, so the weight of spatial attention needs to be increased. If the image features include thin stripes or sudden grayscale changes, it indicates that the surface of the product under test may have scratches.

[0118] For example, when the product under test has a loose solder joint, defect detection requires stronger channel feature differentiation, and the weight of channel attention should be increased. If the image features have uneven grayscale or blurred edges, it indicates that the product under test may have a loose solder joint.

[0119] Therefore, by classifying the image features, we can determine the possible defect categories of the product under test and whether spatial attention or channel attention should be emphasized in the subsequent image processing process. This allows us to dynamically adjust the weights of spatial attention and channel attention.

[0120] According to the embodiments of the present application, based on the image features of the initial image, it is possible to determine the defect category that may exist in the initial image through classification, and to adjust the weights of the channel attention and spatial attention in a targeted manner, so that the subsequent attention feature processing process can pay more attention to more useful information for the defect category. In addition, image features can be obtained through simple feature extraction methods such as convolution and pooling, so the above-mentioned simple basic feature extraction methods will not have too much impact on the load and efficiency of product defect detection. Image features can also be determined based on the target fusion features of the initial image, and since the target fusion features are features that must be calculated when performing product defect detection, they will not increase the computational load.

[0121] Based on the above product defect detection method, this application also provides a product defect detection device. Figure 7 The device is described in detail.

[0122] Figure 7 The figure shows a structural block diagram of a product defect detection device according to an embodiment of the present application.

[0123] like Figure 7 As shown, the product defect detection device 700 of this embodiment includes an attention feature extraction module 710 , an image encoding module 720 , a feature reconstruction module 730 and a defect detection module 740 .

[0124] The attention feature extraction module 710 is used to perform attention feature processing on the initial image of the product to be tested to obtain the attention feature of the product to be tested. In one embodiment, the attention feature extraction module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0125] The image encoding module 720 is used to encode the initial image to obtain multiple encoding features of different levels, and the feature scales of the same level are the same. In one embodiment, the image encoding module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0126] The feature reconstruction module 730 is used to perform multiple feature reconstructions on the multiple coding features to obtain target fusion features that meet the target level. In one embodiment, the feature reconstruction module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0127] The defect detection module 740 is used to determine the defect detection result of the product to be tested based on the attention feature and the target fusion feature. In one embodiment, the defect detection module 740 can be used to perform the operation S240 described above, which will not be repeated here.

[0128] According to an embodiment of the present application, the feature reconstruction module 730 includes a feature decoding submodule and a feature fusion submodule.

[0129] The feature decoding submodule is used to decode the fused features of the previous level to obtain the decoded fused features of the current level.

[0130] The feature fusion submodule is used to fuse the decoding features of the current level, the encoding features of the current level and the decoding fusion features of the current level to obtain the fusion features of the current level. The decoding features of the current level are obtained by decoding using the encoding features of the previous level or the decoding features of the previous level.

[0131] According to an embodiment of the present application, the feature decoding submodule includes a magnification determination unit, a convolution kernel determination unit, a feature processing unit and a feature reorganization unit.

[0132] The magnification determining unit is used to determine the decoding magnification according to the feature scales of the fusion feature of the previous level and the decoding fusion feature of the current level.

[0133] The convolution kernel determination unit is used to determine a plurality of target convolution kernels from a preset convolution kernel set according to a decoding magnification.

[0134] The feature processing unit is used to convolve the fused features of the previous level based on multiple target convolution kernels to obtain multiple output feature matrices.

[0135] The feature recombination unit is used to reorganize multiple output feature matrices to obtain the decoding fusion features of the current level.

[0136] According to an embodiment of the present application, the feature recombination unit includes an eigenvalue combination subunit and a sub-matrix combination subunit.

[0137] The eigenvalue combining subunit is used to combine multiple eigenvalues ​​located at the same matrix position in multiple output feature matrices, determine the decoding submatrix at the matrix position, and obtain multiple decoding submatrices.

[0138] The sub-matrix combination sub-unit is used to combine multiple decoding sub-matrices to obtain the decoding fusion features of the current level.

[0139] According to an embodiment of the present application, the image encoding module 720 includes a feature encoding submodule.

[0140] The feature encoding submodule is used to encode the encoding features of the current level to obtain the encoding features of the previous level. The encoding features of the current level are obtained by encoding the initial image at least once.

[0141] According to an embodiment of the present application, the attention feature extraction module 710 includes an image convolution submodule, a feature combination submodule and a feature processing submodule.

[0142] The image convolution submodule is used to perform convolution processing on the initial image with multiple receptive fields to obtain multiple convolution features.

[0143] The feature combination submodule is used to combine multiple convolutional features in channel dimensions to obtain combined features.

[0144] The feature processing submodule is used to perform attention feature processing on the combined features to obtain attention features.

[0145] According to an embodiment of the present application, the feature processing submodule includes a sub-feature extraction unit and a feature fusion unit.

[0146] The sub-feature extraction unit is used to perform channel attention processing and spatial attention processing on the combined features respectively to obtain channel attention features and spatial attention features.

[0147] The feature fusion unit is used to fuse the combined features, channel attention features and spatial attention features to obtain the attention features.

[0148] According to an embodiment of the present application, the defect detection module 740 includes a feature fusion submodule, a similarity determination submodule, and a defect detection submodule.

[0149] The feature fusion submodule is used to fuse the attention features and the target fusion features to determine the fusion features of the product to be tested.

[0150] The similarity determination submodule is used to determine the feature similarity between the fused features of the product to be tested and the reference features of the defective product, where the defective product and the product to be tested are of the same product type.

[0151] The defect detection submodule is used to determine that the defect detection result indicates that the product to be tested has a defect when the feature similarity is greater than a similarity threshold.

[0152] According to an embodiment of the present application, any multiple modules among the attention feature extraction module 710, the image encoding module 720, the feature reconstruction module 730, and the defect detection module 740 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present application, at least one of the attention feature extraction module 710, the image encoding module 720, the feature reconstruction module 730, and the defect detection module 740 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the attention feature extraction module 710, the image encoding module 720, the feature reconstruction module 730 and the defect detection module 740 can be at least partially implemented as a computer program module, which can perform the corresponding function when it is executed.

[0153] Figure 8 A block diagram of an electronic device suitable for implementing a product defect detection method according to an embodiment of the present application is shown.

[0154] like Figure 8As shown, an electronic device 800 according to an embodiment of the present application includes a processor 801, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 802 or a program loaded from a storage unit 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present application.

[0155] Various programs and data required for the operation of the electronic device 800 are stored in the RAM 803. The processor 801, ROM 802, and RAM 803 are connected to each other via a bus 804. The processor 801 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than the ROM 802 and the RAM 803. The processor 801 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in the one or more memories.

[0156] According to an embodiment of the present application, electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to bus 804. Electronic device 800 may also include one or more of the following components connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 808 including a hard disk; and a communication section 809 including a network interface card such as a LAN card or modem. Communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. Removable media 811, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 810 as needed, so that computer programs read from the removable media can be installed into storage section 808 as needed.

[0157] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of this application is implemented.

[0158] According to an embodiment of the present application, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0159] The embodiments of the present application also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided in the embodiments of the present application.

[0160] The computer program executes the above functions defined in the system / device of the embodiment of the present application when the processor 801 executes the computer program. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0161] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0162] In such an embodiment, the computer program can be downloaded and installed from the network via the communication section 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above-mentioned functions defined in the system of the embodiment of the present application are performed. According to the embodiment of the present application, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0163] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0164] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0165] Those skilled in the art will appreciate that the features described in the various embodiments of this application may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in this application. In particular, the features described in the various embodiments of this application may be combined and / or coupled in various ways without departing from the spirit and teachings of this application. All such combinations and / or couplings fall within the scope of this application.

[0166] The embodiments of the application have been described. However, these embodiments are merely for illustration and are not intended to limit the scope of the application. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Various alternatives and modifications to the embodiments described herein will be apparent to those skilled in the art in view of the foregoing description. Such alternatives and modifications are intended to fall within the scope of the application.

Claims

1. A product defect detection method, characterized in that: The method comprises: Performing attention feature processing on the initial image of the product to be tested to obtain the attention feature of the product to be tested; Encoding the initial image to obtain a plurality of encoding features at different levels, wherein the feature scales at the same level are the same; Performing multiple feature reconstructions on the plurality of coding features to obtain target fusion features that meet the target level; Performing feature fusion on the attention feature and the target fusion feature to determine the fusion feature of the product to be tested; determining a feature similarity between the fused feature of the product to be tested and a reference feature of a defective product, the defective product being of the same product type as the product to be tested; and When the feature similarity is greater than a similarity threshold, determining that the defect detection result indicates that the product to be tested has a defect; The feature reconstruction is performed once, including: Decode the fused features of the previous level to obtain the decoded fused features of the current level; The decoding features of the current level, the encoding features of the current level and the decoding fusion features of the current level are fused to obtain the fusion features of the current level, where the decoding features of the current level are obtained by decoding using the encoding features of the previous level or the decoding features of the previous level.

2. The method according to claim 1, characterized in that The decoding of the fusion features of the previous level to obtain the decoded fusion features of the current level includes: Determining a decoding magnification according to the feature scales of the fusion feature of the previous level and the decoding fusion feature of the current level; Determining a plurality of target convolution kernels from a preset convolution kernel set according to the decoding magnification; Based on the multiple target convolution kernels, convolve the fusion features of the previous level respectively to obtain multiple output feature matrices; and The plurality of output feature matrices are reorganized to obtain the decoding fusion features of the current level.

3. The method according to claim 2, characterized in that The reorganizing the plurality of output feature matrices to obtain the decoding fusion features of the current level includes: Combining multiple eigenvalues ​​located at the same matrix position in the multiple output feature matrices to determine a decoding submatrix at the matrix position, thereby obtaining multiple decoding submatrices; and The multiple decoding sub-matrices are combined to obtain the decoding fusion features of the current level.

4. The method according to claim 1, wherein The initial image is encoded to obtain a plurality of encoding features at different levels, and performing encoding once includes: The coding features of the current level are encoded to obtain the coding features of the previous level, where the coding features of the current level are obtained by encoding the initial image at least once.

5. The method according to claim 1, wherein The performing attention feature processing on the initial image of the product to be tested to obtain the attention feature of the product to be tested includes: Performing convolution processing of multiple receptive fields on the initial image to obtain multiple convolution features; Combining the plurality of convolutional features in terms of channel dimensions to obtain a combined feature; and The combined feature is subjected to attention feature processing to obtain the attention feature.

6. The method according to claim 5, characterized in that The performing attention feature processing on the combined feature to obtain the attention feature includes: Performing channel attention processing and spatial attention processing on the combined features respectively to obtain channel attention features and spatial attention features; The combined feature, the channel attention feature and the spatial attention feature are fused to obtain the attention feature.

7. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images