Vision positioning system and method for alignment and assembly of display screen modules

Abstract

The present application discloses a vision positioning system and method for alignment and assembly of a display module. By using an image processing and analysis algorithm based on artificial intelligence and machine vision technologies, the gray-scale image of the alignment and assembly state is analyzed to capture the multi-scale enhanced fusion features of the display module image therein, and then the image distortion correction is performed using these features to generate a gray-scale image of the alignment and assembly state after distortion correction. In this way, by using the corrected image to calculate the positioning deviation and adjust the pose of the display module, the accuracy and efficiency of the alignment and assembly of the display module can be greatly improved, while reducing human errors and ensuring the consistency of product quality.

Description

Technical Field

[0001] The present application relates to the field of intelligent positioning, and more specifically, to a vision positioning system and method for alignment and assembly of a display module. Background Art

[0002] With the continuous expansion of the smartphone market and the continuous innovation of technology, consumers have higher and higher requirements for mobile phone screens. They not only pursue higher resolutions and faster refresh rates, but also pay more attention to the overall visual experience and touch experience of the screen. To meet these needs, mobile phone manufacturers need to ensure that every production link reaches high-standard quality control. Especially in the assembly process of the display module, the alignment accuracy directly affects the display effect of the final product.

[0003] In the assembly of a mobile phone display module, multiple components such as a touch screen, a liquid crystal panel, a backlight module, etc. need to be accurately aligned and fixed together. This process requires extremely high precision. Any slight misalignment may cause problems such as edge light leakage and touch failure, thereby affecting the user experience. In traditional display module alignment and assembly methods, alignment and assembly often rely on the experience and visual inspection ability of operators. Such a method is not only inefficient, but also prone to inaccurate positioning due to individual differences. In addition, long-term manual observation will cause visual fatigue, thereby affecting the accuracy of judgment. Some early automation attempts used simple mechanical positioning devices. Although such devices can reduce the labor input, due to the lack of intelligent judgment ability and adaptability in complex environments, they are often difficult to cope with the changes in diverse product specifications and production conditions. In addition, both manual and simple mechanical positioning are easily affected by various factors in the production environment, such as light conditions, temperature changes, dust, etc. These factors will increase the alignment difficulty and lead to unstable final assembly quality.

[0004] Therefore, an optimized positioning solution for alignment and assembly of a display module is desired. Summary of the Invention

[0005] To solve the above technical problems, the present application is proposed. Embodiments of the present application provide a vision positioning system and method for alignment and assembly of a display module. By using image processing and analysis algorithms based on artificial intelligence and machine vision technologies to analyze the gray-scale image of the alignment and assembly state, multi-scale enhanced fusion features of the display module image therein are captured, and then the image distortion correction is performed using these features to generate a gray-scale image of the alignment and assembly state after distortion correction. In this way, using the corrected image to calculate the positioning deviation and adjust the pose of the display module can greatly improve the accuracy and efficiency of the alignment and assembly of the display module, while reducing human errors and ensuring the consistency of product quality.

[0006] According to one aspect of the present application, a visual positioning method for alignment and assembly of a display module is provided, which includes:

[0007] Obtaining an alignment and assembly status image of the display module collected by an industrial camera;

[0008] Performing gray-scale processing on the alignment and assembly status image to obtain an alignment and assembly status gray-scale image;

[0009] Performing distortion correction on the alignment and assembly status gray-scale image to obtain a distortion-corrected alignment and assembly status gray-scale image;

[0010] Identifying positioning marks in the distortion-corrected alignment and assembly status gray-scale image to obtain a current positioning result;

[0011] Calculating a deviation value based on a comparison between the current positioning result and a preset target position;

[0012] Adjusting the position and pose of the display module through a mechanical control system based on the deviation value.

[0013] According to another aspect of the present application, a visual positioning system for alignment and assembly of a display module is provided, which includes:

[0014] An alignment and assembly status image acquisition module for obtaining an alignment and assembly status image of the display module collected by an industrial camera;

[0015] A gray-scale processing module for performing gray-scale processing on the alignment and assembly status image to obtain an alignment and assembly status gray-scale image;

[0016] A distortion correction module for performing distortion correction on the alignment and assembly status gray-scale image to obtain a distortion-corrected alignment and assembly status gray-scale image;

[0017] A positioning identification module for identifying positioning marks in the distortion-corrected alignment and assembly status gray-scale image to obtain a current positioning result;

[0018] A deviation calculation module for calculating a deviation value based on a comparison between the current positioning result and a preset target position;

[0019] A position and pose adjustment module for adjusting the position and pose of the display module through a mechanical control system based on the deviation value.

[0020] Compared with the prior art, a vision positioning system and method for alignment and assembly of a display module provided by the present application analyze a grayscale image of the alignment and assembly state by using image processing and analysis algorithms based on artificial intelligence and machine vision technologies, so as to capture multi-scale enhanced fusion features of the display module image therein, and then use these features to correct image distortion to generate a grayscale image of the alignment and assembly state after distortion correction. In this way, using the corrected image to calculate the positioning deviation and adjust the pose of the display module can greatly improve the accuracy and efficiency of the alignment and assembly of the display module, reduce human errors at the same time, and ensure the consistency of product quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other objects, features and advantages of the present application will become more obvious. The accompanying drawings are used to provide a further understanding of the embodiments of the present application, and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application, and do not constitute a limitation to the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 It is a flowchart of a vision positioning method for alignment and assembly of a display module according to an embodiment of the present application;

[0023] Figure 2 It is a schematic diagram of data flow of a vision positioning method for alignment and assembly of a display module according to an embodiment of the present application;

[0024] Figure 3 It is a flowchart of sub-step S3 of a vision positioning method for alignment and assembly of a display module according to an embodiment of the present application;

[0025] Figure 4 It is a block diagram of a vision positioning system for alignment and assembly of a display module according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] Next, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. It should be understood that the present application is not limited by the exemplary embodiments described herein.

[0027] As shown in the present application and the claims, unless the context clearly indicates an exception, words such as "a", "an", "one" and / or "the" are not specifically singular, but may also include plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0028] Although the present application makes various references to certain modules in the system according to embodiments of the present application, any number of different modules may be used and run on a user terminal and / or a server. The modules are merely illustrative, and different aspects of the system and method may use different modules.

[0029] Flowcharts are used in the present application to illustrate the operations performed by the system according to embodiments of the present application. It should be understood that the operations above or below do not necessarily have to be performed precisely in sequence. On the contrary, various steps may be processed in reverse order or simultaneously as needed. At the same time, other operations may also be added to these processes, or one or more steps may be removed from these processes.

[0030] Next, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. It should be understood that the present application is not limited by the example embodiments described herein.

[0031] In the traditional alignment and assembly method of a display screen module, alignment and assembly often rely on the experience and visual inspection ability of operators. Such a method is not only inefficient but also prone to inaccurate positioning due to individual differences. In addition, long-term manual observation can cause visual fatigue, which in turn affects the accuracy of judgment. Some early automation attempts used simple mechanical positioning devices. Although such devices can reduce the labor input, due to the lack of intelligent judgment ability and adaptability in complex environments, they often have difficulty coping with the changes in diverse product specifications and production conditions. In addition, both manual and simple mechanical positioning are easily affected by various factors in the production environment, such as light conditions, temperature changes, dust, etc. These factors will increase the alignment difficulty and lead to unstable final assembly quality. Therefore, an optimized positioning scheme for the alignment and assembly of a display screen module is desired.

[0032] In recent years, with the progress of computer vision technology, vision positioning systems have been widely used in the alignment and assembly process of display screen modules due to their characteristics of high precision, high speed, and non-contact measurement. However, during actual vision positioning, due to the physical characteristics of the camera itself and the influence of the working environment, the collected images may be distorted, such as image deformation caused by factors such as lens distortion and light refraction. These factors will have an adverse impact on the positioning accuracy.

[0033] Based on this, in the technical solution of the present application, a vision positioning method for the alignment and assembly of a display screen module is proposed. Figure 1 It is a flowchart of the vision positioning method for the alignment and assembly of a display screen module according to an embodiment of the present application. Figure 2Schematic diagram of data flow for a vision positioning method for alignment and assembly of a display module according to an embodiment of the present application. As Figure 1 and Figure 2 shown, a vision positioning method for alignment and assembly of a display module according to an embodiment of the present application includes the steps of: S1, acquiring an alignment and assembly state image of the display module collected by an industrial camera; S2, performing grayscale processing on the alignment and assembly state image to obtain an alignment and assembly state grayscale image; S3, performing distortion correction on the alignment and assembly state grayscale image to obtain a distortion-corrected alignment and assembly state grayscale image; S4, identifying positioning marks in the distortion-corrected alignment and assembly state grayscale image to obtain a current positioning result; S5, calculating a deviation value based on a comparison between the current positioning result and a preset target position; S6, adjusting the position and pose of the display module through a mechanical control system based on the deviation value.

[0034] Specifically, for S1 and S2, an alignment and assembly state image of the display module collected by an industrial camera is acquired; and grayscale processing is performed on the alignment and assembly state image to obtain an alignment and assembly state grayscale image. It should be understood that grayscale processing can enhance the contrast in the alignment and assembly state image, making features such as edges and contours in the image more obvious, which is helpful for subsequent image feature extraction.

[0035] Specifically, for S3, the alignment and assembly state grayscale image is subjected to distortion correction to obtain a distortion-corrected alignment and assembly state grayscale image. In a specific example of the present application, as Figure 3 shown, S3 includes: S31, performing feature extraction on the alignment and assembly state grayscale image based on the image of the display module to obtain display module image features; S32, performing space constraint enhancement processing on the display module image features to obtain display module image enhanced features; S33, performing feature perception enhancement processing on the display module image features to obtain display module image perception enhanced features; S34, fusing the display module image enhanced features and the display module image perception enhanced features, and using the multi-scale fusion enhanced features of the fused display module image to generate the distortion-corrected alignment and assembly state grayscale image.

[0036] Specifically, in S31, feature extraction based on the display module image is performed on the grayscale image of the alignment and assembly state to obtain the display module image features. In a specific example of the present application, the grayscale image of the alignment and assembly state is input into a display module image feature extractor based on a depthwise separable convolutional network to obtain a display module image feature map as the display module image features. By inputting the grayscale image of the alignment and assembly state into the display module image feature extractor based on the depthwise separable convolutional network for feature mining, the semantic features regarding the alignment and assembly state of the display module in the grayscale image of the alignment and assembly state are extracted, thereby obtaining the display module image feature map.

[0037] Specifically, in S32, spatial constraint enhancement processing is performed on the display module image features to obtain enhanced display module image features. Since the display module image feature map contains various details and global information of the display module, but the original display module image feature map does not fully utilize the relationship between local and global information. Therefore, in order to better capture the detailed features in the image, improve the detailed semantic correlation and structured information between each pixel point in the feature map, and thus obtain a more abundant feature representation of the alignment and assembly state of the display module, in the technical solution of the present application, spatial constraint enhancement processing is further performed on the display module image features to obtain enhanced display module image features. Through spatial constraint enhancement processing, the expressiveness of the display module image feature map can be improved by combining the local features and global context information of the alignment and assembly state of the display module. The core lies in introducing a spatial sensitivity adjustment mechanism for the semantic relationship between pixels. By performing a detailed analysis and transformation on the display module image feature map, a more abundant and structured feature representation of the alignment and assembly state of the display module is provided, so as to better capture the key patterns and complex relationships in the feature map.

[0038] In an embodiment of the present application, performing spatial constraint enhancement processing on the display module image features to obtain enhanced display module image features includes: performing feature dispersion on the display module image features to obtain a set of display module image pixel granularity features, and then calculating the semantic association constraint score on the set of display module image pixel granularity features to obtain a topological representation of the pixel granularity semantic association spatial constraint score; performing semantic association encoding on the set of display module image pixel granularity features and the topological representation of the pixel granularity semantic association spatial constraint score to obtain the enhanced display module image features.

[0039] Specifically, first, after the display screen module image features are feature scattered to obtain a set of display screen module image pixel granularity features, the set of display screen module image pixel granularity features is subjected to semantic association constraint score calculation to obtain a pixel granularity semantic association space constraint score topological representation. More specifically, first, the display screen module image feature map is feature scattered along the channel dimension to obtain a set of display screen module image pixel granularity feature vectors; that is, the display screen module image feature map is feature scattered along the channel dimension to modulate the feature shape of the input feature map into a pixel granularity feature distribution set; then, the semantic association score between any two display screen module image pixel granularity feature vectors in the set of display screen module image pixel granularity feature vectors is calculated to obtain a pixel granularity semantic association score topological matrix; by calculating the semantic association score between any two display screen module image pixel granularity feature vectors in the set of display screen module image pixel granularity feature vectors and constructing a pixel granularity semantic association score topological matrix, the potential relationship between the alignment assembly states of different regions in the display screen module image can be revealed. This helps to not only rely on local information when processing images, but also comprehensively consider the global context, thereby improving the semantic relevance of the feature map; then, based on the spatial distance between any two display screen module image pixel granularity feature vectors in the set of display screen module image pixel granularity feature vectors, the pixel granularity semantic association score topology matrix is spatially attention attenuated modulated to obtain a pixel granularity semantic association spatial soft constraint score topology matrix; here, considering that images in the real world often have spatial continuity, that is, the correlation between adjacent pixels is usually stronger than that between distant pixels, it is necessary to implement spatial attention attenuation modulation on the pixel granularity semantic association score topology matrix obtained above. Specifically, based on the spatial distance between any two display screen module image pixel granularity feature vectors in the set of display screen module image pixel granularity feature vectors, the pixel granularity semantic association score topology matrix is spatially attention attenuated modulated to obtain a pixel granularity semantic association spatial soft constraint score topology matrix. By introducing spatial attention attenuation modulation, the continuity and attenuation characteristics of image spatial features can be better reflected, so that the expression of feature maps is more consistent with the distribution law of actual images; further, the pixel-granularity semantic association space soft constraint score topology matrix is subjected to hole convolution coding to obtain the pixel-granularity semantic association space soft constraint score topology feature matrix as the pixel-granularity semantic association space constraint score topology representation. Among them, hole convolution coding allows the module to capture the details of the pixel-granularity semantic association space soft constraint score topology matrix at different scales, so that even in the face of a large number of detail changes, important spatial location information can be retained through multi-scale context aggregation.

[0040] Among them, the process of performing feature dispersion on the display module image feature map along the channel dimension to obtain a set of display module image pixel granularity feature vectors includes: performing feature dispersion on the display module image feature map along the channel dimension to obtain a set of display module image feature vectors; performing feature transformation on each display module image feature vector in the set of display module image feature vectors to obtain the set of display module image pixel granularity feature vectors.

[0041] More specifically, the process of calculating the semantic association score between any two display module image pixel granularity feature vectors in the set of display module image pixel granularity feature vectors to obtain the pixel granularity semantic association score topological matrix includes: calculating the square of the first norm of the position difference between the i-th display module image pixel granularity feature vector and the j-th display module image pixel granularity feature vector in the set of display module image pixel granularity feature vectors to obtain the display module image pixel granularity semantic difference first norm representation; calculating the difference between the constant 1 and the square of the first norm of the i-th display module image pixel granularity feature vector to obtain the i-th display module image pixel granularity semantic representation; calculating the difference between the constant 1 and the square of the first norm of the j-th display module image pixel granularity feature vector to obtain the j-th display module image pixel granularity semantic representation; after calculating the product between the i-th display module image pixel granularity semantic representation and the j-th display module image pixel granularity semantic representation, dividing the display module image pixel granularity semantic difference first norm representation by the product value to obtain the display module image pixel granularity semantic association information representation; calculating the sum of twice the display module image pixel granularity semantic association information representation and the constant 1, and then inputting the obtained value into the inverse hyperbolic cosine function to obtain the pixel granularity semantic association score value; arranging multiple pixel granularity semantic association score values in matrix form to obtain the pixel granularity semantic association score topological matrix.

[0042] More specifically, the process of performing spatial attention attenuation modulation on the pixel granularity semantic association score topology matrix based on the spatial distance between any two display module image pixel granularity feature vectors in the set of display module image pixel granularity feature vectors to obtain the pixel granularity semantic association spatial soft constraint score topology matrix includes: calculating the spatial distance between the i-th display module image pixel granularity feature vector and the j-th display module image pixel granularity feature vector to obtain a pixel granularity semantic spatial distance representation; calculating the value of the natural exponential function with the pixel granularity semantic spatial distance representation as the exponent and the natural constant e as the base to obtain a pixel granularity semantic spatial distance class support representation; multiplying the pixel granularity semantic spatial distance class support representation by the spatial attention attenuation modulation hyperparameter to obtain a spatial distance attention attenuation representation, and then dividing the eigenvalue at the (i, j) position in the pixel granularity semantic association score topology matrix by the spatial distance attention attenuation representation to obtain a pixel granularity semantic association spatial soft constraint score; arranging a plurality of the pixel granularity semantic association spatial soft constraint scores in matrix form to obtain the pixel granularity semantic association spatial soft constraint score topology matrix.

[0043] Furthermore, semantic association encoding is performed on the set of display module image pixel granularity features and the pixel granularity semantic association spatial constraint score topology representation to obtain the enhanced features of the display module image. More specifically, first, the set of display module image feature vectors and the pixel granularity semantic association spatial soft constraint score topology matrix are input into the graph convolution encoding module to obtain a set of display module image context pixel granularity feature vectors; during the processing of the graph convolution encoding module, nodes represent display module image feature vectors, and edges represent the relationships between nodes, that is, the pixel granularity semantic association spatial soft constraint score topology matrix. By aggregating the information of neighbor nodes to update the current node state, the learning of the entire image structure is realized, and the global semantic context association feature expression between each display module image feature vector is enhanced. This method can help the model better understand the complex patterns in the image, especially when comprehensive consideration of background knowledge in a wide range is required. Then, the set of display module image context pixel granularity feature vectors is subjected to feature shape reshaping to obtain an enhanced feature map of the display module image as the enhanced feature of the display module image. Through the above processing process, the representation ability of the display module image feature map can be improved, ensuring that key features can be more accurately identified and located during the alignment and assembly process of the display module, thereby improving the accuracy and reliability of the overall positioning.

[0044] In summary, in the above embodiments, performing spatial constraint enhancement processing on the image features of the display screen module to obtain enhanced image features of the display screen module includes: performing spatial constraint enhancement processing on the image features of the display screen module with the following spatial constraint enhancement formula to obtain the enhanced image features of the display screen module; wherein, the spatial constraint enhancement formula is:

[0045]

[0046] h i = W1v i W2

[0047]

[0048] T = AtrousConv(S'')

[0049] V ′ = GCN(V, T)

[0050] F = reshape[X, (H, W, C)]

[0051] wherein, X is the image feature map of the display screen module, H, W, and C are respectively the height, width, and number of channels of the image feature map of the display screen module, reshape is feature reshaping processing, and V is a set of display screen module image feature vectors, the value of HW is the number of feature vectors in the set of display screen module image feature vectors, W1 and W2 are feature transformation weight matrices, h i and h j are respectively the i-th and j-th display screen module image pixel granularity feature vectors in the set of display screen module image pixel granularity feature vectors, ||·|| is the one-norm of the vector, arccosh is the inverse hyperbolic cosine function, dP(·, ·) is the semantic association score calculation operation, S ij is the eigenvalue at the (i, j) position in the pixel granularity semantic association score topology matrix, d(·, ·) is the vector space distance calculation operation, α is the spatial attention attenuation modulation hyperparameter, S ij ' is the eigenvalue at the (i, j) position in the pixel granularity semantic association space soft constraint score topology matrix, S' is the pixel granularity semantic association space soft constraint score topology matrix, AtrousConv is the dilated convolution encoding process, T is the pixel granularity semantic association space soft constraint score topology feature matrix, GCN is the graph convolution encoding operation, V' is the set of display screen module image context pixel granularity feature vectors, and F is the enhanced image feature map of the display screen module.

[0052] Specifically, in step S33, the image features of the display module are subjected to feature perception enhancement processing to obtain the enhanced image perception features of the display module. It should be understood that since the image feature map of the display module contains detailed feature information regarding the alignment and assembly state of the display module in different aspects, in order to perform global long-distance dependence correlation encoding on these features to more comprehensively and fully understand the alignment and assembly state of the display module, in the technical solution of this application, the image feature map of the display module is further input into a feature perception enhancement module based on a transformer to obtain the enhanced image perception feature map of the display module. It should be understood that the architecture based on a transformer is good at processing sequence data and can capture long-distance dependence relationships. In image processing, a transformer can be regarded as a high-level feature abstraction tool that recombines the information in the image feature map of the display module through a self-attention mechanism, thereby generating a more abstract and more discriminative new feature representation to characterize the global patterns and relationships of the alignment and assembly state of the display module, which is crucial for subsequent image processing steps such as distortion correction and helps to accurately align and assemble the display module subsequently.

[0053] Specifically, in step S34, the enhanced image features of the display module and the enhanced image perception features of the display module are fused, and the multi-scale fusion enhanced image features of the display module after fusion are used to generate the gray image of the alignment and assembly state after distortion correction. In a specific example of this application, the enhanced image feature map of the display module and the enhanced image perception feature map of the display module are fused to obtain the multi-scale fusion enhanced image feature map of the display module as the multi-scale fusion enhanced image features of the display module; and the multi-scale fusion enhanced image feature map of the display module is input into a distortion correction module based on a diffusion model to obtain the gray image of the alignment and assembly state after distortion correction. That is to say, different-scale enhanced fusion features of the display module are used to complement each other for image distortion correction, thereby generating the gray image of the alignment and assembly state after distortion correction. In this way, by using the corrected image to calculate the positioning deviation and adjust the pose of the display module, the accuracy and efficiency of the alignment and assembly of the display module can be greatly improved, while reducing human errors and ensuring the consistency of product quality.

[0054] Preferably, in an example of this application, inputting the multi-scale fusion enhanced image feature map of the display module into a distortion correction module based on a diffusion model to obtain the gray image of the alignment and assembly state after distortion correction includes:

[0055] Calculate the sum of the absolute values of the eigenvalues of the multi-scale fusion enhanced feature map of the display module image to obtain the first multi-scale fusion enhanced spatial structure value of the display module image, and calculate the square root of the sum of the squares of the eigenvalues of the multi-scale fusion enhanced feature map of the display module image to obtain the second multi-scale fusion enhanced spatial structure value of the display module image;

[0056] Multiply each eigenvalue of the multi-scale fusion enhanced feature map of the display module image by the first multi-scale fusion enhanced spatial structure value and the second multi-scale fusion enhanced spatial structure value of the display module image respectively to obtain the first multi-scale fusion enhanced structure reference value and the second multi-scale fusion enhanced structure reference value corresponding to each eigenvalue;

[0057] Multiply each eigenvalue of the multi-scale fusion enhanced feature map of the display module image by the scale of the multi-scale fusion enhanced feature map of the display module image and the square root of the scale respectively to obtain the first multi-scale fusion enhanced scale transformation value and the second multi-scale fusion enhanced scale transformation value corresponding to each eigenvalue;

[0058] Divide the first multi-scale fusion enhanced structure reference value by the difference between the first multi-scale fusion enhanced spatial structure value and the first multi-scale fusion enhanced scale transformation value of the display module image to obtain the first multi-scale fusion enhanced transformation adjustment value;

[0059] Divide the second multi-scale fusion enhanced structure reference value by the difference between the second multi-scale fusion enhanced spatial structure value and the second multi-scale fusion enhanced scale transformation value of the display module image to obtain the second multi-scale fusion enhanced transformation adjustment value;

[0060] Calculate the weighted sum of the first multi-scale fusion enhanced transformation adjustment value and the second multi-scale fusion enhanced transformation adjustment value to obtain each eigenvalue of the optimized multi-scale fusion enhanced feature map of the display module image;

[0061] Input the optimized multi-scale fusion enhanced feature map of the display module image into the distortion correction module based on the diffusion model to obtain the gray image of the assembled state after distortion correction.

[0062] Here, the optimization of the multi-scale fusion enhanced feature map of the display module image, denoted as F, is expressed as:

[0063] F' = F1 ⊕ (ω ⊙ F2

[0064]

[0065] f 2i = (α × f i ) / (α - n × f i )

[0066]

[0067] n = W × H × C

[0068] Where F is the multi-scale fusion enhanced feature map of the display module image, f i are the respective feature values of the multi-scale fusion enhanced feature map of the display module image, n is the scale of the multi-scale fusion enhanced feature map of the display module image, α is the multi-scale fusion enhanced spatial structure value of the first display module image, β is the multi-scale fusion enhanced spatial structure value of the second display module image, f 1i is the multi-scale fusion enhanced transformation adjustment value of the first display module image, f 2i is the multi-scale fusion enhanced transformation adjustment value of the second display module image, ⊕ is dot addition, ⊙ is dot multiplication, and F' is the optimized multi-scale fusion enhanced feature map of the display module image.

[0069] That is, the applicant of the present application considered that the enhanced feature map of the display module image and the perceptually enhanced feature map of the display module image respectively represent the local spatial distribution enhancement feature based on the spatial distribution energy significance of the image semantic feature of the alignment and assembly state grayscale image and the global image feature semantic feature context enhancement feature based on the global feature distribution context. When fusing them, the local spatial-global context enhancement distribution differences of the image semantic features in different dimensions will also have the difference in the feature space distribution structure to be fused. Therefore, it is desired to improve the complex spatial structure generative regression convergence and generalization effect of the multi-scale fusion enhanced feature map of the display module image based on different spatial distribution structures.

[0070] Therefore, in the preferred example, for the spatial structure information of the feature set of the multi-scale fusion enhanced feature map of the display module image in the high-dimensional space, by using the class norm space structured representation of the multi-scale fusion enhanced feature map of the display module image as a reference window, performing a scale-based box transformation on each eigenvalue of the multi-scale fusion enhanced feature map of the display module image, and realizing a box attention weight adjustment based on the spatial structure of each eigenvalue of the multi-scale fusion enhanced feature map of the display module image, to ensure the invariance of the spatial transformation (translation, scaling, and rotation) of the multi-scale fusion enhanced feature map of the display module image under feature space interaction, thereby improving the convergence and generalization effects of the generation regression of the feature set of the multi-scale fusion enhanced feature map of the display module image under the complex spatial structure representation, and improving the image quality of the gray-scale image of the aligned assembly state after distortion correction obtained by inputting it into the distortion correction module based on the diffusion model. In this way, a gray-scale image of the aligned assembly state with higher quality can be generated, and then the corrected image is used to calculate the positioning deviation and adjust the pose of the display module, which can greatly improve the accuracy and efficiency of the alignment and assembly of the display module, reduce human errors at the same time, and ensure the consistency of product quality.

[0071] Specifically, in step S4, identify the positioning marks in the gray-scale image of the aligned assembly state after distortion correction to obtain the current positioning result. That is, through image processing techniques, extract the features of the positioning marks from the gray-scale image after distortion correction, such as edges, corners, shapes, etc. In one example, match the extracted features with a predefined template to determine the position of the positioning marks to obtain the current positioning result.

[0072] Specifically, in step S5, based on the comparison between the current positioning result and the preset target position, calculate the deviation value. By calculating the deviation value between the current positioning result and the preset target position, the position deviation of the display module in each direction can be accurately detected. In one example, the deviation value may include the translation deviation in the x-axis and y-axis and the rotation angle deviation.

[0073] Specifically, in step S6, based on the deviation value, adjust the pose of the display module through the mechanical control system. It should be understood that by adjusting the pose of the display module according to the calculated deviation value through the mechanical control system, the alignment deviation can be accurately corrected to ensure that the display module reaches the preset target position during the assembly process.

[0074] In summary, the vision positioning method for the alignment and assembly of a display module according to an embodiment of the present application is elucidated. By using an image processing and analysis algorithm based on artificial intelligence and machine vision technologies, the grayscale image of the alignment and assembly state is analyzed to capture the multi-scale enhanced fusion features of the display module image therein, and then the image distortion correction is performed using these features to generate the grayscale image of the alignment and assembly state after distortion correction. In this way, using the corrected image to calculate the positioning deviation and adjust the pose of the display module can greatly improve the accuracy and efficiency of the alignment and assembly of the display module, while reducing human errors and ensuring the consistency of product quality.

[0075] Furthermore, a vision positioning system for the alignment and assembly of a display module is also provided.

[0076] Figure 4 FIG. is a block diagram of a vision positioning system for the alignment and assembly of a display module according to an embodiment of the present application. As Figure 4 shown, the vision positioning system 300 for the alignment and assembly of a display module according to an embodiment of the present application includes: an alignment and assembly state image acquisition module 310 for acquiring the alignment and assembly state image of the display module collected by an industrial camera; a grayscale processing module 320 for performing grayscale processing on the alignment and assembly state image to obtain a grayscale image of the alignment and assembly state; a distortion correction module 330 for performing distortion correction on the grayscale image of the alignment and assembly state to obtain a grayscale image of the alignment and assembly state after distortion correction; a positioning recognition module 340 for recognizing the positioning marks in the grayscale image of the alignment and assembly state after distortion correction to obtain the current positioning result; a deviation calculation module 350 for calculating a deviation value based on the comparison between the current positioning result and a preset target position; and a pose adjustment module 360 for adjusting the pose of the display module through a mechanical control system based on the deviation value.

[0077] As described above, the vision positioning system 300 for the alignment and assembly of a display module according to an embodiment of the present application can be implemented in various wireless terminals, such as a server having a vision positioning algorithm for the alignment and assembly of a display module. In a possible implementation manner, the vision positioning system 300 for the alignment and assembly of a display module according to an embodiment of the present application can be integrated into a wireless terminal as a software module and / or a hardware module. For example, the vision positioning system 300 for the alignment and assembly of a display module can be a software module in the operating system of the wireless terminal, or can be an application program developed for the wireless terminal; of course, the vision positioning system 300 for the alignment and assembly of a display module can also be one of the many hardware modules of the wireless terminal.

[0078] Alternatively, in another example, the vision positioning system 300 for aligning and assembling the display module and the wireless terminal may also be separate devices, and the vision positioning system 300 for aligning and assembling the display module can be connected to the wireless terminal through a wired and / or wireless network and transmit interaction information in accordance with a predefined data format.

[0079] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of technologies in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A visual positioning method for the alignment and assembly of a display module, characterized in that, Including: Obtain the alignment and assembly status image of the display module collected by the industrial camera; Perform grayscale processing on the alignment and assembly status image to obtain the alignment and assembly status grayscale image; Perform distortion correction on the alignment and assembly status grayscale image to obtain the alignment and assembly status grayscale image after distortion correction; Identify the positioning marks in the alignment and assembly status grayscale image after distortion correction to obtain the current positioning result; Based on the comparison between the current positioning result and the preset target position, calculate the deviation value; Based on the deviation value, adjust the pose of the display module through the mechanical control system; Among them, performing distortion correction on the alignment and assembly status grayscale image to obtain the alignment and assembly status grayscale image after distortion correction includes: Perform feature extraction on the alignment and assembly status grayscale image based on the image of the display module to obtain the display module image features; Perform space constraint enhancement processing on the display module image features to obtain the display module image enhanced features; Perform feature perception enhancement processing on the display module image features to obtain the display module image perception enhanced features; Fuse the display module image enhanced features and the display module image perception enhanced features, and use the multi-scale fusion enhanced features of the fused display module image to generate the alignment and assembly status grayscale image after distortion correction.

2. The vision positioning method for alignment and assembly of a display module according to claim 1, wherein, Performing space constraint enhancement processing on the display module image features to obtain the display module image enhanced features includes: after performing feature dispersion on the display module image features to obtain a set of display module image pixel granularity features, calculating the semantic association constraint score of the set of display module image pixel granularity features to obtain the topological representation of the pixel granularity semantic association space constraint score; performing semantic association encoding on the set of display module image pixel granularity features and the topological representation of the pixel granularity semantic association space constraint score to obtain the display module image enhanced features.

3. The visual positioning method for alignment and assembly of a display screen module according to claim 2, characterized in that, Performing feature extraction on the alignment and assembly status grayscale image based on the image of the display module to obtain the display module image features includes: inputting the alignment and assembly status grayscale image into the display module image feature extractor based on the depthwise separable convolutional network to obtain the display module image feature map as the display module image features.

4. The vision positioning method for alignment and assembly of a display module according to claim 3, wherein, After performing feature dispersion on the display module image features to obtain a set of display module image pixel granularity features, calculating the semantic association constraint score of the set of display module image pixel granularity features to obtain the topological representation of the pixel granularity semantic association space constraint score includes: Perform feature dispersion on the display module image feature map along the channel dimension to obtain a set of display module image pixel granularity feature vectors; Calculate the semantic association score between any two display module image pixel granularity feature vectors in the set of display module image pixel granularity feature vectors to obtain the pixel granularity semantic association score topological matrix; Performing spatial attention attenuation modulation on the pixel granularity semantic association score topological matrix based on the spatial distance between any two display module image pixel granularity feature vectors in the set of display module image pixel granularity feature vectors to obtain a pixel granularity semantic association spatial soft constraint score topological matrix; Performing dilated convolution encoding on the pixel granularity semantic association spatial soft constraint score topological matrix to obtain a pixel granularity semantic association spatial soft constraint score topological feature matrix as the pixel granularity semantic association spatial constraint score topological representation.

5. The vision positioning method for alignment and assembly of a display module according to claim 4, wherein Calculating the semantic association score between any two display module image pixel granularity feature vectors in the set of display module image pixel granularity feature vectors to obtain a pixel granularity semantic association score topological matrix, including: Calculating the square of the first norm of the difference in position between the i-th display module image pixel granularity feature vector and the j-th display module image pixel granularity feature vector in the set of display module image pixel granularity feature vectors to obtain a display module image pixel granularity semantic difference first norm representation; Calculating the difference between the constant 1 and the square of the first norm of the i-th display module image pixel granularity feature vector to obtain the i-th display module image pixel granularity semantic representation; Calculating the difference between the constant 1 and the square of the first norm of the j-th display module image pixel granularity feature vector to obtain the j-th display module image pixel granularity semantic representation; After calculating the product between the i-th display module image pixel granularity semantic representation and the j-th display module image pixel granularity semantic representation, dividing the value obtained by the product using the display module image pixel granularity semantic difference first norm representation to obtain a display module image pixel granularity semantic association information representation; Calculating the sum of twice the display module image pixel granularity semantic association information representation and the constant 1, and then inputting the obtained value into the inverse hyperbolic cosine function to obtain a pixel granularity semantic association score value; Arranging multiple pixel granularity semantic association score values in matrix form to obtain the pixel granularity semantic association score topological matrix.

6. The visual positioning method for alignment and assembly of a display module according to claim 5, wherein Performing spatial attention attenuation modulation on the pixel granularity semantic association score topological matrix based on the spatial distance between any two display module image pixel granularity feature vectors in the set of display module image pixel granularity feature vectors to obtain a pixel granularity semantic association spatial soft constraint score topological matrix, including: Calculating the spatial distance between the i-th display module image pixel granularity feature vector and the j-th display module image pixel granularity feature vector to obtain a pixel granularity semantic spatial distance representation; Calculating the value of the natural exponential function with the pixel granularity semantic spatial distance representation as the exponent and the natural constant e as the base to obtain a pixel granularity semantic spatial distance class support representation; After multiplying the pixel-granularity semantic space distance class support representation by the spatial attention attenuation modulation hyperparameter to obtain the spatial distance attention attenuation representation, divide the eigenvalue at the (i, j) position in the pixel-granularity semantic association score topology matrix by the spatial distance attention attenuation representation to obtain the pixel-granularity semantic association space soft constraint score; Arrange multiple pixel-granularity semantic association space soft constraint scores in matrix form to obtain the pixel-granularity semantic association space soft constraint score topology matrix.

7. The visual positioning method for alignment and assembly of a display module according to claim 6, wherein Perform semantic association encoding on the set of display module image pixel-granularity features and the pixel-granularity semantic association space constraint score topology representation to obtain the display module image enhanced features, including: Input the set of display module image feature vectors and the pixel-granularity semantic association space soft constraint score topology matrix into a graph convolutional encoding module to obtain a set of display module image context pixel-granularity feature vectors; Reshape the feature shape of the set of display module image context pixel-granularity feature vectors to obtain a display module image enhanced feature map as the display module image enhanced features.

8. The visual positioning method for alignment and assembly of a display module according to claim 7, characterized in that, Perform feature perception enhancement processing on the display module image features to obtain display module image perception enhanced features, including: Input the display module image feature map into a transformer-based feature perception enhancement module to obtain a display module image perception enhanced feature map as the display module image perception enhanced features.

9. The vision positioning method for alignment and assembly of a display screen module according to claim 8, characterized in that, Fuse the display module image enhanced features and the display module image perception enhanced features, and use the fused display module image multi-scale fusion enhanced features to generate the gray image of the aligned and assembled state after distortion correction, including: Fuse the display module image enhanced feature map and the display module image perception enhanced feature map to obtain a display module image multi-scale fusion enhanced feature map as the display module image multi-scale fusion enhanced features; Input the display module image multi-scale fusion enhanced feature map into a distortion correction module based on a diffusion model to obtain the gray image of the aligned and assembled state after distortion correction.

10. A vision positioning system for alignment and assembly of a display module, characterized in that, Including: An aligned and assembled state image acquisition module for acquiring the aligned and assembled state image of the display module collected by an industrial camera; A gray processing module for performing gray processing on the aligned and assembled state image to obtain a gray image of the aligned and assembled state; A distortion correction module for performing distortion correction on the gray image of the aligned and assembled state to obtain a gray image of the aligned and assembled state after distortion correction; A positioning and recognition module for recognizing the positioning marks in the gray image of the aligned and assembled state after distortion correction to obtain the current positioning result; A deviation calculation module for calculating a deviation value based on the comparison between the current positioning result and a preset target position; A pose adjustment module for adjusting the pose of the display module through a mechanical control system based on the deviation value; Among them, performing distortion correction on the gray image of the alignment and assembly state to obtain the gray image of the alignment and assembly state after distortion correction includes: extracting features of the display module image from the gray image of the alignment and assembly state to obtain display module image features; performing space constraint enhancement processing on the display module image features to obtain enhanced display module image features; performing feature-aware enhancement processing on the display module image features to obtain perception-enhanced display module image features; fusing the enhanced display module image features and the perception-enhanced display module image features, and using the multi-scale fusion enhanced features of the fused display module image to generate the gray image of the alignment and assembly state after distortion correction.

Images