CLIP control method and system for upper and lower lamination frame product, product and medium

Abstract

The invention provides a CLIP control method and system for an upper and lower lamination frame product, the product and a medium, and belongs to the field of upper and lower lamination frame product control, and the method comprises the steps: obtaining a lamination image group of the upper and lower lamination frame product, the lamination image group comprising a plurality of lamination images of the upper and lower lamination frame product in different lamination states; performing image segmentation on each lamination image in the lamination image group into a plurality of regions, wherein each region only corresponds to one lamination; extracting feature information of the region; based on the region and the feature information thereof, constructing a hierarchical structure representing a lamination space relationship; and training the CLIP model by using the laminated image group and the corresponding hierarchical structure, so as to improve the ability of the model to describe complex concepts and details of the problem, and solve the problem that the CLIP model has insufficient cognition on the stacking relationship at the same time.

Description

Technical Field

[0001] The present application relates to the field of control of upper and lower stacking frame products, and in particular to a CLIP control method, system, product and medium for upper and lower stacking frame products. Background Art

[0002] The upper and lower stacking frame product is a key component used in precision manufacturing and assembly processes. This frame is designed to support and position multiple layers of materials or components and is typically used in industries with high precision requirements such as electronics, semiconductors, and optics.

[0003] In automated production lines, stacking control technology usually involves robotic arms or dedicated loading and unloading devices. These devices can automatically complete the placement and positioning of stacks according to preset programs. To further improve accuracy and reliability, production lines are often equipped with visual inspection systems or laser measurement technology. These advanced monitoring systems can monitor the position and posture of stacks in real time during the production process. Once deviations are detected, the system can be adjusted immediately to ensure the overall quality and consistency of the product.

[0004] However, in the manufacturing and inspection process of upper and lower stacked frame products, the complexity and diversity of product structure and the variability of production processes lead to a wide variety of problems, and the visual inspection system is too poor in describing the complex concepts and details of the problems. Summary of the invention

[0005] The present application provides a CLIP control method, system, product and medium for an upper and lower stacking frame product, which is used to improve the model's ability to describe complex concepts and details of problems, while solving the problem of insufficient understanding of stacking relationships by the CLIP model.

[0006] In a first aspect, the present application provides a CLIP control method for an upper and lower stacking frame product, comprising: obtaining a stacking image group of the upper and lower stacking frame product, the stacking image group comprising a plurality of stacking images of the upper and lower stacking frame product in different stacking states; performing image segmentation on each stacking image in the stacking image group into a plurality of regions, wherein each region corresponds to only one stack; extracting feature information from the regions; constructing a hierarchical structure representing the spatial relationship of stacking based on the regions and their feature information; training a CLIP model with the stacking image group and its corresponding hierarchical structure, wherein: the CLIP model extracts image features of the stacking image and encodes them, and uses the hierarchical structure to train the CLIP model by contrastive learning, so that the CLIP model associates the spatial relationship of stacking with the image features.

[0007] By adopting the above technical solution, the use of the CLIP model can improve the ability to identify and describe problem types. Multiple stacking images in different stacking states are used as training samples, and the samples have information about stacking frame products in different stacking states. The stacking images are segmented and feature extracted to construct a hierarchical structure representing the spatial relationship of the stacking, which provides the model with key information to understand the relative positions between the stacking sheets and solves the problem of the CLIP model's insufficient understanding of the stacking relationship. The CLIP model is trained by contrastive learning, so that it can associate the stacking spatial relationship with image features. Therefore, the ability to identify and describe a variety of problem types is improved, adapting to the variability of the production process, and at the same time, through the understanding of the stacking relationship, the problems of misidentification and inaccurate recognition are reduced.

[0008] In combination with some embodiments of the first aspect, in some embodiments, after the step of training the CLIP model with the stacked image group and its corresponding hierarchical structure, the method also includes: performing geometric transformation on a part of the stacked images in the stacked image group, the geometric transformation including rotation, scaling and cropping; performing optical transformation on another part of the stacked images in the stacked image group, the optical transformation including brightness adjustment, contrast adjustment and color balance; combining the stacked images after the geometric transformation and the optical transformation into an extended stacked image group, wherein the extended stacked image group inherits the corresponding hierarchical structure; and further training the CLIP model with the extended stacked image group and its corresponding hierarchical structure.

[0009] By adopting the above technical solutions, geometric transformations such as rotation, scaling and cropping are performed on part of the stacked images, which enhances the model's adaptability to different viewing angles and scales. Secondly, optical transformations are performed on another part of the images, including brightness adjustment, contrast adjustment and color balance, which improves the model's recognition ability under different lighting conditions. These transformed images form an extended stacked image group, which greatly increases the diversity of training samples. By inheriting the original hierarchical structure, it is ensured that the model's understanding of the spatial relationship of the stacked sheets will not be distorted by data enhancement. Finally, the CLIP model is further trained using the expanded dataset, enabling it to better cope with various changes in the actual production environment, thereby improving the model's detection accuracy in complex and changeable manufacturing processes.

[0010] In combination with some embodiments of the first aspect, in some embodiments, after the step of training the CLIP model with the stack image group and its corresponding hierarchical structure, the method also includes: inputting the stack image to be detected into the CLIP model to obtain text content describing the current stack quality status, wherein the text content includes multiple quantified quality indicators, and the current stack is the topmost stack in the stack image to be detected; comparing the quality indicator with the preset quality assessment standard to determine whether there is a quality indicator that exceeds the quality assessment standard; if there is a quality indicator that exceeds the standard, executing the upper and lower stack control strategy.

[0011] By adopting the above technical solution, the model can convert the image of the stack to be inspected into text content that describes the current quality status of the stack, where the text content contains multiple quantitative quality indicators, providing an objective basis for subsequent quality assessment. By comparing these indicators with the preset quality assessment standards, the system can quickly determine whether there are quality problems. When quality indicators that exceed the standard are found, the system will trigger the upper and lower stack control strategies, achieving seamless connection from detection to control, and improving the quality control capabilities of the production process.

[0012] In combination with some embodiments of the first aspect, in some embodiments, the upper and lower stacking control strategy includes: calculating the deviation value between the quality indicator that exceeds the standard and the corresponding quality assessment standard; based on the deviation value, generating an adjustment instruction set containing specific parameters and operation steps, the adjustment instruction set is used to control the device to adjust the position of the current stack; outputting the adjustment instruction set.

[0013] By adopting the above technical solution, the deviation between the quality index that exceeds the standard and the quality assessment standard is calculated. Based on these deviation values, an adjustment instruction set containing specific parameters and operation steps can be automatically generated. These instructions are directly used to control the equipment to adjust the position of the current stack, achieving a seamless connection from quality detection to actual adjustment, and improving the high quality and consistency of the upper and lower stack frame products.

[0014] In combination with some embodiments of the first aspect, in some embodiments, the upper and lower stacking control strategy includes: calculating the deviation value between the quality indicator that exceeds the standard and the corresponding quality assessment standard; based on the deviation value, modifying the preset motion trajectory to generate an optimized motion trajectory, and the optimized motion trajectory is used to control the equipment to complete the assembly of the next upper and lower stacking frame product according to the optimized motion trajectory; and outputting the optimized motion trajectory.

[0015] By adopting the above technical solution, the deviation values ​​of the quality indicators are calculated. Based on these deviation values, the preset motion trajectory is modified to generate an optimized motion trajectory. Similar problems can be prevented from happening again by adjusting the assembly process of subsequent products. The optimized motion trajectory is directly used to control the equipment to complete the assembly of the next product, realizing real-time adaptive adjustment of the production process. The stability and quality consistency of production are improved, while the adjustment frequency is reduced, and the overall production efficiency is improved.

[0016] In combination with some embodiments of the first aspect, in some embodiments, after the step of training the CLIP model with the stacked image group and its corresponding hierarchical structure, the method also includes: inputting the received query text content into the CLIP model to obtain a query text feature vector; searching in a preset image index database, wherein: the image index database stores the image features of each stacked image in the stacked image group; calculating the similarity between the query text feature vector and the image features; based on the calculated similarity, filtering out stacked images that exceed a similarity threshold; and outputting the filtered stacked images and their corresponding similarity scores as a result set corresponding to the query text content.

[0017] By adopting the above technical solution, the query text is converted into a feature vector, which reflects the model's understanding of the text semantics. The cross-modal matching capability of the CLIP model is utilized by searching in the preset image index database. By calculating the similarity between the query text feature vector and the image features, the stacked images most relevant to the query content can be efficiently identified. Setting a similarity threshold for screening ensures the relevance and accuracy of the retrieval results. The final output result set not only contains relevant stacked images, but also comes with a similarity score, providing users with an intuitive reference. This intelligent retrieval function improves the retrievability and utilization efficiency of product information, and provides strong support for quality control, process optimization, etc.

[0018] In combination with some embodiments of the first aspect, in some embodiments, after the step of outputting the filtered stacked images and their corresponding similarity scores as a result set corresponding to the query text content, the method also includes: if the result set is empty, returning a prompt message, suggesting that the user modify the query text content or adjust the similarity threshold.

[0019] By adopting the above technical solution, the system can guide users to perform more effective retrieval by returning prompt information and suggesting users to modify the query text content or adjust the similarity threshold. This interactive feedback mechanism helps users better understand and utilize the retrieval function of the system.

[0020] In a second aspect, the present application provides a CLIP control system for an upper and lower stacking frame product, the CLIP control system for the upper and lower stacking frame product comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions to enable the CLIP control system for the upper and lower stacking frame product to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0021] In a third aspect, the present application provides a computer program product comprising instructions, which, when the computer program product is run on a CLIP control system of an upper and lower stacking frame product, enables the CLIP control system of the upper and lower stacking frame product to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a CLIP control system of an upper and lower stacking frame product, causes the CLIP control system of the upper and lower stacking frame product to execute the method described in the first aspect and any possible implementation of the first aspect.

[0023] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. The use of the CLIP model can improve the ability to identify and describe problem types. Multiple stacking images in different stacking states are used as training samples. The samples contain information about stacking frame products in different stacking states. The stacking images are segmented and feature extracted to construct a hierarchical structure representing the spatial relationship of the stacking, which provides the model with key information to understand the relative positions between the stacking layers and solves the problem of the CLIP model's insufficient understanding of the stacking relationship. The CLIP model is trained by contrastive learning, so that it can associate the stacking spatial relationship with image features. Therefore, the ability to identify and describe a variety of problem types is improved, adapting to the variability of the production process. At the same time, through the understanding of the stacking relationship, the problems of misidentification and inaccurate recognition are reduced.

[0024] 2. The model can convert the image of the stack to be inspected into text content describing the current stack quality status. The text content contains multiple quantitative quality indicators, which provides an objective basis for subsequent quality assessment. By comparing these indicators with the preset quality assessment standards, the system can quickly determine whether there are quality problems. When quality indicators that exceed the standard are found, the system will trigger the upper and lower stack control strategies, achieving seamless connection from detection to control, and improving the quality control capabilities of the production process.

[0025] 3. It reflects the model's understanding of text semantics. Searching in the preset image index database utilizes the cross-modal matching capability of the CLIP model. By calculating the similarity between the query text feature vector and the image features, the laminated images most relevant to the query content can be efficiently identified. Setting a similarity threshold for screening ensures the relevance and accuracy of the retrieval results. The final output result set not only contains relevant laminated images, but also comes with a similarity score, providing an intuitive reference for users. This intelligent retrieval function improves the retrievability and utilization efficiency of product information, and provides strong support for quality control, process optimization, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of a CLIP control method for upper and lower stacking frame products in an embodiment of the present application; Figure 2 is another flow chart of the CLIP control method for the upper and lower stacking frame products in the embodiment of the present application; Figure 3 is another flow chart of the CLIP control method for the upper and lower stacking frame products in the embodiment of the present application; Figure 4 It is an exemplary hardware structure diagram of the CLIP control system of the upper and lower stacking frame products in the embodiment of the present application. DETAILED DESCRIPTION

[0027] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0028] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0029] To facilitate understanding, the relevant terms and concepts involved in the embodiments of the present application are first introduced below.

[0030] (1) Upper and lower stacking frame products: Upper and lower stacking frame products are key components used in precision manufacturing and assembly processes. This frame is designed to support and position multiple layers of materials or components and is typically used in industries with high precision requirements such as electronics, semiconductors, and optics.

[0031] (2) CLIP model: CLIP is a model that combines natural language processing and computer vision techniques and can simultaneously understand and process text and image data. CLIP uses a contrastive learning approach and is pre-trained on a large-scale image-text pair dataset. This approach enables the model to learn the semantic associations between images and text. CLIP consists of two main components: a text encoder and an image encoder, which map the inputs to a shared multimodal representation space.

[0032] To facilitate understanding, the derivation process of the technical problem of the embodiment of the present application is first introduced below.

[0033] In the manufacturing and inspection process of upper and lower stacked frame products, due to the complexity and diversity of product structure and the variability of production process, there are many types of problems. Therefore, visual inspection is too poor in describing the complex concepts and details of the problems.

[0034] Further research found that the CLIP model has a strong ability to associate image-text semantics, where the range and flexibility of text descriptions far exceeds that of images. This feature is highly consistent with the assembly process of upper and lower stacking frame products. During the assembly process, the robot arm or dedicated loading and unloading device follows the instructions generated by CAM (computer-aided manufacturing), which are essentially structured text content. This text-based instruction forms a natural synergy with the text processing capabilities of the CLIP model, providing an ideal foundation for more accurate problem identification and description.

[0035] However, in actual use, when CLIP processes products with stacking relationships, it lacks knowledge of the stacking relationships, which leads to problems of misidentification and inaccurate recognition.

[0036] See also Figure 1 , Figure 1 It is a flow chart of a CLIP control method for upper and lower stacking frame products in an embodiment of the present application; S101, obtaining a stacking image group of an upper and lower stacking frame product, the stacking image group comprising a plurality of stacking images of the upper and lower stacking frame product in different stacking states; The stacking image group represents a series of images reflecting different stacking states of the product, such as the image of the upper and lower stacking frame product after installing stacking A and the image after installing stacking B. The stacking image refers to a single image that captures a specific stacking state of the upper and lower stacking frame product.

[0037] In some embodiments, operators or automated equipment may take multiple images at different stages of the product production process, or at different angles and states of the finished product. These images cover the entire process of the product from its initial state to its final finished state.

[0038] S102, segmenting each laminate image in the laminate image group into a plurality of regions, wherein each region corresponds to only one laminate; In some embodiments, image segmentation algorithms are applied to each acquired laminate image. These algorithms identify the boundaries of different laminates based on the color, texture, edge and other features in the image, and divide the image into multiple non-overlapping regions. Each region precisely corresponds to a laminate in the product, so that the characteristics and positional relationship of each laminate can be analyzed in more detail.

[0039] For example, in some specific embodiments, step S102 specifically includes: S1021, applying an edge detection algorithm (such as Hough transform, not limited here) to each image to detect straight lines, identify possible overlapping boundaries, and preliminarily divide potential overlapping areas based on the detected boundaries.

[0040] S1022. In each potential overlapping area, calculate the distance transform, select the local maximum point of the distance transform as the seed point, and ensure that each initially divided area has at least one seed point.

[0041] S1023. For each potential overlapping region, calculate the local grayscale statistical features (mean, standard deviation), set an adaptive similarity threshold T for each region, define a spatial continuity constraint, and ensure that the shape of the growing region meets the overlapping characteristics.

[0042] S1024. Perform the following operations on each seed point: S10241. Initialize a marked image with the same size as the original image.

[0043] S10242. Create a priority queue Q and sort the pixels by the similarity between the pixels and the seed points.

[0044] S10243. Add the seed point to Q and mark it in the marked image.

[0045] S10244. When Q is not empty, execute: a) Take the most similar pixel P from Q.

[0046] b) Check the 8-neighborhood pixels of P.

[0047] c) For each unlabeled neighborhood pixel N: When the grayscale difference between N and the seed point is less than the similarity threshold T, N is marked as the current overlapping area and added to Q, with the priority based on the similarity with the seed point.

[0048] S10245, repeat S10244, until Q is empty or reaches the preset upper limit of the stacking area.

[0049] In some embodiments, other methods may be used to perform image segmentation, which are not limited here.

[0050] S103, extracting feature information from the region; Feature information refers to the key attributes that can describe and distinguish different regions. For example, for a stacked region, feature information such as shape, texture, and color distribution may be extracted.

[0051] In some embodiments, feature extraction algorithms are applied to each region segmented in S102. These algorithms analyze the pixel distribution, edge features, texture patterns, etc. of the region, and convert complex visual information into a series of numerical feature vectors. Specifically, traditional computer vision methods can be used to first calculate the shape descriptors of the region, such as geometric features such as area, perimeter, and roundness; then extract color features, such as color histograms or color moments; then analyze texture information, using grayscale co-occurrence matrices or local binary patterns (LBP); finally, combine these different types of features into a comprehensive feature vector. It is understandable that feature extraction can also be achieved in other ways, which are not limited here.

[0052] S104, constructing a hierarchical structure representing the spatial relationship of overlapping slices based on the regions and their feature information; A hierarchy is a data organization that can represent multiple levels of relationships. For example, a tree structure can be constructed where the root node represents the entire product, the child nodes represent the different stacks, and the connections between the nodes represent the spatial relationships between the stacks.

[0053] In some embodiments, the position of each laminate area is analyzed to infer the spatial relationship between the laminates, such as up and down, inside and outside, and the degree of overlap. Then, based on this relationship information, a hierarchical data structure is constructed, which not only contains the characteristics of each laminate, but also encodes the relative position and organization of the laminates. This structure enables the CLIP model to understand the overall layout of the product and the relationship between the various parts.

[0054] In some embodiments, each stack region is represented as a node in a graph; then, weighted edges are added between nodes based on the spatial relationship between stacks; then, a graph embedding algorithm, such as Node2Vec, is used to convert the graph structure into a low-dimensional vector representation; finally, based on these vector representations, a hierarchical clustering algorithm is used to construct a tree-like hierarchical structure. Other methods can also be used to implement the construction of the hierarchical structure, which is not limited here.

[0055] S105, training a CLIP model with the stacked image group and its corresponding hierarchical structure, wherein: the CLIP model extracts image features of the stacked image and encodes them, and trains the CLIP model by contrastive learning using the hierarchical structure, so that the CLIP model associates the stacked spatial relationship with the image features.

[0056] Contrastive learning is a machine learning method that learns effective representations by comparing similar and dissimilar sample pairs, and image feature encoding refers to converting an image into a vector representation of a fixed dimension.

[0057] It should be noted that the CLIP model is already very mature in related technologies, and the innovation of the present invention does not lie in the training of the CLIP model.

[0058] Therefore, only the modified parts are described below: Create a corresponding text description for each stacked image, including the hierarchical information. Design a language template to describe the hierarchical relationship of the stacked images, ensuring that the positional relationship is covered. Generate multiple descriptions for each stacked image: a complete hierarchical description, a partial hierarchical description (such as only describing the top and bottom layers), and an incorrect hierarchical description (used as a negative sample). Create (image, text description) pairs as training data.

[0059] Keep the original image encoder and text encoder architecture of CLIP unchanged. Adjust the batch processing strategy to ensure that each batch of data contains: correct complete hierarchical description, correct partial hierarchical description, and incorrect hierarchical description. Implement hierarchical-aware contrastive learning: define positive sample pairs (stacked images, correct hierarchical description) and negative sample pairs (stacked images, incorrect hierarchical description). Adjust the contrastive loss function to increase the weight of hierarchical understanding: give higher rewards to samples that correctly describe hierarchical relationships, and impose greater penalties on samples that incorrectly describe hierarchical relationships.

[0060] Input the stacked image to the image encoder and the corresponding hierarchical structure description to the text encoder. Calculate the similarity between the image features and the text features. Apply the modified contrast loss function. Update the model parameters through back propagation, so that the model gradually learns to recognize and understand complex stacking relationships.

[0061] It can be seen that the use of the CLIP model can improve the ability to identify and describe problem types. Multiple stacking images in different stacking states are used as training samples. The samples contain information about stacking frame products in different stacking states. The stacking images are segmented and features are extracted to construct a hierarchical structure representing the spatial relationship of the stacking. This provides the model with key information to understand the relative positions between stacking sheets and solves the problem of the CLIP model's insufficient understanding of stacking relationships. The CLIP model is trained by contrastive learning, enabling it to associate stacking spatial relationships with image features. Therefore, the ability to identify and describe a wide range of problem types is improved, adapting to the variability of the production process, and at the same time, through the understanding of stacking relationships, it reduces the problems of misidentification and inaccurate recognition. In some preferred embodiments, after step S105, the method further includes: S106, performing geometric transformation on a part of the laminated images in the laminated image group, where the geometric transformation includes rotation, scaling and cropping; Rotation refers to rotating an image around its center point by a certain angle, such as 45 degrees clockwise. Scaling refers to changing the size of an image, which can be enlarged or reduced. Cropping refers to cutting out a portion of the original image to remove unwanted parts or focus on a specific area.

[0062] It should be noted that the purpose of step S106 is to expand the diversity of the training data set and improve the generalization ability of the model. Specifically, a portion of images are selected from the original stacked image group, and different degrees of rotation, scaling and cropping operations are applied to each selected image. Rotation can be a random angle or a preset angle, such as 90 degrees, 180 degrees, etc.; scaling can randomly adjust the image size within a certain range; cropping can select different areas of the image, such as the central area, four corners, etc. These transformations can be applied individually or in combination to produce more diverse image variants.

[0063] S107, performing optical transformation on another part of the laminated images in the laminated image group, where the optical transformation includes brightness adjustment, contrast adjustment and color balance; Brightness adjustment refers to changing the overall brightness of an image, which can make the image brighter or darker. Contrast adjustment refers to changing the difference between the light and dark parts of an image, which can enhance or weaken the layering of the image. Color balance refers to adjusting the proportion and intensity of various colors in an image, which is used to correct color casts or create specific color effects.

[0064] It should be noted that the purpose of step S107 is to simulate the influence of different lighting conditions and imaging devices and improve the adaptability of the model to various visual conditions. Specifically, another part of the images is selected from the original stacked image group, and different degrees of brightness, contrast and color adjustment are applied to each selected image. Brightness adjustment can be achieved by increasing or decreasing pixel values; contrast adjustment can be achieved by stretching or compressing the pixel value range; color balance can be achieved by adjusting the ratio of RGB channels.

[0065] S108, combining the geometrically transformed and optically transformed laminated images into an extended laminated image group, wherein the extended laminated image group inherits the corresponding hierarchical structure; S109, further training the CLIP model with the expanded stacked image group and its corresponding hierarchical structure.

[0066] It should be noted that the principle and details of this step are similar to those of step S105. The relevant principles and details can be referred to step S105 and are not limited here.

[0067] It can be seen that geometric transformations such as rotation, scaling, and cropping of some stacked images enhance the model's adaptability to different viewing angles and scales. Secondly, optical transformations, including brightness adjustment, contrast adjustment, and color balance, are performed on another part of the images, which improves the model's recognition ability under different lighting conditions. These transformed images form an extended stacked image group, which greatly increases the diversity of training samples. By inheriting the original hierarchical structure, it is ensured that the model's understanding of the spatial relationship of the stacks will not be distorted by data augmentation. Finally, the CLIP model is further trained using the expanded dataset, enabling it to better cope with various changes in the actual production environment, thereby improving the model's detection accuracy in complex and changeable manufacturing processes.

[0068] The above embodiment elaborates on the training process of the CLIP model, and emphasizes how to enhance the model's understanding of the lamination relationship through hierarchical information. However, the real value of the model lies in its practical application. Next, we will specifically describe how to use the trained CLIP model to guide the production process of the upper and lower lamination frame products.

[0069] See also Figure 2 , Figure 2 is another flow chart of the CLIP control method for the upper and lower stacking frame products in the embodiment of the present application; After step S105, the method further includes: S201, inputting the laminate image to be detected into the CLIP model to obtain text content describing the quality status of the current laminate, wherein the text content includes multiple quantitative quality indicators, and the current laminate is the uppermost laminate in the laminate image to be detected; In some embodiments, the captured image of the stack to be inspected is input into a pre-trained CLIP model. The model analyzes the image features and generates a descriptive text that details the quality status of the current stack. This text contains not only qualitative descriptions, but also multiple quantitative quality indicators, such as coordinates, alignment, etc.

[0070] It should be noted that the CLIP model focuses on the most recently placed stack, i.e. the top stack, in each round of processing. Stacks in other layers are analyzed in their respective processing rounds.

[0071] S202, comparing the quality indicators with the preset quality assessment standards to determine whether there are quality indicators that exceed the quality assessment standards; The quality index represents various numerical parameters used to measure the quality of the lamination.

[0072] In some embodiments, pre-set quality assessment standards are called, which may come from industry specifications, customer requirements, or internal quality control policies. Then, each quality indicator is compared with its corresponding standard value one by one.

[0073] S203: If there are quality indicators that exceed the standards, the upper and lower stacking control strategy is executed.

[0074] The upper and lower lamination control strategy represents a series of operation methods for adjusting the lamination process.

[0075] It can be seen that the model can convert the image of the stack to be inspected into text content that describes the current quality status of the stack. The text content contains multiple quantitative quality indicators, which provides an objective basis for subsequent quality assessment. By comparing these indicators with the preset quality assessment standards, it is possible to quickly determine whether there are quality problems. When quality indicators that exceed the standard are found, the upper and lower stack control strategies will be triggered, achieving a seamless connection from detection to control, and improving the quality control capabilities of the production process. In some specific embodiments, the upper and lower lamination control strategy includes: S204, calculating the deviation value between the quality index exceeding the standard and the corresponding quality assessment standard; In some specific embodiments, for each exceeded indicator, the corresponding quality assessment standard value will be called. Then the deviation value is calculated S205, generating an adjustment instruction set including specific parameters and operation steps based on the deviation value, wherein the adjustment instruction set is used to control the device to adjust the position of the current stack; In some embodiments, the deviation value of each exceeded indicator is first analyzed, considering the direction and size of the deviation. Then, a preset adjustment rule library is called, which contains standard processing methods for different deviation situations. Based on these rules, an adjustment instruction set is generated. Each instruction in the instruction set contains specific parameter settings and detailed operation steps, such as "move 1.2mm to the left" or "rotate the alignment device 2 degrees clockwise".

[0076] In some specific embodiments, the deviation value and the adjustment rule are input into the rule engine, and then the reasoning mechanism of the rule engine is used to generate preliminary adjustment suggestions, and then the priority algorithm is applied to sort the adjustment suggestions, and finally the sorted suggestions are converted into a specific instruction set, including parameter settings and execution steps; it can be understood that other methods can also be used, which are not limited here.

[0077] S206: Output an adjustment instruction set.

[0078] Input into the robot arm so that the robot arm adjusts the position of the current stack according to the adjustment instruction set.

[0079] It can be seen that by calculating the deviation between the quality indicators that exceed the standard and the quality assessment standard, based on these deviation values, an adjustment instruction set containing specific parameters and operation steps can be automatically generated. These instructions are directly used to control the equipment to adjust the position of the current stack, achieving a seamless connection from quality detection to actual adjustment, and improving the high quality and consistency of the upper and lower stack frame products.

[0080] In some other specific embodiments, the upper and lower lamination control strategies include: S207, calculating the deviation value between the quality index exceeding the standard and the corresponding quality assessment standard; It should be noted that the principle and details of this step are similar to those of step S204. The relevant principles and details can be referred to step S204 and are not limited here.

[0081] S208, based on the deviation value, modify the preset motion trajectory to generate an optimized motion trajectory, the optimized motion trajectory is used to control the device to complete the assembly of the next upper and lower stacking frame product according to the optimized motion trajectory; The above embodiments mentioned that the robot arm or dedicated loading and unloading device follows the instructions generated by CAM (computer-aided manufacturing), and these instructions are essentially structured text content (preset motion trajectory), so it is sufficient to modify the numerical value of the structured text content, including changing the speed, strength, angle or sequence of certain actions.

[0082] S209: Outputting the optimized motion trajectory.

[0083] Input the robot arm or the dedicated loading and unloading device so that the robot arm or the dedicated loading and unloading device can complete the assembly of the next upper and lower stacking frame product according to the optimized motion trajectory.

[0084] It can be seen that the deviation values ​​of the quality indicators are calculated, and based on these deviation values, the preset motion trajectory is modified to generate an optimized motion trajectory. Similar problems can be prevented from happening again by adjusting the assembly process of subsequent products. The optimized motion trajectory is directly used to control the equipment to complete the assembly of the next product, realizing real-time adaptive adjustment of the production process. The stability and quality consistency of production are improved, while the adjustment frequency is reduced, and the overall production efficiency is improved.

[0085] In actual use, due to the complexity of the manufacturing process of laminated products involving multiple links, it is difficult for relevant personnel to quickly locate and retrieve image data corresponding to specific process problems during debugging and maintenance.

[0086] See also Figure 3 , Figure 3 is another flow chart of the CLIP control method for the upper and lower stacking frame products in the embodiment of the present application; After step S105, the method further includes: S301, inputting the received query text content into the CLIP model to obtain a query text feature vector; In some embodiments, the query text received from the user may be a descriptive sentence or a few keywords. Then, the text is preprocessed, such as removing stop words and standardizing. Then, the pre-trained CLIP model is called, and the CLIP model converts the text input into a high-dimensional vector that captures the semantic features of the text.

[0087] S302, searching in a preset image index database, wherein: the image index database stores image features of each laminated image in the laminated image group; The preset image index database refers to a pre-established structured data set storing image feature information. Image features refer to numerical representations describing image content.

[0088] In some embodiments, the query text feature vector is used as a retrieval condition to search in the database. The search process may use an approximate nearest neighbor (ANN) algorithm to quickly find similar image features in a large-scale data set. A preliminary set of candidate images will be returned, and the features of these images may have a certain degree of similarity with the query vector.

[0089] S303, calculating the similarity between the query text feature vector and the image feature; In some embodiments, a suitable similarity measurement method is first selected, commonly used ones include cosine similarity, Euclidean distance or dot product, etc. Then, each image feature vector retrieved from the database is traversed. For each image feature, it is paired with the query text feature vector and the selected similarity calculation method is applied. The calculation process may consider vector normalization to eliminate the impact of dimensional differences. If the data set is large, batch processing or parallel computing techniques may be used to speed up the process. At the same time, some optimization strategies may be applied, such as terminating low-similarity calculations early, to improve efficiency. Finally, a similarity score is generated for each image, which reflects the degree of match between the image content and the text query.

[0090] S304, based on the calculated similarity, filter out the stacked images that exceed the similarity threshold; In some embodiments, all images for which similarity is calculated are traversed. For each image, its similarity is compared with a set threshold. If the image's similarity exceeds the threshold, it is retained in the result set; otherwise, it is excluded. A list of filtered images is generated, which are considered to be most relevant to the query text.

[0091] S305 : Output the filtered stacked images and their corresponding similarity scores as a result set corresponding to the query text content.

[0092] In some embodiments, it can be seen that the conversion of query text into feature vectors reflects the model's understanding of text semantics. Retrieval is performed in a preset image index database, utilizing the cross-modal matching capability of the CLIP model. By calculating the similarity between the query text feature vector and the image features, the laminated images most relevant to the query content can be efficiently identified. Setting a similarity threshold for screening ensures the relevance and accuracy of the retrieval results. The final output result set not only contains relevant laminated images, but also comes with a similarity score, providing users with an intuitive reference. This intelligent retrieval function improves the retrievability and utilization efficiency of product information, and provides strong support for quality control, process optimization, etc.

[0093] In some other embodiments, the method further includes: S306, when the result set is empty, returning a prompt message to suggest the user to modify the query text content or adjust the similarity threshold.

[0094] It can be seen that by returning prompt information, it is suggested that users modify the query text content or adjust the similarity threshold, guiding users to perform more effective retrieval. This interactive feedback mechanism helps users better understand and utilize the retrieval function.

[0095] The following introduces a CLIP control system 400 of an exemplary upper and lower stacking frame product provided in an embodiment of the present application. Figure 4 Schematic diagram of an exemplary hardware structure of a CLIP control system 400 of an upper and lower stacking frame product provided in an embodiment of the present application.

[0096] In some embodiments, the CLIP control system 400 of the upper and lower stacking frame product is a computer device or the CLIP control system 400 of the upper and lower stacking frame product includes a computer device. The computer device includes a processor, a memory and a network interface connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with other external terminals or servers through a network connection. In some embodiments, the network interface can be a wired network interface, and in some embodiments, the network interface can also be a wireless network interface. When the computer program is executed by the processor, the method in the embodiment of the present application is implemented.

[0097] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0098] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0099] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0100] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0101] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A CLIP control method for upper and lower lamination frame products, characterized in that: include: Acquire a stacking image group of the upper and lower stacking frame products, the stacking image group comprising a plurality of stacking images of the upper and lower stacking frame products in different stacking states; Performing image segmentation on each of the laminated images in the laminated image group into a plurality of regions, wherein each region corresponds to only one laminated image; Extracting feature information from the region; Based on the regions and the feature information thereof, constructing a hierarchical structure representing the spatial relationship of the stacked sheets; The stacked image group and its corresponding hierarchical structure are used to train a CLIP model, wherein: the CLIP model extracts image features of the stacked images and encodes them, and the CLIP model is trained by contrastive learning using the hierarchical structure, so that the CLIP model associates the stacked spatial relationship with the image features.

2. The method according to claim 1, characterized in that After the step of training the CLIP model with the stacked image group and the corresponding hierarchical structure, the method further comprises: Performing geometric transformation on a portion of the laminated images in the laminated image group, wherein the geometric transformation includes rotation, scaling and cropping; Performing optical transformation on another part of the laminated images in the laminated image group, wherein the optical transformation includes brightness adjustment, contrast adjustment and color balance; Combining the geometrically transformed and optically transformed laminated images into an extended laminated image group, wherein the extended laminated image group inherits the corresponding hierarchical structure; The extended stacked image set and its corresponding hierarchical structure are used to further train the CLIP model.

3. The method according to claim 1, characterized in that After the step of training the CLIP model with the stacked image group and the corresponding hierarchical structure, the method further comprises: Inputting the stack image to be detected into the CLIP model to obtain text content describing the quality status of the current stack, wherein the text content includes multiple quantitative quality indicators, and the current stack is the top stack in the stack image to be detected; Compare the quality indicator with the preset quality assessment standard to determine whether there is any quality indicator that exceeds the quality assessment standard; If there are quality indicators that exceed the standards, the upper and lower lamination control strategy is executed.

4. The method according to claim 3, characterized in that The upper and lower lamination control strategy includes: Calculate the deviation value between the quality indicator exceeding the standard and the corresponding quality assessment standard; Based on the deviation value, generating an adjustment instruction set including specific parameters and operation steps, wherein the adjustment instruction set is used to control the device to adjust the position of the current lamination; The adjustment instruction set is output.

5. The method according to claim 3, characterized in that: The upper and lower lamination control strategy includes: Calculate the deviation value between the quality indicator exceeding the standard and the corresponding quality assessment standard; Based on the deviation value, modify the preset motion trajectory to generate an optimized motion trajectory, and the optimized motion trajectory is used to control the device to complete the assembly of the next upper and lower stacking frame product according to the optimized motion trajectory; The optimized motion trajectory is output.

6. The method according to claim 1, characterized in that After the step of training the CLIP model with the stacked image group and the corresponding hierarchical structure, the method further comprises: Inputting the received query text content into the CLIP model to obtain a query text feature vector; Searching in a preset image index database, wherein: the image index database stores image features of each of the laminated images in the laminated image group; Calculating the similarity between the query text feature vector and the image feature; Based on the calculated similarity, screening out the laminated images that exceed a similarity threshold; The filtered stacked images and their corresponding similarity scores are output as a result set corresponding to the query text content.

7. The method according to claim 6, characterized in that After the step of outputting the filtered laminated images and their corresponding similarity scores as a result set corresponding to the query text content, the method further comprises: When the result set is empty, a prompt message is returned to suggest that the user modify the query text content or adjust the similarity threshold.

8. A CLIP control system for upper and lower laminated frame products, characterized in that: The CLIP control system of the upper and lower stacking frame products includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the CLIP control system of the upper and lower stacking frame products to execute the method described in any one of claims 1-7.

9. A computer program product comprising instructions, characterized in that When the computer program product runs on a CLIP control system of an upper and lower lamination frame product, the CLIP control system of the upper and lower lamination frame product is enabled to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on the CLIP control system of the upper and lower lamination frame products, the CLIP control system of the upper and lower lamination frame products is enabled to execute the method according to any one of claims 1 to 7.

Images